The new server at 1&1 has been showing signs of unreliability, it has crashed and died mysteriously three times, the last while I was away for a couple of days. Late at night when I got back, I decided it was time to actually make it into a Fedora box with a kernel later than 2.6.16 and to get rid of the xfs-formatted partitions, which I suspect of causing the instability. So here are my notes on how to force-upgrade the weird FC4-based OS on those boxes to fully true Fedora 7, grub, ext3 and selinux. The notes might not be complete, but they contain all the major steps and will be useful for anyone contemplating changing their server over to "Genuine Fedora". Don't embark on this unless you have some Linux-fu and know how to get yourself out of trouble, because at every step you can easily trash your server and lose all your data. We are literally going to format the main filesystems and install a new bootloader on a remote server... we can call that "not a beginner project".

Sanity check that we have the same layout

Zero'th move is a sanity check. Confirm that your 1&1 server still has the same Raid 1 layout (the result of fdisk /dev/sd[a-b] and pressing p): Disk /dev/sda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 123 987966 fd Linux raid autodetect /dev/sda2 124 367 1959930 82 Linux swap / Solaris /dev/sda4 368 19457 153340425 5 Extended /dev/sda5 368 976 4891761 fd Linux raid autodetect /dev/sda6 977 1585 4891761 fd Linux raid autodetect /dev/sda7 1586 19457 143556808+ fd Linux raid autodetect Disk /dev/sdb: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 123 987966 fd Linux raid autodetect /dev/sdb2 124 367 1959930 82 Linux swap / Solaris /dev/sdb4 368 19457 153340425 5 Extended /dev/sdb5 368 976 4891761 fd Linux raid autodetect /dev/sdb6 977 1585 4891761 fd Linux raid autodetect /dev/sdb7 1586 19457 143556808+ fd Linux raid autodetect # mount /dev/md1 on / type ext3 (rw) none on /proc type proc (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/md5 on /usr type xfs (rw) /dev/md7 on /var type xfs (rw,usrquota) none on /tmp type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) If that is the same you are in with a chance with these instructions.

Create missing raid config file

First create /etc/mdadm.conf ... NOT /etc/mdadm/mdadm.conf ARRAY /dev/md5 devices=/dev/sda5,/dev/sdb5 ARRAY /dev/md6 devices=/dev/sda6,/dev/sdb6 ARRAY /dev/md7 devices=/dev/sda7,/dev/sdb7 ARRAY /dev/md1 devices=/dev/sda1,/dev/sdb1 Grub will need this later and mkinitrd in the next step for the new kernel will use it also.

Dist-upgrade to F7

Next move is to dist-upgrade to F7 from inside the mutant FC4 environment. # wget http://www.mirrorservice.org/sites/download.fedora.\ redhat.com/pub/fedora/linux/releases/7/Fedora/i386/os/\ Fedora/fedora-release-7-3.noarch.rpm # rpm -Uvf --nodeps fedora-release-7-3.noarch.rpm # yum clean all # yum update yum # yum update # yum install grub This package contains the yum repo set down /etc/yum.repos.d configured for Fedora 7 repos. So the next yum update you do is going to "update" you to F7 since all the packages it can see are now of F7 vintage. However, on reboot, you are going to come back up in the old mutant 2.6.16 Debian kernel. If you try to use the Fedora kernel, you will have immediate deadly trouble with lilo and xfs support. Nice "Fedora" they got there, with freaking lilo.

Reformatting all xfs into ext3

Next move is to get rid of this xfs-format crud and replace it with ext3. I started out doing it inside the normal boot environment for /usr, and then was forced to change to using the rescue environment to do /var, so I will describe both ways as I did them. I logged into the serial console from my local machine here using $ ssh (magic userid)@sercon.onlinehome-server.info and gave my "original root password". You can find this and the (magic userid) in your 1&1 control panel under "serial console" Then I logged into the box there using my real root credentials, and did # telinit 1 this will kill all your networking and disable all services. Confirm everything went down with ps -Af and kill anything that is still up, except your sh session. Now we get rid of any junk in the yum cache and then backup /var sh-3.1# yum clean all sh-3.1# tar czf /usr/backup-var.tar.gz /var sh-3.1# umount /var At this point, the 2.6.16 weirdo kernel blew a warning, which I ignored because that XFS formatted filesystem is about to get a well-deserved deletion Badness in mutex_destroy at kernel/mutex-debug.c:458 Call Trace: {mutex_destroy+109} {xfs_qm_destroy+140} {xfs_qm_rele_quotafs_ref+165} {xfs_qm_destroy_quotainfo+18} {xfs_mount_free+160} {xfs_unmountfs+171} {xfs_unmount+275} {vfs_unmount+34} {linvfs_put_super+49} {generic_shutdown_super+153} {kill_block_super+36} {deactivate_super+103} {sys_umount+111} {error_exit+0} {system_call+126} Oh well, let's destroy the /var filesystem by rewriting it ext3. sh-3.1# mkfs.ext3 /dev/md7 mke2fs 1.39 (29-May-2006) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 17956864 inodes, 35889184 blocks 1794459 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 1096 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 39 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. Edit /etc/fstab to reflect our change (shows from -> to) /dev/md7 /var xfs defaults,usrquota 0 2 /dev/md7 /var ext3 defaults 0 2 Okay now we mount our clean empty ext3 /var back in place sh-3.1# mount /var kjournald starting. Commit interval 5 seconds EXT3 FS on md7, internal journal EXT3-fs: mounted filesystem with ordered data mode. sh-3.1# ll /var total 16 drwx------ 2 root root 16384 Sep 4 20:49 lost+found sh-3.1# cd / sh-3.1# tar zxf /usr/backup-var.tar.gz sh-3.1# rm /usr/backup-var.tar.gz At this point /var is back how it was, but it is now on ext3. Now we backup /usr and /home into our nice new /var sh-3.1# rsync -a /usr /var sh-3.1# rsync -a /home /var Now unfortunately I was unable to get /usr into a state that I could umount it cleanly... the sh had handles open to /usr/lib/ libraries. So I had to use a different technique to reformat /usr and /home in place. Go to your 1&1 control panel and select the "recovery tool" option. Make sure "reboot now" is unchecked, and select "Linux Rescue System (debian/woody - 2.6.x) ". Confirm it and you will get a one-time login password generated for the rescue system. Wait a couple of minutes and then sh-3.1# shutdown -r now your server. When it reboots, it will come up in the recovery system, which is a network boot with none of your local partitions mounted... this is most excellent and a really powerful solution for the kind of work we are doing on this server. However, save yourself some time and go back to the "recovery tool" page now, and again with "reboot now" unchecked, select "Normal System" again and confirm it. Otherwise you keep coming back into the rescue system in future boots too. Next we reformat the /usr partition, /dev/md5 rescue:~# mkfs.ext3 /dev/md5 mke2fs 1.40-WIP (14-Nov-2006) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 611648 inodes, 1222912 blocks 61145 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=1254096896 38 block groups 32768 blocks per group, 32768 fragments per group 16096 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 22 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. Let's mount our new, empty ext3 /usr partition at /mnt in the rescue filesystem rescue:~# mount /dev/md5 /mnt kjournald starting. Commit interval 5 seconds EXT3 FS on md5, internal journal EXT3-fs: mounted filesystem with ordered data mode. and we can mount our ext3 /var partition we made earlier at /opt rescue:~# mount /dev/md7 /opt kjournald starting. Commit interval 5 seconds EXT3 FS on md7, internal journal EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. Restore the contents of /usr from the copy we made in /var, and nuke the redundant copy: finally unmount the new, filled /usr rescue:~# rsync -a /opt/usr/* /mnt rescue:~# rm -rf /opt/usr rescue:~# umount /mnt Do the same for /home rescue:~# mkfs.ext3 /dev/md6 rescue:~# mount /dev/md6 /mnt rescue:~# rsync -a /opt/home/* /mnt rescue:~# rm -rf /opt/home rescue:~# umount /mnt Alright, xfs is GONE, everything is ext3 and has its old content back in it :-) Let's get our / filesystem mounted at /mnt now and update our fstab to record the demise of xfs rescue:~# mount /dev/md1 /mnt and edit /mnt/etc/fstab (notice the /mnt!) to reflect that we got rid of xfs on /usr and /home and replaced it with ext3 (from -> to again) /dev/md5 /usr xfs defaults 0 2 /dev/md5 /usr ext3 defaults 0 2 /dev/md6 /home xfs defaults,userquota 0 2 /dev/md6 /home ext3 defaults 0 2 Alright, the filesystem messing is done.

Goodbye lilo

Next job is to expunge lilo and replace it with grub. rescue:~# grub-install --no-floppy /dev/sda rescue:~# grub-install --no-floppy /dev/sdb You need to change the bogus /boot/grub/grub.conf that 1&1 mislead you with, into this (change the kernel and initrd version number to what you actually have from your f7 kernel in /mnt/boot ). rescue:~# vi /mnt/boot/grub/grub.conf serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1 terminal --timeout=5 console serial default=0 timeout=10 title Normal Fedora root (hd0,0) kernel /boot/vmlinuz-2.6.22.2-42.fc6 ro root=/dev/md1 console=ttyS0,57600 panic=30 initrd /boot/initrd-2.6.22.2-42.fc6.img

Reboot into F7 with F7 kernel

Finally we should umount and reboot into your new Fedora kernel rescue:~# umount /mnt rescue:~# umount /opt rescue:~# reboot On reboot you should see a grub menu on the serial console which will timeout and boot you (all being well).

Adding selinux

The bogus FC4 install from 1&1 did not include any selinux. This is a pretty bad omission, and we can fix it now. Edit /etc/sysconfig/selinux and set initially to be SELINUX=permissive Then # touch /.autorelabel # reboot This will cause the initscripts to relabel all your filesystems according to your Fedora 7 policy. It will reboot automatically after doing this, when it comes back up selinux will be working in a "firing blanks" mode. It just reports any errors and lets the access occur anyway. You can judge from this what will break when you enable it properly. In my case there were three areas that were broken, first one user has ~/public_html, it needed to be enabled in selinux and then marked up as okay to serve by httpd # setsebool -P httpd_enable_homedirs 1 # chcon -R -t httpd_sys_content_t /home/user/public_html Second for some reason named couldn't start because it wasn't allowed to write its own pid in the chroot, I worked around it with this # setsebool -P named_write_master_zones on The third problem was gitweb, I am asking about it on the selinux mailing list and will update when I have a resolution. UPDATE 2007-09-05: No response from the fedora-selinux ml, I resolved it myself as described here. When you are sure that any remaining avcs (you can find them in /var/log/messages) are trivial or there are no more being generated, you can properly turn selinux on by editing /etc/sysconfig/selinux again and this time setting SELINUX=enforcing and rebooting.

Conclusion

Hopefully I recorded everything that was needed to convert the craptasic bogus unsafe FC4 install in 1&1 servers to clean and true Fedora 7. Certainly being able to yum update kernels as usual is a major step forward, getting you 2.6.22 (in fact I installed a Fedora development repo kernel which is 2.6.23-rc5) from the original non-Fedora 2.6.16. And it's crazy to not have selinux when it's provided by normal Fedora.