Putting root and /boot on LVM on RAID

Posted on by

As a holdover from its original install on Fedora 14 or 15, my server had the main system partitions (/, /boot, swap) on md raid across 2 80GB disks (sda and sdk) while all my media is on LVM backed by RAID5. In a fit of madness, I recently decided to switch those system partitions to LVM as well, also backed by RAID5 (with an additional 80GB drive [sde]). Amazingly this worked without a hitch (and only 2 reboots) but I’m going to document my steps just in case someone else finds them useful. Also, I’ll get to the details later but GRUB2 is required, old GRUB will not work. Another note is that while there is lots of documentation saying that GRUB2 can boot of LVM or MD raid I couldn’t find anywhere that said if you could combine them (LVM on RAID) but as this shows, you can and it works just fine.

First, let me give more details on the original setup. The system partitions in the old fstab looked something like this:

/dev/md0    /     ext4 #the rest of the space
/dev/md1    /boot ext4 #512MB
/dev/md2    /tmp  ext4 #2GB
/dev/md3    swap  swap #4GB

md0 through md2 were RAID1 partitions and md3 was RAID0. While I was doing this I also decided I wanted /home and /var to be their own volumes and /tmp should become a tmpfs like /dev/shm Hopefully that is enough info to make sense of the rest of this.

The first thing I did was create the new LV (logical volume) for /home (called lv_home) on my big fast VG (volume group) that primarily holds my media (called vg_data), then formatted and rsync’d the live /home to the new /home and swapped them out:

lvcreate -L 50G -n lv_home vg_data #create the LV
mkfs.ext4 /dev/vg_data/lv_home #format it
mkdir /mnt/tmp
mount /dev/vg_data/lv_home /mnt/tmp/
rsync -avx /home/ /mnt/tmp/ #sync it with live
#swap it out for live
mv /home{,.orig} && mkdir /home && \
mount /dev/vg_data/lv_home /home/ && rm -rf /home.orig

Now you’ll want to update /etc/fstab to point to the new /home (if you use UUIDs you can get the new one with blkid). I rebooted at this point because I needed to remove a custom kernel that I was running at the time (for iSCSI stuff in the last post) but if it’s not needed for this LVM stuff.

Now before we get into this mess we need to install grub to the disk (sdk) in the system RAID1 that we’ll keep until the end in case things go south, and remove swap completely so that this recovery environment can boot cleanly:

grub2-install /dev/sdk
swapoff /dev/md3 #turn swap off
mdadm --stop /dev/md3 #stop the swap disk
#edit /etc/fstab to remove the swap line
#edit /etc/mdadm.conf to remove /dev/md3
#remove the sda partitions from /dev/md[012]
mdadm --zero-superblock /dev/sda? #wipe all md data from sda

Now we format the two disks that will form our (degraded) RAID5 partition to use as the PV (physical volume) in the new system VG (vg_system), create the new system LVs and sync them with the live partitions. Note that I use GPT partitions instead of MBR, MBR would probably work but I wanted to be consistent with my other disks:

gdisk -l /dev/sda #partition table after formatting
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            6143   2.0 MiB     EF02  BIOS boot partition
   2           10240       156215085   74.5 GiB    FD00  Linux RAID

There are a couple of things to note about that partition table:

  • The 2MB BIOS Boot partition is where grub will install it’s extra modules so that we can use both LVM and RAID
  • I stick a few MBs of space between partitions and at the end of disk to account for the slight sector count differences between disks that are the same “size”
sgdisk -G -R /dev/sde /dev/sda #copy the partition table from sda to sde
partprobe #make sure the new partition tables are visible
mdadm --create /dev/md127 -n 3 -l 5 /dev/sd[ae]2 missing #create the raid5 partition
pvcreate /dev/md127
vgcreate vg_system /dev/md127
lvcreate -L 512M -n lv_boot vg_system
#repeat lvcreate for /, /var, and swap
mkswap /dev/vg_system/lv_swap #create swap space
mkfs.ext4 /dev/vg_system/lv_{root,boot,var} #make new filesystems

Now you’ll need to mount the new root, boot, and var LVs and sync them with their live counterparts (using rsync). After that we can get everything ready to boot from the new LVs:

#setup mount points for chroot
mount /dev/vg_system/lv_root /mnt/tmp
mount /dev/vg_system/lv_boot /mnt/tmp/boot
mount /dev/vg_system/lv_var /mnt/tmp/var
mount -o bind /home /mnt/tmp/home
mount -o bind /dev /mnt/tmp/dev
mount -t proc none /mnt/tmp/proc
chroot /mnt/tmp/ /bin/bash #chroot into the new root

From here, we need to edit /etc/mdadm.conf to add the new RAID5 disk we created and remove the old RAID1 disks we don’t want anymore. You can see what the new entry should look like by running mdadm --detail --scan. After that, fix /etc/fstab to point at the new LV-based partitions, and I added the tmpfs entry for /tmp. Then we can tell grub to load the right modules by adding a line to /etc/default/grub like this:

GRUB_PROLOAD_MODULES="raid mdraid1x lvm"

Order may be important there, I didn’t test to find out. Now we just need to finish a couple things in the chroot and we’ll be ready to reboot onto the new LVs:

grub2-mkconfig -o /boot/grub2/grub.cfg #rebuild the grub config
#you can probably ignore the errors it spits out about LVM
mv /boot/initramfs-$(uname -r).img{,.orig}
dracut /boot/initramfs-$(uname -r).img $(uname -r) #make new initrd
grub2-install /dev/sda

Now exit the chroot, reboot, and hope everything works! If the machine comes back up and is on the new LVs we can go ahead and finish cleaning up after the old partitions:

# use mdadm --stop to stop any of the old arrays that were automatically started
sgdisk -G -R /dev/sdk /dev/sde #give sdk the new partition table
partprobe #make it visible
mdadm /dev/md127 -a /dev/sdk2 #add sdk to the new RAID5 which will start rebuilding
#install grub to the other system disks so any of them can be used if sda fails
grub2-install /dev/sde
grub2-install /dev/sdk

And that’s it, enjoy your new LVM backed system partitions!

SecuROM is the Cancer Killing the PC Gaming Industry

Posted on by

DISCLAIMER: Pretty much everything in this post has probably already been said (and said better) elsewhere on the internet, I just feel like ranting after dealing with DRM for 3 days.

I like gaming. Not as much as I used to, but every now and then I want to fire up a game of Homeworld or Crysis or whatever. What I don’t want to have to do is search through my big binder of discs for the game and hope it still works/isn’t scratched/lost/etc. Clearly a better solution is needed, but I didn’t want to turn to no-cd cracks and other questionably safe/legal methods if possible. I mean, I have the official, store-bought discs, surely I can figure some way to do this… right?

Note: This part is somewhat technical, the tl;dr is that a neat solution to a stupid problem doesn’t work because of DRM. The rough plan when I started out was “use STGT iSCSI target to emulate DVD drive backed by ISO images of the game discs”. Unfortunately, it seems STGT uses non-standard ISO images with some extra metadata added so I wouldn’t be able to do this without making special copies from the real physical disc to the special ISO images mounted in STGT which didn’t seem like a great idea. So the next idea was to emulate a real DVD drive on the target, then use STGT’s pass through to deliver that to the initiator. CDemu worked to emulate the DVD drive (and is a pretty neat piece of software) although it was somewhat of a pain to install on Fedora. Unfortunately it seems STGT’s pass through capability is either incompatible with the MS WIndows initiator or just incomplete or broken as this never quite worked. Windows would hang whenever the pass through drive was attached and tgtd would spew errors on the server. The last thing I could think of was to try a different target implementation so I switched to SCST (IET seemed to be too old to work from the last time I was looking at iSCSI targets and LIO seemed extremely complicated). SCST configuration documentation is lacking but I eventually got it working and Windows saw the emulated drive, things seemed to be working great. Unfortunately (I am using that word a lot in this post), I ran into a brick wall called SecuROM. After many hours of trying different things, I got Crysis (and only Crysis) to work with the iSCSI drive using SecuROM’s diagnostic tool. None of the other SecuROM-protected games worked and as far as I can tell, there is no way to get them working without either getting a full no-cd patch or using a program to hide the non-iSCSI drives from SecuROM. I’m not interested in either solution so I guess I’ll just be sticking to the original discs for those games. After wasting about 3 days trying to setup a work-around for the retarded fallout of paranoid publishing companies (and failing), all I can say is “FUCK YOU SECUROM”.

Of course, this is really the fault of the companies that insist on ineffective strategies to try and turn pirates into paying customers. Right now, you can ignore the obvious free/non-free difference and the pirates are still offering the better product. Let me say that again because it is truly mind-boggling: the pirated game could cost more than retail and still be the better product. No DRM, easy to use, no worries about lost/damaged discs, and you don’t even have to get off your ass to get it. This is why Steam is destroying it’s competition; they fucking get it. Ease of use, low price, reliability, un-obtrusive DRM, it’s all there with Steam. If they started adding music/movies/TV episodes and seasons with a sane pricing model and the same kind of sales that the games get, well… I don’t know how many olympic-sized swimming pools GabeN would be able to fill with money, but I would wager more than a few. Now that’s not to say that Steam can do no wrong, just that they do so much more right than everyone else. I wish Steam would go back and start adding older titles to their catalog (like Homeworld, Black & White, etc). Their sales can also get kind of stale (“Oh boy, $5 off that $60 game for the 8th time in a row!”) or just suck sometimes (“50% off all Train Simulator DLC, who gives a shit!?”), but those are really just minor issues.

At the end of the day, the fact of the matter is that I (as a paying customer) have been frustrated by DRM and pirates that got the game for free have a better experience. Well fuck that, if the companies are going to treat their customers like shit, I just won’t buy (or play) their games anymore. Naturally, the sales loss from decisions like mine will be attributed to piracy, and you know what? They’d be completely correct, the pirates are eating their fucking lunch because they offer better service.

Steam on a Network Share 2: Electric Boogaloo

Posted on by

Unfortunately, the NTFS junctions in the previous post didn’t actually end up working that well. I ran into more of the same issues with the Steam buildbot failing on games with install processes. After spending some time looking at other solutions, I decided to give iSCSI a try. I knew a little about it but had avoided it previously since it can’t (easily) be shared between multiple machines at the same time and it probably wouldn’t work with Wine at all, since Wine has issues with NTFS filesystems, but nothing else I could find was better.

Fortunately iSCSI works out very well! Setting it up on the server side was as easy as creating a new logical volume with:

lvcreate -L ${SIZE}G -n $NAME $VOLUME_GROUP

Then installing and starting the iSCSI target (server) tools:

yum install -y scsi-target-utils
chkconfig tgtd on
service tgtd start

A quick note about the scsi target utils: it seems that this particular implementation is relatively new and was only added to the mainline kernel in 3.0 or 3.1 so if you were using an earlier kernel you’d need to use the set of patches or something. There are also a couple other iscsi target implementations but I didn’t have much luck with getting them installed since they seemed to be limited to pre-3.0 kernels.

Now we can create our LUN with:

tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b $LOGICAL_VOLUME
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

Note that this gives full access to the device to anyone that connects to it so limiting access with firewall rules is probably a good idea. There is some authentication support but I didn’t try it. Of course, you don’t have to use a LV, you can use any old block device and I think it would even work with just a regular file.

At this point you can switch to Windows and connect using Microsoft’s iSCSI Initiator (client) by going to Control Panel > Administrative Tools > iSCSI Initiator . Then just enter the target’s IP address/hostname and hit connect. The new disk then acts like a locally connected drive and you can partition, format, and assign a drive letter.

Since it acts just like a regular local disk, Steam and everything else seems to work fine. Initially performance was pretty terrible for me (like 10-20MB/s read speeds) but I tracked that down to a hardware issue on the server and once it was fixed I was able to saturate the gigE connection (~100MB/s).