VM Fail

Work today was full of fail.

It started out with a push-update to my Windows VM from IT. Simple enough, right? Well, this update somehow pushed my VM over the edge of hard drive usage and caused it to start reporting low space errors (the VM thinks it’s a 130GB drive, but it’s located on a 60GB partition, and it had eaten up around 59.7 GB). This meant I could no longer run automated tests in it, which was my primary work for today. Ok, fine, spend some time cleaning up the VM.

This, unfortunately, isn’t enough… since the VM HD files are in 2GB chunks, cleaning up stuff inside the VM and reducing the used HD space down to 26GB doesn’t reduce the VM files’ sizes any. So I attempt a defrag. The available space on the drive very quickly goes down from about 300MB to 110MB, at which point I decide this is a very bad idea and stop.

Spend the morning finding tools to resize the physical drive partitions to give the VM some more room (as it is now running in the 110MB of free space and is generally unusable). Finally fall back to Gparted and discover that the partitions are actually logical volumes, so I have to resize them within the OS. Great.

Spend most of the afternoon figuring out how to boot Ubuntu into a command line as root so I can unmount the appropriate drives to reduce the logical volume size of /home so I can increase the logical volume size of /. (Issues mainly stemming from the fact that all of the physical drive space was already allocated between the two logical volumes, so I needed to reduce one to increase the other, and reductions cannot be done while the partition is mounted.) Discover that I first need to reduce the file system size. Finally succeed at doing so and at giving / an additional 2GB of memory. This was around 4 PM. Success!

Finally get the VM booting back up, and it’s running mostly ok. I then stupidly decide to resize the HD size within Windows so it doesn’t eventually use up the newly allocated 2GB and die again (which is something most of the online conversations on VM sizes I’m finding recommend anyway). This kicks off and starts eating up the newly-allocated 2GB. Eventually, it has 0KB left on its partition, signals an abort, and kills the VM in the middle of the process.

So yeah, now I have a VM in an unbootable state on a partition with 0KB available space (so it can’t boot anyway). Awesome.
I filed a ticket to IT to either get a new VM or get help in adding a VM I have to the corporate domain, but that won’t happen until Monday. Blah. Unfortunately, as there was no VM backup and it was an image of my old Windows workstation, everything I had on there is lost. Oh well.

So yeah, I essentially got nothing done at work today because of stupid VM issues. As soon as I get a new VM working, I’m going to snapshot it and set it to revert to snapshot every shutdown.

Edit: For my own future reference, the relevant commands were:
umount /home
resize2fs /dev/mainvg/home 19609750
lvreduce -L -2G /dev/mainvg/home
mount /home
lvresize -L +2G /dev/mainvg/root
resize2fs /dev/mainvg/root somenumber

3 thoughts on “VM Fail

  1. Ouch. VM’s and diskspace are fun. I don’t know about your particular situation, but I’ve got a couple (hopefully) generally helpful thoughts:

    – I’m guessing “logical drives” means lvm; if so, many liveCD’s (I know Gentoo, and probably Debian/Ubuntu as well) have all the Lvm tools, so you can just pop in the CD/DVD, and not have to worry about unmounting since they were never mounted in the first place.

    – This may not help you at the moment, but I suspect that the issue with not reducing the VM size is also due to the fact that deleting stuff generally leaves data there, since actually zero’ing it out is expensive. Secure delete (e.g., sdelete.exe) is handy for actually zero’ing them out, which lets the VM store the “page” as a zero page instead of stale data.

    – Depending on your VM, you may be able to mount it directly without the VM; e.g., qemu can work off of raw disks, which can be loop-mounted as normal disks. A quick search for VMware turned up vmware-mount.pl. At the very least, this might let you get files off; at best, you might be able to delete some stuff, zero the remainder, and somehow convince the VM to compact the image.

  2. Sorry to hear about your VM fail! Coincidentally, most of my last week or so has been full of banging my head on various desks due to virtual machine-related issues.
    Dynamically resizing volumes (virtual hard disks that don’t allocate the entirety of their space from the get-go) is just asking for trouble. Performance on dynamic volumes is so much worse, reliability issues like the ones that triggered your whole problem are bound to happen… the list goes on and on. Regardless of your virtual server environment, you’ll save yourself a lot of stress by putting your efforts towards securing a decent amount of hard disk space and allocating it all right away.

  3. …also, this may or may not be useful in the future, but I’ll keep blabbing just in case.
    -Splitting disk image sizes in the host OS is also sometimes asking for trouble. The best thing you can do is use a host filesystem that allows for huge files (ext3/ext4/ntfs/hfs+ all work) and have it as defragmented as possible before throwing it on. You want to get your VM’s disk file as contiguous as possible. But even if it’s a little fragmented, the most important thing is that you have one image file instead of a bunch of split up ones that your host’s filesystem is probably not treating properly.
    I’ll shut up now ;-)

Comments are closed.