Linux Journal | December 2012 | Issue 224 - ACM Digital Library

COLUMNSHACK AND /and e-mail for me and my immediatefamily, a number of different Websites and blogs, and even my mainIrssi session sits on one of those twoVMs. I end up hosting secondaryDNS and e-mail from a server onmy home connection, but due to aone-megabit upstream connection, Idon’t host much else at home for theoutside world.One day (while a relative happenedto be visiting from out of town), Inoticed that both my main server andthe physical server that was hostingit were unavailable. I notified mycontact at the data center, and itended up being an accidental poweroutage that affected my cabinet. Iwas taking my relative out to thecoast for the day, far away fromdecent cell-phone reception. So,since there wasn’t much I could do, Iassumed that long before I got backinto town that afternoon, powerwould be restored, and other thanlosing over a year’s uptime, I wouldbe back up and running.Everything but the SyncThe first time I knew there was a realproblem was when I got back intotown and my main server still wasdown. I could log in to the physicalhost, however; so at first I wasn’t tooworried. After all, I had seen KVMinstances not recover from a physicalhost reboot before. In the past, it waseither from not setting a VM to startat boot or sometimes even a waywardlibvirt apparmor profile that got inthe way. Usually once I logged in tothe physical host, I could change anybad settings, disable any troublesomeapparmor module, then manuallylaunch my VM with virsh. This timewas different.When my VM wouldn’t bootmanually, I was ready to blameAppArmor. It had blocked VMs frombooting in the past, but this time,neither setting the libvirtd AppArmormodule to complain mode, disablingall AppArmor modules nor evenforcefully stopping AppArmor seemedto help. I even resorted to rebootingthe physical host to heed AppArmor’swarning that forcibly stopping itafter it was running may cause somemodules to misbehave. Nothinghelped. When I connected a consoleto the VM as it booted, I startedseeing initial kernel errors as thoughit was having trouble mounting theroot filesystem. Great. Did the poweroutage corrupt my data?The next step in the troubleshootingprocess was to attempt to bootfrom a rescue disk. With KVM, it’srelatively easy to add a local ISOimage as though it were a CD-ROM.WWW.LINUXJOURNAL.COM / DECEMBER 2012 / 43