Backup KVM Virtual Machines on OCFS2 Filesystems on an OpenFiler SAN Appliance

April 13th, 2010 by Alex 2 comments »

We’re using a large OCFS2 partition shared across multiple hosts to provide shared storage for our KVM virtual machines.  We chose OCFS2 because it has a relatively easy setup when compared to GFS (especially since we use Ubuntu as the host OS). The main drawback however is you can’t mount a snapshot of an OCFS2 volume on the same host as the machine that has the live filesystem mounted.

For a long time now we’ve been backing up our OpenFiler-based SAN by taking snapshots using the OpenFiler web interface, and then on a separate machine using a combination of dd and ssh to take a block-level image of the SAN and archive it off for possible disaster recovery.

It works quite well but restoring anything from that image is a total pain. I compress the image using gzip too so firstly I have to find enough disk space on a different machine to extract the archive and then a significant amount of hassle to get OCFS2 setup on a standalone PC and to mount the file image. It follows then that recovering one VM from the image can take days to complete – which isn’t very satisfactory.

Another option would be to share the snapshot devices on the SAN as a new target and then mount them remotely on a server outside the cluster. That’s ideal, except that when you modify iSCSI targets or luns on OpenFiler it reloads the ietd daemon which has a nasty habit of causing the two Ubuntu hosts to stop seeing the target and taking the disk offline.

After some work I discovered you can manually tell iet to start sharing a new target or lun from the command line without needing to restart the service.

Here then is the command history I used:

  1. Take a manual snapshot on OpenFiler (I called it “backup”)
  2. From the command line on the OpenFiler appliance, add a new iSCSI target: ietadm –op new –tid=0 –params
  3. Find the target ID for the new target we just created. There’s a list in the /proc/net/iet/volume file. In my case I got tid:3
  4. Add the snapshot to the target. I’m using blockio with my iscsi targets. The default is fileio:  ietadm –op new –tid=3 –lun=0 –params Type=blockio,Path=/dev/vm/of.snapshot.vm1.backup
  5. Check the volume file again. You should see the export complete with the path to the snapshot.
  6. Connect to, mount and backup the snapshot filesystem using a different PC.
  7. Unmount the filesystem and disconnect from the iSCSI target
  8. Remove the lun: ietadm –op delete –tid=3 –lun=0
  9. Remove the target: ietadm –op delete –tid=3
  10. Remove the snapsh

You can now connect to the iSCSI target from a remote machine, mount the OCFS2 filesystem as needed and backup the files from the filesystem. There’s no guarantee that the images will be consistent but my experience is the machines normally come up fine after an fsck. Any lost data can be restored from the nightly backups of the machines that we take using BackupPC.

My next move is to script all this so that the OpenFiler appliance can kick off a full backup on a schedule.


Here’s the finished script:


# Create a snapshot
echo Creating Backup Snapshot
lvcreate -L100000M -s -n backup /dev/vm/vm1

# Export the snapshot
echo Exporting Snapshot over iSCSI
ietadm –op new –tid=3 –params
ietadm –op new –tid=3 –lun=0 –params Type=blockio,Path=/dev/vm/backup

# Sleep a few seconds to make sure iet is sharing the device
sleep 2

# Connect to nas-backup and connect to the share
echo Connecting to iSCSI target
ssh root@backup iscsiadm -m discovery -t sendtargets -p
ssh root@backup iscsiadm -m node –targetname –portal –login
sleep 30

# Mount the share
echo Mounting filesystem
ssh root@backup mount /dev/sde1 /mnt/vm

# Backup
echo Backup Starting
read -p “Press a key when your backup is complete!”

# Unmount the share
echo Unmounting filesystem
ssh root@backup umount /mnt/vm
sleep 2

# Disconnect
echo Disconnecting from iSCSI share
ssh root@nas-backup iscsiadm -m node –targetname –portal –logout
sleep 10

# Remove the iscsi lun and target
echo Removing iSCSI share
ietadm –op delete –tid=3 –lun=0
ietadm –op delete –tid=3

# Remove the snapshot
echo Removing Snapshot
lvremove -f /dev/vm/backup

# Finished

Microsoft Browser Choice Update – WSUS

March 15th, 2010 by Alex 7 comments »

I got bored waiting for KB976002 to show up in my WSUS install so after a little googling I discovered you can manually sync it in to WSUS.

On the front page of the WSUS console under resources there’s a link to “Microsoft Update Catalog”. If you click through that you’ll be asked to install an ActivX control at which point you can search Microsoft Update for KB976002 and manually sync it on to your WSUS install.

Once it’s synced over, you can then manually approve or decline it as you wish.

Apparently it won’t be showing in WSUS automatically until April.

Copying a file to multiple destinations

March 10th, 2010 by Alex No comments »

Today we needed to copy a file to a whole bunch of users home folders, and fix the permissions after copying.

Here’s what I came up with:

ls -A | xargs -n 1 -IUSERNAME cp source_file.txt ~USERNAME/eportfolio
ls -A | xargs -n 1 -IUSERNAME chown USERNAME.staff ~USERNAME/eportfolio/source_file.txt
ls -A | xargs -n 1 -IUSERNAME chmod 664 ~USERNAME/eportfolio/source_file.txt

The cwd was the directory containing the home directories for the users I wanted to copy the file to.

Declining the Microsoft Browser Choice Update

March 1st, 2010 by Alex No comments »

1st of March brings the “Browser Choice” screen to all XP SP2 or later users via Windows Update, or in my works case via WSUS.

If you don’t want your users to be presented with a choice of browsers to run, (because either they don’t have permission to install software or because you already give your users a choice of browsers – as we do) then there’s a couple of options:

  • Decline update KB976002 in WSUS (you might want to set your WSUS server not to auto-approve updates until you’ve seen it arrive and have declined it).
  • Add a registry key to disable showing the balance screen after the update installs.

The update hasn’t appeared on our WSUS server yet – in fact I can’t find anyone on the Edugeek forums who have sucessfully located the update in WSUS yet.

Converting Windows VMWare machines to KVM

February 24th, 2010 by Alex 5 comments »

I had to do some work on a Windows 2003 virtual server running on VMWare this week which is running on our old playground VMWare install (Server 1.0.4!) and it was sooooo slow I decided enough was enough and it was time to move it on to a sensible platform.

It actually turned out to be quite easy to convert from one platform to the other, with a bit of help from my old boss James Lidderdale.

  1. Shutdown the VMWare server and take a full backup
  2. Boot the VMWare server and uninstall VMWare tools
  3. Apply mergeide.reg registry patch to enable windows to boot on KVM virtual hardware. I’ve no idea where this came from. James had it. I’d like to credit the original author though! Rename .txt to .reg and then merge in as normal.
  4. Shutdown the VMWare server
  5. Convert the vmdk disk image to a single pre-allocated monolithic image:
    • vmware-vdiskmanager -r Server.vmdk -t 2 /some/other/folder/Server.vmdk
  6. Copy the resulting Server-flat.vmdk image over to your KVM server
  7. Now optionally convert the disk to a qcow2 file:
    • qemu-img convert Server-flat.vmdk -O qcow2 Server.qcow2
  8. Finally create a suitable virtual machine definition using that file as the main hard drive. If all went well you should see your VMWare machine boot inside KVM.

The first time I tried this it failed miserably. Turned out that the VMWare machine I was working with had snapshots associated with it. In my case, the disk I needed to flatten with vmware-vdiskmanager was actually Server-000001.vmdk. Once I figured that out it worked first time.

Why I unsubscribed Audible

January 28th, 2010 by Alex 2 comments »

Because they don’t support Android. Other than that I was a pretty happy customer.

School Christmas Dinner Photo

December 18th, 2009 by Alex No comments »

There’s been a literally a submission for the “best school Christmas dinner” crown:

My Christmas School Dinner, Longhill High School

My Christmas School Dinner, Longhill High School

So here’s the first, and to date only entry. From me! Longhill High School by Innovate Ltd. I can honestly say it’s the best school Christmas dinner I’ve had in recent times.

Turkey roll, stuffing ball,  roast and boiled potatoes, carrots, sprouts and parsnips with gravy. Pudding was Christmas pudding and custard.

Well worth a mention were the parsnips. They were so sweet – absolutely delcious. Apparently they’re cooked in oil and honey which explains it.

If you’ve taken a pic of your Christmas Dinner, send it over along with a short review if you like and I’ll add it on!

Best School Christmas Dinner

December 13th, 2009 by Alex No comments »

At work our canteen is due to be enlarged next year as we can’t seat all the students during a lunchbreak.

So next week we’ve got a Christmas dinner marathon – with the festive meal three days on the trott – and I was wondering how these three dinners will stack up against the school dinners available nationwide.

I therefore call all school staff to forward me a photo of your school chrismas dinner. I’ll stick them up here and we can see who’s getting the best deal!

Converting JVC Hard Disk Camcorder Footage

December 12th, 2009 by Alex 2 comments »

When will manufacturers learn?

JVC seem to record their footage in MOD format – which is some strange MPG variant. Very few video editors will touch that, hence the need to convert it.

Trusty ffmpeg to the rescue:

ffmpeg -i input.mod -b 9191k -aspect 16:9 -s 720×576 -acodec wmav2 output.wmv

Linux Teacher PC Update

December 11th, 2009 by Alex No comments »

Well we’ve spent a little time on this. Seb’s been helping out and has done the majority of the work so far.

We’ve got the following working:

  • “Domain” logins (against the LDAP database)
  • CCTV Software (via Wine)
  • Themed to look like our standard XP workstations (but not to the extent that you can’t tell the difference. We’re still using the standard Gnome menus). This needs further work.
  • Promethean ActivInspire for Linux (full marks Promethean)