• Multiple disk snapshots
• VM State Migration (VMotion – moving a running VM between hosts with no downtime)
• >4GB RAM allocation per host OS
• iSCSI initiator support within the VM technology – such that a host OS can access an iSCSI target without a software iSCSI initiator.

• VMWare
• VirtualBox
• KVM
• Xen

So after the testing it looks like iSCSI may well be the way to go. I spoke to Stuart at M-Tec about it and he didn’t seem to think that iSCSI was going to be a problem performance wise with VMWare. He’s sending over a quote for the basic paid for VMWare package with a list of the benefits over the free VMWare Server 2.0 which is in beta now – but should be in production by the time we go live with this.

Next problem is then to spec the hardware. In an ideal world I’d like to stay totally HP as their kit is reasonably priced and to date has been extreemely reliable – however they’ve discontinued the DL320s which I had thought would make an ideal iSCSI target with the addition of some RAM and either Ubuntu or OpenFiler. There is no direct replacement yet for the DL320 which means I’d have to go for a server using the 2.5 SFF drives – which are far more expensive, especially at larger capacities.

HP do a StorageWorks box that might fit the bill – the MSA2000i series enclosures. They’re the same size and density as the DL320 but work out more expensive. One would hope however that being purpose built for the job they would be faster than the DL320s.

Dell offer the AX150i which is a rebadged EMC enclosure. This is particularly tempting because EMC are a big name when it comes to SANs – and their kit ought to be up to par. I’ve not got a quote for one of these yet because the thought of Dell technical support based on past experience makes me shudder. There’s also the obsticle of actually buying one from Dell Education Sales which makes it an even more unlikely proposition!

Infortrend do an enclosure that would fit the bill nicely. I wasn’t that impressed with the build quality of the Infortrend fibre SAN I put in at BHCC though – and I understand Ian has been having hassle getting support on the unit when one of them failed recently. With that in mind I’ll only really consider Infortrend as a last resort.

The final option is to buy a server from another vendor – IBM or Dell – who do still support LFF drives in a rack configuration and proceed as planned, or source a NOS DL320s via eBay or the like.

And all that before we even consider what VM software to run. I’ve been assuming VMWare but there’s others to be considered…

Further to yesterday’s work with iSCSI, I plugged my iSCSI target directly in to the workstation using a crossover cable giving uncontended 1GB network between the hosts.

I then ran DISKSPEED again and here are the results:
C:\>disksped.exe 512 i:\
DISKSPEED (C) Alexander Grigoriev, alegr@aha.ru
Test File: “i:\$$test$$.tst”
Test File Size: 512 MB
Testing Uncached New File Write Speed….
Data Transfer: 23.60 MB/s, CPU Load: 11.2%
Testing Uncached Write Speed….
Data Transfer: 32.38 MB/s, CPU Load: 12.7%
Testing Uncached Read Speed….
Data Transfer: 69.73 MB/s, CPU Load: 22.0%
Testing Cached Write Speed….
Data Transfer: 36.78 MB/s, CPU Load: 15.3%
Testing Cached Read Speed….
Data Transfer:1261.09 MB/s, CPU Load:  1.5%

C:\>disksped.exe 512 i:\
DISKSPEED (C) Alexander Grigoriev, alegr@aha.ru
Test File: “i:\$$test$$.tst”
Test File Size: 512 MB
Testing Uncached New File Write Speed….
Data Transfer: 22.97 MB/s, CPU Load: -1.0%
Testing Uncached Write Speed….
Data Transfer: 31.27 MB/s, CPU Load: -1.1%
Testing Uncached Read Speed….
Data Transfer: 69.57 MB/s, CPU Load: 19.8%
Testing Cached Write Speed….
Data Transfer: 36.49 MB/s, CPU Load:  2.1%
Testing Cached Read Speed….
Data Transfer:1213.28 MB/s, CPU Load:  1.5%

It seems therefore that iSCSI over 1Gb LAN is capable of keeping up with direct attached SATA – and in some instances such as Cached Reads able to significantly out perform it.

There was also a significant increase in CPU load on the initiator when running at gigabit speeds – but still in line with UDMA utilization so I’m not too concerned.

The target saw its load average reach 1 during both 100Mb and 1Gb tests, however at virtually 0 CPU loading indicating the load average is being affected by time the CPU is IO bound waiting for disks. Certainly in the case of the 1Gb ethernet, it looks like we have hit the limit of the SATA disks or controller rather than the LAN.

It would be interesting to run the tests again on an iSCSI target using SCSI U320 drives to see where the limit comes there.

It seems HP have discontinued the DL320s quite recently so I’m stumped now as to what would be the best disk platform to use. Perhaps something from IBM or Dell would fit the bill better? I shudder at the thought of Dell support though!

I was surprised how easy it was to get all this set up. I broadly followed the instructions here.

Firstly I installed Ubuntu Hardy 8.04 LTS server on one of the PCs. It already has iSCSI support built in to the kernel. During the setup I made a / partition, some swap and an LVM partition which I then split in to 1 volume group (vg1) with two logical volumes in it (homes and vmware).

Next I needed to install the services for iSCSI – apt-get install iscsitarget iscsitarget-source – and then edit the config file to share a block device.

I exported /dev/vg1/homes and /dev/vg1/vmware as iqn.2008-05.uk.sch.brighton-hove.longhill:homes.iscsi and iqn.2008-05.uk.sch.brighton-hove.longhill:vmware.iscsi respectively.

Next I downloaded and installed the Windows XP iSCSI initiator from Microsoft’s website. I installed it and within 30 seconds I’d got the VMware block device mounted and formatting as NTFS. At this point, the PC I’m using is my normal desktop PC and is separated from the iSCSI target by our normal LAN – the bottleneck being the 100MB link to the desktops at both ends. That said, for normal fileIO operations it seems to be acceptably fast – the only time it’s really shown up is on big transfers.

I downloaded a copy of DISKSPEED from their website and ran it as follows:

The options there create a 512MB file on I:\ (where I have the VMware iSCSI target mounted). I was surprised how high the CPU load was on initial creation so I ran it again:

This seems a lot more consistent – perhaps there was some kind of background task going on when I did the initial test. The DISKSPEED website recons UDMA transfers use up to 10% CPU time and that anything less than that is pretty good. I was worried we might see more like 40% CPU load similar in performance to PIO mode.

For comparison, I ran the same test on the C: drive of my desktop PC (which is identical in specification to the iSCSI target PC).

So clearly writing to local disk is less CPU intensive – but interestingly we only see about a 50-60% drop in performance with iSCSI – bearing in mind we’re using a 100Mb LAN and that traffic is being routed between subnets I don’t think that’s an unreasonable performance hit.

Next I’ll try linking the two PCs with a crossover cable at 1000Mb with jumbo frames enabled and see what the difference is.

I also found OpenFiler which is a small rPath Linux distro designed specifically to serve files – and has the ability to serve as an iSCSI target. This may work out to be the best way of doing this on a long term basis.