Further to yesterday’s work with iSCSI, I plugged my iSCSI target directly in to the workstation using a crossover cable giving uncontended 1GB network between the hosts.

I then ran DISKSPEED again and here are the results:
C:\>disksped.exe 512 i:\
DISKSPEED (C) Alexander Grigoriev, alegr@aha.ru
Test File: “i:\$$test$$.tst”
Test File Size: 512 MB
Testing Uncached New File Write Speed….
Data Transfer: 23.60 MB/s, CPU Load: 11.2%
Testing Uncached Write Speed….
Data Transfer: 32.38 MB/s, CPU Load: 12.7%
Testing Uncached Read Speed….
Data Transfer: 69.73 MB/s, CPU Load: 22.0%
Testing Cached Write Speed….
Data Transfer: 36.78 MB/s, CPU Load: 15.3%
Testing Cached Read Speed….
Data Transfer:1261.09 MB/s, CPU Load:  1.5%

C:\>disksped.exe 512 i:\
DISKSPEED (C) Alexander Grigoriev, alegr@aha.ru
Test File: “i:\$$test$$.tst”
Test File Size: 512 MB
Testing Uncached New File Write Speed….
Data Transfer: 22.97 MB/s, CPU Load: -1.0%
Testing Uncached Write Speed….
Data Transfer: 31.27 MB/s, CPU Load: -1.1%
Testing Uncached Read Speed….
Data Transfer: 69.57 MB/s, CPU Load: 19.8%
Testing Cached Write Speed….
Data Transfer: 36.49 MB/s, CPU Load:  2.1%
Testing Cached Read Speed….
Data Transfer:1213.28 MB/s, CPU Load:  1.5%

It seems therefore that iSCSI over 1Gb LAN is capable of keeping up with direct attached SATA – and in some instances such as Cached Reads able to significantly out perform it.

There was also a significant increase in CPU load on the initiator when running at gigabit speeds – but still in line with UDMA utilization so I’m not too concerned.

The target saw its load average reach 1 during both 100Mb and 1Gb tests, however at virtually 0 CPU loading indicating the load average is being affected by time the CPU is IO bound waiting for disks. Certainly in the case of the 1Gb ethernet, it looks like we have hit the limit of the SATA disks or controller rather than the LAN.

It would be interesting to run the tests again on an iSCSI target using SCSI U320 drives to see where the limit comes there.

It seems HP have discontinued the DL320s quite recently so I’m stumped now as to what would be the best disk platform to use. Perhaps something from IBM or Dell would fit the bill better? I shudder at the thought of Dell support though!

I was surprised how easy it was to get all this set up. I broadly followed the instructions here.

Firstly I installed Ubuntu Hardy 8.04 LTS server on one of the PCs. It already has iSCSI support built in to the kernel. During the setup I made a / partition, some swap and an LVM partition which I then split in to 1 volume group (vg1) with two logical volumes in it (homes and vmware).

Next I needed to install the services for iSCSI – apt-get install iscsitarget iscsitarget-source – and then edit the config file to share a block device.

I exported /dev/vg1/homes and /dev/vg1/vmware as iqn.2008-05.uk.sch.brighton-hove.longhill:homes.iscsi and iqn.2008-05.uk.sch.brighton-hove.longhill:vmware.iscsi respectively.

Next I downloaded and installed the Windows XP iSCSI initiator from Microsoft’s website. I installed it and within 30 seconds I’d got the VMware block device mounted and formatting as NTFS. At this point, the PC I’m using is my normal desktop PC and is separated from the iSCSI target by our normal LAN – the bottleneck being the 100MB link to the desktops at both ends. That said, for normal fileIO operations it seems to be acceptably fast – the only time it’s really shown up is on big transfers.

I downloaded a copy of DISKSPEED from their website and ran it as follows:

The options there create a 512MB file on I:\ (where I have the VMware iSCSI target mounted). I was surprised how high the CPU load was on initial creation so I ran it again:

This seems a lot more consistent – perhaps there was some kind of background task going on when I did the initial test. The DISKSPEED website recons UDMA transfers use up to 10% CPU time and that anything less than that is pretty good. I was worried we might see more like 40% CPU load similar in performance to PIO mode.

For comparison, I ran the same test on the C: drive of my desktop PC (which is identical in specification to the iSCSI target PC).

So clearly writing to local disk is less CPU intensive – but interestingly we only see about a 50-60% drop in performance with iSCSI – bearing in mind we’re using a 100Mb LAN and that traffic is being routed between subnets I don’t think that’s an unreasonable performance hit.

Next I’ll try linking the two PCs with a crossover cable at 1000Mb with jumbo frames enabled and see what the difference is.

I also found OpenFiler which is a small rPath Linux distro designed specifically to serve files – and has the ability to serve as an iSCSI target. This may work out to be the best way of doing this on a long term basis.

One of the things we’ve been thinking about of late is that we really ought to retire some of our crustiest servers – they are little more than 6 year old PCs in rack cases – and farm their workload on to VMWare versions of themselves. Some run completely bespoke software and others have very specific dependencies in the environment that would be a nightmare to attempt to recreate.

I’m looking therefore at what hardware to host this new system on. I saw a very impressive demo of HP SAN kit at Eastbourne College with ESX server and VMotion – but we don’t have the budget for that kind of kit, so it’s going to have to be something semi-home-brew.

First I looked at getting a DAS (Direct Attached Storage) box like the MSA60 or MSA70 from HP – they work out about £1700.00 plus drives – however for the performance I think it’s going to need we’re looking at using LFF SAS drives in an MSA60 which run at £170.00 each for 146GB and £300.00 each for 300GB meaning we can only really afford about 500GB of useable storage at RAID5.

To go with that, we’d be looking at a Proliant DL360-G5 with twin quad core 3GHz Xeon chips to give us our first eight-way server! We’d have to scrimp on the RAM thought at about 4GB but that’s something we can address easily at a later date. It would need a P800 Smart Array controller fitted to it to connect to the MSA60. Total cost with 500GB SAS storage comes out just over £5000.00 + vat.

Not bad, but it doesn’t scale well. Sure I can add more disc easily enough but the limit will come when that 8 way box runs out of puff – with DAS there’s no way to share that storage with a second server and leverage the investment in the fast drives etc. It also makes some kind of failover configuration much harder than in a SAN environment.

I read an interesting article here suggesting using an HP DL320s storage server (which are approximately the same cost as an MSA60) as an iSCSI target and therefore the basis of an iSCSI SAN. This is a neat solution as it scales in every way – the DL320 with a P800 Smart Array controller fitted can daisychain additional storage chassis from it (eg an MSA60) and since it gives our proposed frontend DL360 access via iSCSI, we should be able to access those enclosures and drives from multiple servers via a dedicated copper gigabit network and a clustered filesystem like GFS or OCFS2.

My main concern there though is throughput to disk. Last time I played with Fibre Channel SANs at BHCC they were 2GB in each direction – 4GB total throughput per link – whereas my DL360 will only have 1 available gigabit copper interface free giving a maximum of 1GB less protocol overheads in any given direction.

I’m therefore going to do a bit of testing on iSCSI with two of our standard desktop PCs – Intel Core2Duo machines running at 1.86GHz with 4MB cache. Onboard Intel gigabit NICs.