Using IOMeter for iSCSI Performance

[Newb449]

Guys,

I'm interested in determining whether we should be using jumbo frames or not in our iSCSI environment. I've read a lot about the use of IOmeter, but haven't managed to find any any information as to actually setup IOmeter for this test.

Can anyone point me in the right direction?

BTW If there is a simpler method to accomplish the same result I'm all ears.

Thanks

[drummonds]

Here are some instructions I wrote up on Iometer a while back:

http://communities.vmware.com/docs/DOC-3961

They will help you run the tool and mimic real workloads. What I expect you will see is that JF show little to no gain on real workloads.

Scott

More information on my blog and on Twitter:

[Newb449]

I saw this post, but it doesn't tell me what would need to be configured in order to test iSCSI performance. (I.e woudl I need to setup a MS iSCSI initiator within the VM)

Having said this, is RParker's reply correct in that you can't use IOmeter to measure iSCI performance?

[drummonds]

Well, iSCSI is a type of storage. So, you want to use a storage test tool to run a workload against an iSCSI store. Iometer is the perfect tool for that.

If you want to test Microsoft's iSCSI initiator, you are probably better off using Microsoft's instructions.

The other response about Iometer being unable to test iSCSI is not correct. We use Iometer at VMware for all of our storage tests: Fibre Channel, iSCSI, and NFS.

Scott

More information on my blog and on Twitter:

[Newb449]

Fair enough. I thought I might have needed to use the MS iSCSI in order for the VM running IOmeter to see the iSCSI storage, as opposed to simply running on the storage. So you are saying a VM can abstract itself and see the iSCSI storage or am I testing the VM performance as it operates within the iSCSI storage? Looking at your doco, when I launch the application and look at my disk targets I only see the OS C partition.

[drummonds]

There are three ways to configure a VM so it can access an iSCSI store:

1. Use an in-guest iSCSI initiator, such as the MS iSCSI one you already know about.

2. Using ESX's software initiator, which will present the storage as a virtual direct-attached SCSI device.

3. Using a hardware initiator, which will present the storage as a virtual direct-attached SCSI device.

I think what you want to do is #2. There are instructions on the somewhat outdated VI3 iSCSI design/deploy document.

More information on my blog and on Twitter:

[Newb449]

That's what I have in place. A ESX software iSCSI initiator with multipath setup as per the ESXi iSCSI configuration guide. My original question still stands though:

How am I meant to gauge performance to determine whether I should enable jubmo frames or not? IOmeter on a VM only see's its local disk, which resides on the iSCSI-based datastore. Do I run IOmenter under this premise to determine performance, or do I need to implement something different so that I can see the performance of ISCSI throughput? If so, what mechanism and what's the metric to observe?

[drummonds]

You have got the right idea. Configure your iSCSI storage, format the LUN for VMFS, put a VMDK on the volume, and attach that VMDK to the VM. Do not format that virtual disk with a guest file system. Just have Iometer test it directly. Follow the instructions in the document I listed. Run the test once, enable JF on the host, switch, and array, and run the test again. Compare.

The reason why you probably will probably not see any difference in performance is that 100% of practical storage configurations are bottlenecked by the spindle count. It will generate the same throughput whether JF is enabled or not, and regardless of protocol (FC, iSCSI, NFS) for that matter. For you to make the server have to sweat a little bit, so that you could see efficiency differences in the different approaches, you need to add a lot of disks. If you are running an Intel Xeon 5500 for instance, you are going to need 1,000 disks or more. Maybe you could see differences in CPU load with fewer disks, but certainly no less than 200.

Scott

More information on my blog and on Twitter:

[Newb449]

OK. So I did as you suggested and hung an unallocated disk. Based on your doc I set the # of Outstanding I/Os to 5 to start off with and set the max disk size to 4000000 Sectors, (as the VM has 512MB RAM). I manually created a SQL type profile (64K block 100% randomness, 66% read) and got 365.29 total IOPS per second. Is this good, bad or par?

I will re-run the test with jumbo frames enabled to see the difference.

Or am I wasting my time given what you said about the number of spindles? I only have 8 SAS disks in a RAID 5 array :smileysad:

[erwinrivera]

Why don't you post your iometer config file here and I will see how that compares with a our san, das storage, nfs and iscsi.. not high spec stuff mind you... all small business gear.

You could also compare this to the openperformance to the http://communities.vmware.com/message/586222 thread from ChristianZ as mentioned, I have seen some result there with low spindle counts.

[Newb449]

Just when I thought I was starting to get a handle on things:…I’m confused again.

These tests operate on the C drive. I thought for iSCSI that I should be testing against the unformatted vmdk associated with VM.

Anyway I went through the excersise; although I wasn’t’ sure about the statement about “Systems with cache size over 2GB the test file should be increased” so I left that at the default of 800000..

For the results I assume “Av Ios/sek is IOPS and Av Mb/sek is MBps

##################################################################################

TEST NAME--

---

Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__17.14633__..........__3508.986_.........__109.6558____

RealLife-60%Rand-65%Read......__49.41024___..........___884.8101_.........__6.912584__

Max Throughput-50%Read..........__12.05425__..........___5052.318__.........____157.885___

Random-8k-70%Read.................___53.97957__.........____877.9359_.........___6.858874___

EXCEPTIONS: CPU Util.-XX%;

Attached are the reults 1-4 for each test, plus the ones I ran against the unallocated vmdk disk.

Thanks a bunch

[Newb449]

Anybody still ou there?

[drummonds]

I would not call it "wasting your time". I would call your exercise "proving a valuable point".

Storage-bound operations tend to be bound by the storage hardware, not the efficiency at the host. The majority of the time storage performance is dictated by spindle count and RAID configuration.

Protocol differenceschoosing between FC and iSCSi or Jumbo Frames versus TSOaffect the efficiency and resultant CPU utilization. Check the CPU utilization on your two tests. It will probably be 3% on one test and 2% with jumbo frames enabled. This efficiency improvement is probably useless on your configuration but would be invaluable for IO-intensive operations.

Scott

More information on my blog and on Twitter:

[Newb449]

I'm learning...

That's the important part :smileyhappy:

[RParker]

Jumbo frames should be used, however not all ethernet switches support the use of Jumbo frames or 9000 MTU sizes. If you can utlize the jumbo frame, moving large blocks of data in larger increments always results in better performance.

IO Meter will not tell you this however... it only measures the OS at the file level not the block level.

[Newb449]

So how do I measure iSCSI performance?

[ac2010]

Hi,

Is it correct that IOMeter is only good for measuring the IOPS and not throughput (MB/sec)? The latest edition of IOMeter that we're using provides both results, so I'm not sure.

rgds,

Anil