iSCSI over 10GB

[dailowai]

Hi,

I'm running iSCSI over a single 10GbE link for testing.  I have a LUN presented from some fast storage, but I'm only seeing rougly 200mb/s of bandwidth max.  I'd expect to be able to get much higher and I know the storage isn't the bottleneck.  From a linux machine over iSCSI I'm able to push well over 800 mb/s.  Anyone have any tips on where to start looking for the bottleneck?

I'm testing the performance by generating a workload on a VM that is sitting on top of the iSCSI datastore.  If I fire up another VM and run the same workload the performance just decrases evenly across the two VMs.

[rickardnobel]

dailowai wrote:

I'm running iSCSI over a single 10GbE link for testing.  I have a LUN presented from some fast storage, but I'm only seeing rougly 200mb/s of bandwidth max.  I'd expect to be able to get much higher and I know the storage isn't the bottleneck.  From a linux machine over iSCSI I'm able to push well over 800 mb/s.

A question on your numbers: when you say you get 200 or 800 mb/s of bandwidth, is that 800 Mbit (Mb) or 800 MByte (MB) per second? If it is megabit as you wrote than this is just 1 Gbit speed.

[dailowai]

I was referring to MB, but I found the problem is with my NICs I think.  Having a really strange problem.  I put iperf on my ESX server through the service console and I'm only getting 1.7 Gbits (200 MB/s), but if I put the NIC pass through to a VM guest and run iperf I am getting 9.7 Gbits which is what I would expect.  I tuned every parameter I can think of and ESX 4.1 has netqueue enabled by default so I can't think of what could possiblty be slowing it down.

[rickardnobel]

dailowai wrote:

I put iperf on my ESX server through the service console and I'm only getting 1.7 Gbits (200 MB/s), but if I put the NIC pass through to a VM guest and run iperf I am getting 9.7 Gbits

A guess could be that the vSwif virtual nic in the Service Console is not made to work with 10 Gbit/s network at all, but the vNICs for the guests are optimized for this. I really think it is better to do performance testing from the guest perspective.

[dailowai]

That could be the case, but I'm getting the same speeds when I run IOmeter on the VM guest with a disk that is on the iSCSI datastore which is bad.  If I put the NIC through pass through and mount an iSCSI volume directly to the the Linux VM I get triple the speed.

[rickardnobel]

Will wrote:

Anyone have any tips on where to start looking for the bottleneck?

That is strange indeed. If you have access the ESXTOP you could study the DAVG and KAVG values while testing from an ordinary VM with VMkernel iSCSI. The KAVG is the time spent in the VMkernel for each command and the DAVG is the latency for the device (vmhba for iSCSI) to complete the IO request. The KAVG should be very low, almost zero in most cases.