Improving IO performance with software iSCSI and vmxnet3 adapter

[Aristizabal]

Hi,

I have been trying to improve the IO performance of software iSCSI when using a virtual SAN within the same ESX server. So suppose I have a virtual appliance that has direct access to high end storage and I want to expose this storage to the same ESX server where the VA lives. I have heard that enabling jumbo frames helps, but how about if I create a vmxnet3 adapter on the VA (which in theory gives me 10 Gbps) and since the vmkernel port group and the VA are on the same vSwitch I can get speeds over 1 Gbps.

I have configured vmxnet3 on 2 Ubuntu 32 bit VMs and having them on the same vSwitch I get high transfer rates between them. Using iperf I can get up to 4.2 Gbps compared to the 1.1 Gbps I get when I use regular vmxnet adapter.

So I am not sure if I should expect the same performance when I use software iSCSI against a vSAN Virtual appliance with a 10 Gbps vNic when the vmkernel and the network adapter are on the same vSwitch.

Thanks,

Juan Aristizabal.

[Phokay]

Using vmxnet3 adapter will helps IO performance in most situation. However, even if you can see that NIC speed is 10Gbps insdie the VM, it will not exceed the physical network adapter and switching performance. When transferring data between the VMs that are on the same vSwitch, you will get the speed of memory since the data transfer occurs in ESX server's memory.

Enabling Jumbo Frames not only helps improve IO throughput but also reduces your CPU processing on the host delivering more performance. If you'd like to use jumbo frames, you've got to enable jumbo frames end to end i.e. from vNICs inside the VMs through to ESX's vSwitces, iSCSI pNICs and VMkernel stack, physical switch and the iSCSI ports on the storage.

Hope this helps.

[Aristizabal]

Thanks for your feedback.

So if I use software iSCSI and the vmkernel port group is on the same vSwitch where the virtual SAN exporting the target is, should I expect similar speeds to I get between VMs on the same vSwitch?, which as you explained is the ESX memory speed.

[Phokay]

Even though you communicate between the two VMs in the same vSwitch, the actually bottleneck would be the connections to the SAN device.

[Aristizabal]

Ok, but how about the connection from the iSCSI software initiator to the VM, would it be the bottleneck then??

[KyawH]

In your case, this wouldn't be a bottleneck.

[Aristizabal]

I guess I will have to run some tests with high end storage, but having vmxnet3 on the vSAN appliance and jumbo frames enabled on the ESX will be definitely an improvement for the IO throughput.

Thanks.