iSCSI performance issues

[7007VM7007]

I am setting up a new vSphere 6 environment with 2 ESXi servers and for the shared storage I am using a Windows Server 2016 box with Starwinds Virtual SAN. Each server has a quad 1Gb NIC in it which is dedicated to iSCSI traffic and is connected to a single Cisco SG300-28 switch. I have a VLAN created for the iSCSI storage traffic and have Round Robin MPIO enabled/configured.

The problem is I'm getting awful storage performance. To troubleshoot this I ended up connecting the 4 x 1Gb NICs in the ESXi server directly into the SAN server. Here are the results:

ESXi server connected to Cisco SG300-28 switch:

Running cables directly between the ESXi server and SAN:

As you can see there is a huge difference in performance! When the ESXi host is directly connected to the SAN I can also see that file copies in the VM and general copy operations work quite quickly but when connected to the switch again drop to well under 80MB/s.

Is there any reason for the poor performance with the Cisco SG300-28 switch? I'm not using jumbo frames but is there anything I need to configure on the switch or in vCenter to get iSCSI performance working like it should? I'm using network port binding for iSCSI.

This is a home lab so there is literally no load on this switch and I only have two VMs running at the moment, vCenter and a test Windows VM.

I am totally baffled as to why the iSCSI performance would be so bad when I have read such good things about this switch being used in home labs!

[NetxRunner]

First of all, I would recommend splitting all iSCSI networks into different subnets like 172.16.1.x/24, 172.16.2.x/24 and so on instead of iSCSI port binding and VLANs since all additional network layers and processing might influence storage traffic performance. These measures might offload your switch too.

Round Robin is fine. I would recommend applying maximum iops=1 adjustment https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069356.

Jumbo frames are totally worth trying since they will significantly reduce the number of packets running through your network and might speed up everything and offload the switch too. Please note that you will need to enable those everywhere in your network meaning VMkernel on the ESX side, on the SAN side and the switch itself.

For proper benchmarking, I usually use a Windows 2012 R2 virtual machine and Diskspd https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223 since this tool gives the most reliable and reproducible testing results. Anyways ATTO should also show good numbers.

[Garry_N]

I would recommend trying to go for Win 2012R2 as the OS for the StarWind Virtual SAN controller VMs, also try Diskspd instead of ATTO as a benchmarking tool.

Another thing you might consider doing is writing a similar post with detailed description of the configuration your have on StarWind forum, their engineers might advise you some configuration changes for optimization.

Here is the link