Clone process and cold migration won't pass 2Gbps on 10Gbps network

[mggonzalez]

Hello everyone

I have a scenario where I have 2 stand-alone ESXi hosts hosting dedicated VMs and every Hardware cycle we need to move the VM to new Hardware. What we do is we connect a patch cable on the back of the 2 servers using 10Gbps Nics (Intel X710) and clone the VM to the new host using that one connection, which is configured with a new vmkernel with Provisioning and vMotion services. iperf3 tests shows we can reach 10Gbps without issues but the cloning (or cold vmotion) process only reaches 2Gbps.

I'm running SSDs at 6Gbps and the RAID controller supports 12Gbps. I would expect a little more than 2Gbps with this setup. Initially was testing with RAID6 but created a small array with RAID0 just to see if it would improve - it didn't.

I also have MTU configured as 9000 on all involved connections at source and target.

I have been on this back and forth with Vmware since 7.0.3. which at the time wouldn't pass 1Gbps. Clearly there was an improvement from 7.0.3 to 8.0.2 (since now i can get 2Gbps with the same HW specs).

Does anyone know if there's a command or anything I can do to customize the hypervisors to bypass this clear limitation of 2Gbps?

[NickDaGeekUK]

ESXi has an inbuilt limitation on VMKernels that VMWare does not document.

see here in the Veeam R&D Forum 

and here in the ESXi community forum which seems to indicate exactly what you are experiencing i.e. a 20% of max bandwidth bottleneck.

I have had some success at bypassing that limit by supplying a seperate vSwitch and VMK with a custom TCP stack on a different IP range to the Management interface. Without that the global default TCP/IP route is via the Management interface IP.

------------------------------
IT Manager. That person just about managing to keep the IT going with no budget or staff despite the effects of Senior Management and End Users.

IT Worker. A biological system for turning caffeine into code.
------------------------------

[mviel]

Hello, 

I would think that an Raid 12 Gigabit Pro Seconds can only achieve the amount you measure over the Network.  So maybe the saturation of the Storage is the Limitation rather then the Network ?  

With Gbit you have to divide by 8 to get the Byte Ratio . 

In a Scenario with VSAN "ESA" ( ESXI 8u2 ) , we can clone or Migrate over the full 100Gbit (!)  Network card , so I would assume that there are no limitation on the VMWARE Kernel Side now . On classic Iscsi we have seen that the full Potential of a Migration or Clone could only be achieved if parallel Operation where used ( multiple VM Clone / Migration ) 

https://en.wikipedia.org/wiki/Serial_Attached_SCSI