ESXi has detected 24ms round-trip vMotion network latency

[lwhitworth]

Hi all,

Whenever I vMotion a VM between hosts I get something along the lines of the following logged:

ESXi has detected 24ms round-trip vMotion network latency between host source_host_ip and  destination_host_ip .  High latency vMotion networks are supported only if both ESXi hosts have been configured for vMotion latency tolerance.

warning

13/09/2012 12:46:23

hostname

Doing a vmkping on the hosts does indeed show that latencies (which typically are sub 1ms) do indeed go up and up during the vMotion and then immediately drop off once it's done (vMotions are still completing fine).

I have tested removing the switch from the configuration and putting a network cable between the two hosts but the problem remained.

Both hosts have two dedicated vMotion NIC ports, and vMotion is split off into it's own vSwitch.  The NICs are auto-negotiate and are running at 1000 full.

Anyone got any ideas why I'm seeing this warning logged?

Cheers,

Luke

[lwhitworth]

Forgot to say, jumbo frames is enabled on all vMotion NICS, vswitches, etc.

[ccslancaster]

Have noticed exactly the same warnings on our cluster - which seemed to have crept in at some point during various upgrades from esx -> esxi or 3.x to 4.x to 5.x. We're currently on 5.0U1.

Scenario is exactly as you describe - latency is low when testing vmkping between vmotion kernel ports but jumps up during vmotions (especially bulk 4-way vmotions of vms when putting a host in maintenance mode). Various values between 15 ms up to 30ms.

Are you using standard or distributed switches (we're using standard)?

Can confirm we're also using jumbo frames on the vmotion kernel port.

If I get a chance I'll investigate removing jumbo frames to see if that fixes things.

[lwhitworth]

We're using standard too.

Nice to know I'm not alone!

[TimSecure]

We are seeing the same warning (perhaps that is all it is?) on our 5.1 (799733) cluster.  vMotion network is not using jumbo frames.  Cisco switch and Dell switch produced the same warning so a switch issue is unlikely.  Network adapter used for vmotion is an Intel 82580.  Host network settings are about as simple as they can be, two NICs, one vSwitch, one vmkernel/IP.

I will likely open a ticket with VMware and determine if this is simply a warning when multiple vMotion activities are happening (switch into maintenance mode), or it is something to be concerned about.

[nealbarrett]

Does anyone have updated information regarding this?

I am seeing this too... between 6-25ms in the warning messages during an "Enter Maintenance Mode" vMotion event.

I am currently running VMware Enterprise, so no vNetwork Distributed Switches... We are on ESXi 5.0 U1... and discovered these warnings during "Enter Maintenance Mode" to perform the ESXi 5.0 U2 patching... We are not ready for ESXi 5.1 yet.

Our vMotion network is an isolated untagged VLAN (switchport access vlan xx) on a Dell PowerConnect 6248 switch. We are not running jumbo frames on this switch at all... The switch is set for standard MTU of 1500 on all ports, and our vmk's for vMotion are set to the standard 1500 MTU, as well as the standard vSwitch it is connected to.

We have QLogic hardware HBA's for SAN traffic (which are also running 1500 MTU) on a separate physical switch (Dell PowerConnect 5424) so no software iSCSI, vmk's, or vSwitches are involved anywhere in the ESXi network stack. Our management vmk is connected to a separate vSwitch, IP, etc... used to attach to our vCenter. I should also note that the Dell PowerConnect 6248 switch load does not indicate nearing any capacity maximums during the vMotions (CPU, Memory, etc...)

This KB article seems to indicate that this is simply a warning that will not cause any problems, just impact performance of the vMotion... however, it seems cause for concern based on the wording of the warning message... So, I would really like some clarification before I write this off.

My $0.02... seems like a vmk issue... as that seems to be the only point that could be responsible for generating this latency, especially in lwhitworth's situation where a direct cable connection between NICs was performed. Either that or a pNIC driver stack issue... I see that TimSecure is running Intel 82580, but what NICs are everyone else seeing this happen on?

@lwhitworth: I have a 2 host cluster so I can try the direct NIC to NIC connection with crossover cable as well... but since you've already done that, I'd prefer not to repatch things in production. Do you think it would be worth testing this for added confirmation that it is not a switch issue?

@TimSecure: I see you've had the same issue with ESXi 5.1... So it's not just 5.0 U1 or 5.0 U2. Have you hear anything back from support? I am curious if this is a "safe" warning, or if I need to address it to avoid a possible VM corruption during vMotion...