vMotion always fails at 21%

[DavideD]

Hello,

at my customer, a new server infrastructure (4 ESXi 6.0U2 hosts, vcenter VCSA, shared storage already used with old servers).

The configuration is:

4 1Gbsp NICs used for VMnetworks

2 10Gbps NICs used only for VMotion

2 hosts in Production Site (host1a, host2a) and 2 hosts in the DR Site (host1b, host2b) (about 200mt. between them)

All is working as espected except the vMotion, that:

Is full woking

from host1a to all other hosts

from host2a to host1b and host2b

from host1b to host2a and host2b

from host2b to host2a and host1b

Is NOT working:

from host2a, host1b, host2b to host1a

vmkping is working from and to all hosts.

The error message is the following

****************

Failed waiting for data. Error 195887371. The ESX hosts failed to connect over the VMotion network.

vMotion migration [168435367:1473922772312347] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout

Migration to host <192.168.8.157> failed with error Connection closed by remote host, possibly due to timeout (195887167).

Migration [168435367:1473922772312347] failed to connect to remote host <10.10.30.167> from host <192.168.8.157>: Timeout.

vMotion migration [168435367:1473922772312347] failed to create a connection with remote host <10.10.30.167>: The ESX hosts failed to connect over the VMotion network

****************

Does anyone can help me to find where is the problem ?

Thanks in advance

Davide

[Texiwill]

Hello,

Let us go back to basics...

You have 10GB for vMotion with 1 GB for VMs. Actually I would switch that myself, unless you also are planning on using the vMotion network hardware for SMP-FT or VSAN.... Something to consider.

That aside, I would look at your switches, in the past, switches either had to have their ports hardcoded to a speed or set to autonegotiate. If they were not set properly, the vMotion would stall or fail outright.  Look at the switches for any errors associated with the ports to which vMotion is configured. Also, ensure any VLANs are trunked to those ports correctly. I would (a) clear any errors in the switch (b) verify VLANs trunked porperly (c) run a vMotion (d) look at switch error counts.

From the ESXi side of the shop, there is not much you can do actually. It is either connected or it is not, not much tuning needed there. However, that being said, ensure all the cables are plugged in properly (unplug, replug is my suggestion), look for any network card errors in the vSphere Log files (LogInsight is extremely helpful for this)... if there are errors fix those if possible.

Ensure that ONLY ESXi hosts are connected to the vMotion network VLAN/switch. If anything else is connected it can contribute to failure. This is one of the sacrosanct networks.

Note, most of vMotion failures in the past are due to physical mis-configurations not due to software issues unless it is in the switch itself.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009-2016

Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.

Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast

[DavideD]

Hello Texiwill,

thank you for your detailed explanation.

On switch side all is ok, otherwise also the other three hosts wold not work.

Not having more time to search for the problem, I reinstalled ESXi on the server and now vmotion is working also on this host.

Thank you

Davidde

[feelingcool]

I have met the same problem like you.

From my case, it is not the physical switches's config issue.

I checked all my ESXi 6.0 Hosts, found one machine which can not vmotion is due to vMotion function is also enabled on Management network.

Once unchecked vmotion on Management Network as below screenshot, vmotion is back to normal.