ping loss vmware tools 11

[Mickeybyte2]

Hi,

We are running server VM's on a VMware Cluster. We are running VDI VM's (Horizon) on another VMWare Cluster running. The clusters are managed by a service provider. All I know is that they are using NSX and Trend Micro  Deep Security on the network.

We have a pool of VDI's running vmware tools 10.3.5, let's call this Pool10

We have another pool of VDI's running vmware tools 11.0.1, let's call this Pool11

We have servers running vmware tools 11.0.1.

We noticed that we have >1% ping loss when pinging between servers or between servers and Pool11

We don't have ping loss when pinging to servers from Pool10

In the release notes of VMWare tools 11.0.5 I read:

Outbound IPv6 traffic for ICMP and UDP protocols could experience packet drops.

NSX Network Introspection Driver is used to retrieve the network context from the Guest VMs.
From VMTools 11.0.0, support has been added for ICMP and UDP protocols. The driver now intercepts the UDP and ICMP traffic, collects the required context, and re-injects the packets back. There was a problem in the packet re-injection code for the ICMP and UDP IPv6 packets for outbound traffic. As a result, outbound IPv6 traffic for ICMP and UDP protocols could experience packet drops.

Note: Network Introspection Driver is not installed by default and can be installed with VMware Tools ‘Complete’ installation for NSX IDFW and NSX Intelligent features only.

This issue is fixed in this release.

We are using IPv4 only (IPv6 is not disabled, but not used)

I've tested the ping problem with VDI Pool11 running 11.0.5 and the issue is still there

I've tested the ping problem with VDI Pool10 running 10.3.5 and there were no ping losses anymore!

Is anybody here aware of a solution other than to revert all our VMWare tools to 10.3.5? (As said, I don't have experience, nor access to the NSX setup, I just know it's there)

Thanks a lot,

Michiel.

[davidi123]

We have the same issue also with Tools 11.0.1/11.0.5 and TMDS -> I have opened a VMWare case

[joking]

We are experiencing the same issue with tools 11.0.1 and have opened a service request with VMware. The ticket was initially assigned to the networking team and we were able to narrow it down to the Guest Introspection Driver in tools 11.0.1 and 11.0.5. We are currently using Trend Micro Deep Security in our environment, which requires using the driver, so removing it is not an option. The current work around to leave the guest introspection driver enabled is to downgrade tools to version 10.3.10. The ticket has been cloned to the NSX team to troubleshoot the issue further, which I have a call scheduled today at 1:00 PM EST.

[davidi123]

so you can say them it's the same problem in case: SR20104757202 (our case number).

if you want you can share youre number so i can told the same to my techie.

thanks

[Mickeybyte2]

Thanks for the update. I would really appreciate it if you could keep me updated on the outcome of the support requests.

As much as I'd like to, I'm not in the ability to open a support request unfortunately...

Regards.

[davidi123]

by the way:

I have another case opened. When I tracer with the new VM Tools and the NSX driver.
I see only the last hop -> because the NSX drivers faulty manipulates the TTL of the traceroute packets.

It sets the default TTL instead of using an increment of 1. therefore traceroute is broken..

[jwl6197]

Also saw this behavior with 11.0.5 and we backed down to VMware Tools 10.3.10. Will be interested in seeing what the outcome of your tickets with VMware are. I was going to enter one in myself after I figured it out but will save myself the headache :smileyhappy:

Please update with any info you all get, much appreciated!

[Jonathan McDowell]

We are also experiencing the same issue. It only seems to happen to windows 10 workstations I haven't had the same loss on any Server 2016 machines.

[ChicaneUK]

Another user experiencing this exact same issue with 11.0.5 tools, on Windows 10 desktops. A colleague found unloading the Network Introspection Driver resolves the issue for us, or alternatively stopping and disabling the vnetWFP service will immediately rectify the issue. I'd be interested to hear the output of any of these tickets with VMware.. does anyone have any updates / progress? I've opened one myself this morning.

[davidi123]

Ok finally VMWare found now the bug:

>This issue has been addressed in VMTools 11.1. which is due for release on 14/04/2020 (please be advised this is subject to change.).

>Meanwhile we advise you to workaround this issue by uninstalling the Network Introspection Driver if you are not using any Identity Firewall or Endpoint Monitoring feature.

[Mickeybyte2]

davidi123​

Thanks for the update!

Did VMware by any chance say what to do if you do need the introspection driver for endpoint monitoring? I suppose reverting to 10.3.x will be the case then? It's still a long time to wait until 11.1 will be available apparently.

Regards,

Michiel.

[ChicaneUK]

My ticket went through standard support and has now been passed to the NSX support group although my last contact from them was on Friday morning. I replied later in the same day and have yet to have a response from them, which is a little disappointing.

[rmouramx]

Hey guys!

I have the exactly same environment and issues.

Did VMware provide any update?

Tks,

[davidi123]

no news. its clear: wait for the new tools or downgrade to 10.3 branch

[rmouramx]

Hello Guys,

Any news from VMware regarding the new release?

[jmatz135]

The notes specify that "There was a problem in the packet re-injection code for the ICMP and UDP IPv6 packets for outbound traffic. As a result, outbound IPv6 traffic for ICMP and UDP protocols could experience packet drops."

Presumably this means that only IPv6 traffic for ICMP and UDP connections are effected.  This doesn't really seem to be the case.  What actually is effected by the bug?  What packets are being dropped?

[ChicaneUK]

I notice that VMware Tools 11.0.6 is out though no mention of a fix nor any acknowledgement that it's a known issue.. which seems a bit sneaky to me!

[jwl6197]

Just installed VMware Tools 11.1 (was released yesterday) and am still seeing dropped pings. Not seeing any mention of the issue in resolved bugs or known bugs in the 11.1 release notes.

[VMHero4Ever]

Same for me. VMware Tools 11.1 has the same issue like the versions before.

[tcommander]

Has anyone tried the VMware tools from June 18 that list a fix for this issue?  We have installed it based on this issue and are beginning to test but wanted to see if anyone else had feedback on the issue.

[jmatz135]

I have a desktop pool with VMware tools 11.1.1 that I am testing.  From what I have seen so far the dropped packets issue has been fixed.

[VMHero4Ever]

This issue seems now to be fixed with the lastet version 11.1.5.
But it's nothing mentioned in the release notes about the ping drops.

[Jamadag]

I'm using the latest drop of 11.2 as of 10 Nov 2020. The problem stil persists. I'm seeing ICMP packets loss, echo REQUESTS are swallowed and there is no ECHO response.

[CalW]

Yes. Same here. As of 8 Dec 2020, i am getting the same error still.

I am going to log the official support call to VMWare helpdesk. Hope that they can solve it asap. 

[CalW]

After upgraded to VMTool version 11.2.0 then problem solved.