Slow (only) SpeedTest (only) upload (only) from ESXi VMs - Why?

[Pawel Lipko]

I have strange performance issue most probably with my Vmware ESXi.

When I run any speedtest service I'm getting upload results of 30,90% of my upload link.

My ISP connection is 1Gb down / 300Mb up but any SpeedTest service gives me 909,93 / 92,71 when run from Windows 11 or Ubuntu VM running on ESXi. The results are the same with browser test and speedtest cli test (I also tested other - speedtest.pl service. Same results)

On the other hand my physical laptop with the same tests gets 914,57 / 307,40 which is 91,46% / 102,47%. So it's not LAN nor ISP problem.

This is getting more interesting when I started to test between physical laptop and Ubuntu VM:

• ·         Iperf3: 878 / 844 (87,80% / 84,40%)
• ·         SMB single file transfer: 847,22 / 737,73 (84,72% / 73,77%)

So this is not LAN / ESXi problem.

But there is a problem…

My test results below – each line is an average of 3 separate tests, just to be sure there wasn't some better or worse moment in time.

Updated with Win11 VM to Ubuntu VM iperf test and Ubuntu VM internal iperf test (loopback)

My HomeLAN looks as follows (I marked test flows with green and red arrows – showing what is running fine and what's not)

Before I did iperf3 tests I started to check FO cable by replacing it – it's now obvious, that the cable was ok.

Home server short spec

• ·         CPU: AMD Ryzen 9 7900
• ·         RAM: 96GB DDR5,
• ·         M2 SSD: 4TB + 1TB,
• ·         SSD: 2x 4TB,
• ·         HDD: 20TB,
• ·         NIC: Ethernet Controller X710 for 10GbE SFP+

Vmware software versions are:

• ·         vCenter Version: 8.0.3, Build: 24322831
• ·         ESXi version: 8.0 Update 3

·Początek formularza

I don't see a choke point here.

All my VMs do use VMXNET3 NICs, they are running most fresh Vmware tools

The NIC in use is officially supported by Vmware.

ESXi è Manage è Hardware è PCI devices shows SR-IOV disabled and no passthrough

My vSwitch DMZ that connects Ubuntu SRV with FW as follows:

Note there are some Ubuntu test copies, that are normally unused

The vSwitch DMZ settings:

The main vSwitch topology (it connected FW with outside world and provides direct connection for Win11 VM)

Note: physical port is 10Gb auto negotiated

Main vSwitch0 settings

Physical Mikrotik switch has 10Gb auto negotiated link between Home server and Internet router (and 1Gb for physical Dell laptop)

Summarizing: I have slow upload when testing with few speedtest services (no matter if it's browser or cli app) only from ESXi VMs

Why?

[ManlioFrizzi]

Hello,

Before reading keep in mind that I've read few sources on reddit that solve changing ISP. (https://www.reddit.com/r/vmware/comments/1bgoas6/slow_internet_upload_speeds_on_vms/ )

There are several reasons why the upload speed test results by Ookla executed inside a VM on an ESXi host might be slower than expected:

1. Virtualization Overhead: VMs inherently introduce some level of performance overhead compared to physical machines. This overhead can affect network throughput, especially upload speeds which can be more sensitive to latency and resource contention.

2. Virtual Network Adapter Type: The type of virtual network adapter configured for the VM can significantly impact performance.

   • VMXNET3: This is the paravirtualized adapter designed for high performance and is generally the recommended choice.
   • E1000/E1000E: These emulate older Intel network cards and might have lower performance, especially for high-speed connections.
   • Other Adapters: Ensure the chosen adapter is appropriate for your network environment and performance needs.

3. ESXi Host Network Configuration: The physical network adapter (pNIC) on the ESXi host and its configuration are crucial.

   • Speed and Duplex Mismatch: Ensure the speed and duplex settings on the pNIC match the physical switch port it's connected to. Mismatches can lead to significant performance degradation.
   • Network Congestion on the Host: If other VMs on the same ESXi host are heavily utilizing the network, it can impact the available bandwidth for your test VM.
   • Uplink Capacity: The total uplink capacity of the ESXi host's network configuration (e.g., number and speed of NICs in a vSwitch) can be a bottleneck if it's insufficient for the combined demand of all VMs.

4. Resource Allocation to the VM: The amount of CPU, RAM, and network resources allocated to the VM directly affects its ability to process and transmit data. Ensure the VM has sufficient resources to handle the expected network speeds.

5. Network Congestion Outside the Virtual Environment: The bottleneck might not be within the virtual environment itself.

   • Physical Network Infrastructure: Congestion on your local network (switches, routers) or your internet service provider's (ISP) network can limit upload speeds.
   • Ookla Server Load: The speed test result can also be affected by the load and distance of the Ookla server you are connecting to. Try testing with different servers.

6. VMware Tools: Ensure that VMware Tools are installed and up-to-date within the guest operating system of the VM. These tools provide optimized drivers for virtual hardware, including network adapters, which are essential for good performance.

7. Guest Operating System Network Configuration: The network configuration within the VM's operating system itself can play a role.

   • Firewall Rules: Aggressive firewall rules might be inspecting or limiting upload traffic.
   • TCP/IP Settings: While usually auto-configured correctly, incorrect TCP/IP settings could potentially cause issues.

8. Quality of Service (QoS) Settings: QoS settings at any level (VM, ESXi host, physical network) could be prioritizing download traffic over upload traffic, leading to slower upload results.

9. Storage Performance (Less Likely but Possible): While less direct for upload speed, if the VM's storage is heavily utilized or performing poorly, it could indirectly impact the VM's ability to stage data for upload efficiently in some scenarios.

10. ESXi vSwitch and Port Group Configuration: Review the configuration of the vSwitch and port group the VM's network adapter is connected to. Ensure there are no unexpected traffic shaping or security policies in place that might be limiting upload speeds.

Troubleshooting Steps:

• Verify Virtual Network Adapter: Ensure the VM is using the VMXNET3 adapter.
• Check VMware Tools: Confirm VMware Tools are installed and running.
• Monitor ESXi Host Resources: Check the CPU, memory, and network utilization of the ESXi host during the speed test.
• Test from the ESXi Host: If possible, run a speed test directly from the ESXi host's console (if it has network connectivity and tools) to see if the issue is with the physical connection.
• Test Other VMs: Compare the upload speeds of other VMs on the same ESXi host. If they also have slow uploads, the issue might be at the host or network level.
• Simplify the Configuration: Try testing with a minimal VM configuration and no other heavy network traffic on the host.
• Check Physical Network: Verify the speed and duplex settings on the physical switch port.

By systematically checking these potential causes, you should be able to identify the reason for the slower than expected upload speeds in your VM.

------------------------------
------------
Manlio
PMP®-PMI, Azure & AWS certified, VCP34562019/21 DCV VCAP5DCD, vExpertx10, RHCVA
------------------------------

[Pawel Lipko]

You're talking about virtualization overhead but I remember after installing those Linux VM behind Sophos FW and Windows 11 VM in my outside VLAN (without any FW in between) I've been testing them with SpeedTest, ierf, FTP transfers and Linux implemented NextCloud transfers.

I remember I was getting almost 100% speed results everywhere with Netxcloud transfers dropping to 200Mb/s which was acceptable for me.

Now I see upload at 33% expected (100Mb/s with 300Mb/s ISP link) and it works this was for all ESXi VMs, no matter where they are runnig (behind FW or not).

Also I don't have exact proof, but this might happen after I upgraded from VC-8.0U3b to VC-8.0U3d. There was some CVE announced, so I did the upgrade but I haven't tested the VMs for performance (only are my services run at all).

I noticed this poor upload... So it's hard to point that NIC driver is wrong (I use VMXNET3 everywhere though) or there is network congestion (if there would be a loop, nothing would be even reachable. In best case there would be any reachability, but with 5 min. waiting to get anywhere)

Yhe ESXI pNIC config - I pasted screenshots with info this is 10Gb full duplex on both sides - ESXi and physical switch

I will take a look through your list but I guess I have checked all (or almost all) mentioned points before writing here...

I wouldn't be writing detailed question as my first action

I updated test list with iperf from Win11 VM to Ubuntu VM and Ubuntu VM loopback iperf test

For instance - physical laptop SpeedTest gets ~100% result while ESXi VMs notice ~33% for upload.

Both PCs are connected to the same CSS switch and next to RB4011 router, so connection is similar.

But results are not

[StephenMoll]

I think a read of "VMware vSphere 6.5 Host Resources Deep Dive" by Frank Denneman and Niels Hagoort is in order here. It was written at the time of vSphere 6.5, but most of it still holds true.

In essence though, I think your observation is as a result of the following:

A VM is comprised of several threads running in the hypervisor. These are the vms-vcpu threads. Clearly these need pCPU core time to operate. What a lot of people overlook is that fact that the VMs network hardware needs to be emulated as well. For VM outbound network traffic, this is handled by what is called a TXTHREAD. Every VM has one of these by default, and these threads are constrained by the same CPU resource reservations, limits and sharing as the VM they serve. So in overprovisioned environments, like yours, the TXTHREAD is not going to get 100% of the host pCPU time it needs to run at full speed all the time. 

Inbound traffic to the host is handled by what are called NETPOLL threads. There is at least one of these per physical network interface, and if your NIC supports RSS, then there will be a NETPOLL thread for every RSS channel. NICs typically have 8 or 16 channels per port. For the X710 I think it is 16, so your system will have something like 32 NETPOLL threads waiting to jump onto physical cores if needed. 

As NETPOLL threads service more than one VM, their CPU time is effectively 'paid for' by the destination VM's CPU resource allocation, and again subject to the same reservations, limits and sharing. Therefore, for inbound traffic, 100% performance in an overprovisioned environment cannot be expected.

[Pawel Lipko]

Hi!

I haven't found what caused this, so I decided to work this through on 2nd example (the hard way)

I bought a Dell laptop and enabled there another ESXi, 8.0.3, 24674464. I recreated the VNets and vSwitches, etc. added it to my vCenter, vMotioned some Linux VM and see this

Linux and Windows VM on Homeserver with 
ESXi, 8.0.3, 24022510 
AMD Ryzen 9 7900 12-Core, 
96GB DDR5 RAM 
NVME SSD drive
Connected to LAN with 10Gb Eth

Linux VM on Dell Precision 3541 laptop with 
ESXi, 8.0.3, 24674464 
i7-9850H, 
32GB RAM 
NVME SSD drive
Connected to LAN with 1Gb Eth

It seems my upgrade just went wrong, the image wasn't too lucky (ESXi, 8.0.3, 24022510) and I will need to reinstall my primary ESXi to ESXi, 8.0.3, 24674464

Will let know if anyone cares :-)