25GbE NIC's Installed and added to VDS - but VM's not running at 25GbE still showing 10Gbps on NICs

[danielmgrinnell]

Hey all, 

This is my first post on the new "Community"! 

I have been running 10GbE SFP+ for years in my ESXI hosts and i finnally upgraded to 25GbE SFP28 HBA's and the show up perfectly under Physcal Adapters. 

Now the issue is when i started doing some performance testing. 

First off, should i expect my Guest OS to recognize 25Gbps on the NIC's?

I am running VMXNet3 of course and have been doing some testing adding and removing NIC's reinstalling the VMware Tools, but nothing seems to push my testing past 2-4Gbits / sec in iPerf. 

In my VDS i moved the new onboarded 25G nics / uplinks up in nic teaming as Active and on the VM's the speed doesnt change. I move the 10G back up as active and its basically the same performance. 

I am going to deploy a clean vm from scratch, but i have a few test vms and one i have attached a few NIC's to w/ multiple Adapter Types and nothing is showing up as 25G. 

Thanks 

[ggathagan]

This article suggests that you have to do some configuration changes to the
VM.

https://knowledge.broadcom.com/external/article/368260/how-to-change-or-add-vmxnet3-link-speed.html

Regards,
Grant

[danielmgrinnell]

Thanks so much for that suggestion!! That KB was a not working for me w/ the X being the value, but there is actually a newer one that worked for me and i was able to get my Local NIC showing up at 25GB and another 65GB. 

Here is the other article that i found after your suggestion, the key was ethernetx being the NIC so i changed that to 0 and then set the value in my vm advanced parameters and that worked like a charm! 

Attribute = ethernet0.linkspeed and Value is the speed = 65000 or lower ( can go up to 65000) 

How to change the vmxnet3 link speed of a VM

https://knowledge.broadcom.com/external/article?articleNumber=368812 

I will be doing some performance testing on multiple guest OS speeds, thanks so much! 

[bb2bb2]

How many VMs and how many HBAs?   Just remember if you have one 25Gbit NIC shared by 10 VMs, each VM is sharing that resource.  Not a bad thing to have multiple 10Gbit VMs sharing a 25Gbit NIC.  But as already mentioned, you need to mod your nic config to get more than 10Gbit via VMXNet3.   Also it takes more streams to saturate 10 and 25Gbit and faster storage to keep up with feeding the bandwith, and possibly jumbo frames and fast enough CPU and storage.  If your source and destination storage can't read and write at 25Gbit (3.125GB sustained), your VM's won't be able to saturate  your network at 25Gbit.  also make sure you are using multiple threads in iperf, and have enough CPU cores and core perf in source and destination VMs you are testing. 

[danielmgrinnell]

Hey! 

I have many VM's, but just for testing after the intergration i only update a few vdPortgroups and moved up the new 25GbE Uplinks to make those VM's use the 25G network. 

So i have about 3 VM's now running on the Portgroups where i have the 25G uplinks running, i have a 2Xport SFP28 HBA per host. The VM's are on the Same hosts. 

So i actually upgrade iPerf from a lower version to the latest version iperf3.17_64 and i got way better results as well and i had deployed a few net test vm's specifically to test the performance of this network and 25G uplinks.

I updated the Advanced Paramerted to ethernet0.linkspeed  = 65000 (and played w/ other vms at 25000) but did 65 to see if it made a difference (didnt really increase any performance, but cool to see the NIC change to 65GB in Guest OS). 

I am now getting speeds up to 8 Gbits/sec, but im curious if i should ever expect to see anything OVER 10G since I have a 2x Port SFP28 HBA. I started Playing w/ Streams and pushed the speed up to 22 Gbits / Sec! WOW now were talking! 

Thanks for the tip on Streams that was helpful!