Does # of VMs per host affect VM network throughput?

[Strago]

I have observed a network performance degredation on my VMs that is linearly proportional to the # of VM's on the same esx host and same vSwitch. i.e.: if I have only 1 VM on my esx host, NIC throughput is identical to a physical server. If I have 30 VM's, throughput is at 66% or lower. Some full iometer metrics posted below.

This would make sense, except the network usage for my VM's are generally idle or very small, even with 30 VM's per host. According to VC performance metrics for the span of time before and after I took the below tests, total usage for the entire host never exceeded 2 MBps. I also had more than sufficient cpu and memory resources. It's as if there is some kind of reservation in place that keeps a single vNIC from taking the full resources of a single pNIC.

My esx hosts have 4 pNIC's, with 2 devoted to my VM network connection on a single vSwitch, which is configured for teaming via port ID. Another odd thing I notice is that there is never any network usage on the 2nd pNIC. I have 3 esx hosts and notice same behavior on all.

Questions:

1) Is this performance degredation normal, and if so how can I resolve this bottleneck?

2) Why would I only see network traffic on 1 of the 2 pooled pNICs?

Below are my observed performance metrics I took with iometer, using the standard config used in this thread: http://communities.vmware.com/thread/197844. Apologies if the formatting sucks:

1 VM on esx host:

MBps (combined) I/Ops (combined) Avg res time

147.762649 4728.404763 0.422072

84.078535 10762.05242 0.185083

129.469283 4143.017045 0.48172

74.907211 9588.123057 0.207828

10 VMs on esx host:

MBps (combined) I/Ops (combined) Avg res time

135.627161 4340.069138 0.45996

59.18213 7575.312687 0.263199

115.333058 3690.65786 0.540999

60.200264 7705.633782 0.258738

20 VMs on esx host:

MBps (combined) I/Ops (combined) Avg res time

116.575716 3730.422917 0.535215

46.279005 5923.712661 0.336761

92.205273 2950.568733 0.676814

51.330306 6570.279192 0.303519

MBps (combined) I/Ops (combined) Avg res time

98.387753 3148.408084 0.63427

47.310841 6055.787589 0.329366

89.395174 2860.645574 0.698114

48.690622 6232.399628 0.320018

[RParker]

> 2) Why would I only see network traffic on 1 of the 2 pooled pNICs?

Because you can either load balance or failover, but in either case it's one or the other NIC not both at the same time. If the first NIC is not failed or overloaded, the other NIC has no reason to be utilized.

As far as the degradation, I never personally saw that before, but it would probably depend on your NICs, since the "switch" is technically a shared NIC port, not really a switch, there probably has to be some priority of traffic or reserved bandwidth for each of the VM's. Then again I don't pool the NICs either, so that may be the reason. Have you tried a single non-pooled NIC to see if this is the same problem?

[Erik_Zandboer]

You should see data flowing through both pNICs if you have more than one VM active (and both pNICs are teamed together and both set to active). If you use the default teaming (based on originating virtual port id), each VM that connects up to the vSwitch will use another uplink. This balancing is round-robin based.

If you want to see a single VM use both uplinks, you should use teaming based on IP hashing. The ESX switch will then take the source and destination IP, create a hash from that and use that hash to determine the pNIC to use. So remember, your VM has one originating IP, so your VM would need to have sessions to multiple target IPs for the balancing to work. Ow and important, your physical switches should be configured to have an etherchannel (cisco) or port aggregate (other vendors) for it to work. Switches normally do not like seeing the same MAC address on multiple ports...

Do not forget that the IP hash teaming option uses more resources than originating port ID teaming. Maybe you are seeing this impact?

Visit my blog at http://www.vmdamentals.com

[RParker]

> This balancing is round-robin based.

Round Robin is one then the other, so I think you just contradicted yourself. Round robin cannot have traffic on both NIC at the same time, same with Fiber Connections, round robin. It finds one route,if one is busy, it then switches to the other, but not both.

So in the end, I am thinking it's NOT a TEAM, it's either failover or round robin, in both cases, the data is sent as needed, and not split between the two.

[Strago]

Same results when using vSwitch with single NIC.

[Dave_Mishchenko]

Moved to the performance forum.




Dave

VMware Communities User Moderator

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.

[Strago]

Sorry for the wrong forum there, thanks.

I'm configured for load balancing by port ID, and failover by link status. We do not require IP hash teaming, and are not etherchanneled.

You two seem to disagree as to whether I should see traffic on both of load balance is configured, I would expect that I should even if not overloaded as Erik explains. i.e. approx half of my VM's should be on each of my 2 NICs.

RParker good idea, I will try a single non-pooled vSwitch when I get a chance and let you know.

[RParker]

Well disagree or not, I have tried it, and it doesn't work. The other thing is that nothing in VM Ware is a TRUE team, everything is priority based, one NIC has priority over the other, and they aren't truly TEAMED together to work as a single unit.

NICs don't work this way, neither do HBA, and my testing also showed this.. and YOUR testing clearly shows this.

So if it's NOT working (as a team) then there can only be one definitive answer, and that is this is the way it was designed, as one NIC then the other.. as I stated. Because if it SHOULD be working, then why isn't it? I think its because that's how VM Ware intended it. Maybe LATER drivers will be a TEAM, but as of right now they clearly don't work that way.

[Erik_Zandboer]

@RParker: What exactly is not working? If you have more than one nic in a team, and both are set to "active" for the VM portgroup, and you set teaming to "based on originating port ID" then it should balance load across nics. It is a static balance though, each VM is uplinked through one of the nics.

Remmeber, "based on originating port ID" is exactly what it says: It uses the port ID where the VM is connected to the vswitch. So it might very well be if you have 10 VMs, and start load on two, they just happen to share one pNIC.

As for round robin: It is not really round robin, but the PLACEMENT of the VMs on each pNIC appears to be in a round robin order. Why? Because as soon as you connect the next VM, the port ID of the vSwitch it connects to puts it on one of the pNICs, appearing to "balance" them in a round robin way of working.

Visit my blog at http://www.vmdamentals.com

[Duncan Epping]

You might be hitting a known issue:

http://www.yellow-bricks.com/2010/01/20/nic-teaming-load-balancing-does-not-work-with-global-vswitch-configuration-on-esx-3-5/

Duncan

VMware Communities User Moderator | VCP | VCDX

-

---

Now available: Paper - vSphere 4.0 Quick Start Guide (via amazon.com) | PDF (via lulu.com)

Blogging: | Twitter:

[Strago]

I did notice that KB, but am confused by it. The resolution steps only describe how to set up load balancing, which I of course already did in the first place. It mentions "global" vswitch config, I can't find any documentation that explains what that is.

My plan for now is to put this on the back burner until I upgrade to version 4 and call support if problem persists. Thanks fellas.