We have a large project underway to virtualize our Windows 2003 terminal server farms and I am having an issues with network performance on my VM's.
Here is the scenario:
- My physical TS app server is a Dell 1950 Gen III with dual Quad core procs and 8GB of RAM. A pair of local 146GB, 10K SAS HDD's in RAID-1 config. We use an Intel Dual port PCI-E adapter for our network connectivity (instead of the on-board Broadcom NetExtreme II adapters)
- Server is loaded with Windows 2003 Ent x32 with all the latest SP's and Patches and the physical box has the latest firmware and drivers.
- We also have a bunch of applications installed that are required for or TS farm.
- Typical app server can support up to 80 users before resources become an issue.
- From a networking standpoint, the server is attached to two Cisco 2960G edge switches (one port to each switch). We run our RDP traffic through one switch and our local data access through the other. (We do this because our server farm app load balancer does not use MSNLB; we use a Cisco F5 device to manage connection distribution and it has to own the RDP VLAN.) Note: we only configure 1 gateway address (on the RDP nic) and have static routes in place to direct local traffic to the data NIC in case anyone was worried about multi-homing issues.
We have taken one of these app servers (with the exact same HW config) and pulled it from the farm. we added 1 additional connection to an on-board Broadcom adapter for management purposes so we can use the Intel NIC's as we did before. Wiped the server and loaded ESXi 4.1..fully patched.
- We have pretty much left the ESXi server in an out of the box config. \
- We have three vswitches: vswitch0 for management access to our VCenter server, vSwitch1 for RDP and vSwitch2 for Data.
- Built from scratch a new v7 VM using the paravirtualized SCSI adapter and VMXNET3 adapters. Server is configured with 1 vCPU and 4GB's of RAM, 2 18GB Hard drive VMDK's (configured used the local storage on the server)
- Loaded the same software, revision levels, etc that we have on our physical servers. (BTW, all our servers are built using automation scripts to ensure consistency and minimize any potential user errors in configuration). Obviously, the drivers are different and we do not have Dell OpenManage installed...so technically, it is a little simpler build. Needless to say, we are also running the latest VMWare tools version.
So, long story short, with only 1 VM running on ths app server, we wanted to compare performance to the physical to see exactly how we would go about scaling out vs up.
Testing scenario:
- We pulled another physical app server, cleared off all users and rebooted it to make sure it was clean.
- We have just 1 VM running on the ESXi app server, also with no users on it.
- Part of out testing included running basical log on/off timings as well as application work flows to (with only 1 user logged on the physical and virtual servers).
The problem:
We are seeing a significant difference in run times between the two, specifcially for tasks that reach out to network resources, in some cases, the VM is running about 30-40% slower.
What have we tried:
- replacing the vNIC with a VMXNET2, e1000 and flexible....result...performance became even worse
- went down to a single nic configuration....no change.
- increased CPU and Memory...no change
- when down to a single vSwitch config...no change
- tried using broadcom NIC instead of the intel....still, no change.
- We through every option available in the guest OS for the VMXNET3 adapter...most had no effect, others made things way worse.
Bottom line, we expect (hopefully correctly) that with only 1 user on a single VM running on a server exactly configured like a physical, that the performance should be the same...but it is not. I know that there is a substantial amount of tuning paramters available in ESXi under Advanced Setting/NET...but I have not found ANY guide out there that explains in lamens terms what those setting are supposed to do nor have I found anything out there that says: 'heres how to configure VMWare to optimize networking for Terminal services and/or network based applications that require low-latency."
So, lot's of info here, but I am at a point where i need to start asking for help from those that may have actually encoutered this behavior before. ANY input/suggestions would be much appreciated as this is a show-stopper for us moving forward in our Virtualization project.