Windows Server 2012 R2 Bad Performance

[DXK5314]

Hello,

I've a vSphere cluster running with 7x IBM HS22V blades / 2x Intel Xeon X3650 / 256 GB memory in every host.

Running at vSphere 5.5 23202651, with around 150 VM's.

Since a few months we are implementing Windows Server 2012 R2, and since a few weeks we've a new Citrix environment running at 2012 R2 to.

Currently the performance of the Windows Server 2012 machines is really really bad.

The respond slow and it doesn't feel right, the older VM's (W2K8 R2) are running great.

I've searched the internet, and I found that a lot of people had/have problems running 2012 on vSphere.

But I didn't found any solutions or things I could try to solve our problems.

Hopefully you guys have some tips for me, because I'm stuck...

Thanks in advance!

Best regards,

DXK

[CoolRam]

please do get the esxtop logs for this virtual machine and analyze those with standard value.

[Srinivasu13]

Hi,

I agree with you even our 2012 are running slow in our environment.

Recommend you to verify below things on your 2012 VMs.

• do you have VMware tools installed in your guest? do you use paravirtualized SCSI adapter with your guest?
• host CPU usage - host -> performance tab -> advanced -> cpu - what is average CPU load on the host? is it over 75%? are there peaks over 90%? it may signal that host CPU is saturated.
• guest CPU usage - host -> select the vm -> performance tab -> advanced -> cpu - the same like above plus check the CPU READY TIME, is it over 2000ms? again, the host CPU may be saturated.
• guest memory usage - host -> vm -> performance tab -> advanced -> memory - if memory average is over 80% or peaks over 90% the guest may have higher memory demand and so it is another performance bottleneck.
• guest memory swapping/ballooning - host -> performance tab -> advanced -> memory - check swap-in/swap-out and balloon rates. if it is over zero the host is swapping or ballooning vm memory which may make the vm sluggish as it requires access to disks.
• storage overloaded - host -> performance tab -> advanced -> disk - check STOP DISK metric, if it is over 0 zero disks/LUNs may be overloaded. additionally, check read/write latencies, if it is over 10ms in average or over 20ms in peaks disks/LUNs are slow or overloaded.
• Microsoft patches are up to date?

[DXK5314]

"Hi,

I agree with you even our 2012 are running slow in our environment.

Recommend you to verify below things on your 2012 VMs.

 

• do you have VMware tools installed in your guest? do you use paravirtualized SCSI adapter with your guest?
• host CPU usage - host -> performance tab -> advanced -> cpu - what is average CPU load on the host? is it over 75%? are there peaks over 90%? it may signal that host CPU is saturated.
• guest CPU usage - host -> select the vm -> performance tab -> advanced -> cpu - the same like above plus check the CPU READY TIME, is it over 2000ms? again, the host CPU may be saturated.
• guest memory usage - host -> vm -> performance tab -> advanced -> memory - if memory average is over 80% or peaks over 90% the guest may have higher memory demand and so it is another performance bottleneck.
• guest memory swapping/ballooning - host -> performance tab -> advanced -> memory - check swap-in/swap-out and balloon rates. if it is over zero the host is swapping or ballooning vm memory which may make the vm sluggish as it requires access to disks.
• storage overloaded - host -> performance tab -> advanced -> disk - check STOP DISK metric, if it is over 0 zero disks/LUNs may be overloaded. additionally, check read/write latencies, if it is over 10ms in average or over 20ms in peaks disks/LUNs are slow or overloaded.
• Microsoft patches are up to date?"

- Yes, up to date version.

- No, we don't use SCSI adapters.

- CPU load = 8369 avg --> Max = 12195, Min 4995, so I guess this is good. No peaks over 90%. --> imgbox - fast, simple image host

- I see multiple lines, the green one goes over 2000 Mhz. --> imgbox - fast, simple image host

- Guest memory --> http://imgbox.com/UTg9bz5v

- Disk --> imgbox - fast, simple image host

- Storagepath --> imgbox - fast, simple image host

- All Microsoft Windows updates are installed on the W2K12 VM's.

Hopefully you can take a look at my printscreens, I'm stuck and don't see where it's going wrong.

[Srinivasu13]

Sorry I can't access your screen shots.

Can you test one VM with paravirtualized SCSI adapter & VMXNET3 NIC.

[Srinivasu13]

Also verify ESXi hosts are running with latest patches.

[williambishop]

Is it ALL 2012 boxes or just the ones running citrix?

[Wh33ly]

Not sure if your problem is VM performance related or maybe guest related, here is something I encountered:

We have a lot of Win2012 machines which run great, one of the first things I noticed when we started deploying  was the slow and sluggy mous which was caused by the mouse pointer shadow.

Go to the “control panel” and open “System”

Open “Advanced system settings and then click the “Settings” button at the performance field. In the tab “Visual effects” you see an option “Show shadows under mouse pointer”. After disabling this, the performance of the mouse was back to normal again.And the server looked much more responding

I also decided to set the settings to “Adjust for best performance” because this will decrease any other almost not noticable lag

Could be as simple as this if applicable.

[MCOV]

Please check this:

- change network adapter to vmxnet3 (instead of E1000E)

- disable tcp chimney offload (checking status with cmd: netsh int tcp show global)

- in adapter properties in adapter configuration on Advanced tab disable: Large Send Offload V2, Recv Segment Coalescing and TCP Checksum Offload

[skelpie]

Hi guys !

I think i've solved the problem, but i don' know how can this happened.

Here what i've done.

I have ESXI 6.5 Dell version on Dell R900 Server. There is a Windows 2012 R2 on it. After installation the performance was very poor. When i started to upgrade the Hypervisor (because of hardware incompatibility of R900 'vve installed first ESXI 6.0 then updated to 6.5)

After upgrading i've reregister th VM W2012R2 and the performance was brilliant. Then i upgrade VM's version from 9 to 13 - (esxi 6.5 and later) - performance was sh..t.

My conclusion

When use for ESXI 6.5. VM W2012R2 with hw. compatibility level 9 - everything is superb

My best config on 4 Xeon 7350 is 2 VCPU's 4 Cores each

Now is blasting!!!!

[Maddas68]

We are experiencing the same problem with some Windows 2012 R2 servers, especially RDP/Citrix servers. Downgrading from Virtual Machine Version 13 to version 9 solves the problem.

The question is, why is this happening? Why do Windows Server 2012 R2 have problems with VMWare Version 13?

[peterriederer]

Hi Folks,

you also have performance issues with Windows 2016 Servers? Our 2012R2 Servers are running quite well on ESXi 6.5 latest patch and VMware Version 13.

But the 2016 Servers are really poor, even if they only have 1 vCPU and only 4GB RAM. With more vCPU they are a bit faster.

Regards

Peter

[peterriederer]

Since a few days we have the same Problem here.

We are using vSphere 6.5 and HW-Level 13 several weeks now, and it worked quite well, with our RDP and Fileservers.

But since a few days It all started with Veeam Backup stopped / wasnt able using VSS anymore while backing up our main fileserver.

From this time our W2K12R2 File-Server went slower and slower. 100% Disk-Activity, Disk-Queues running high and Disk-Latency in Windows showed more then 700 - 1000 ms. We tried a lot... uninstalled Windows Updates, uninstalled Anti Virus, rebooted the SAN, rebooted ESX-Hosts, nothing helped.  First we used LSI SAS Controller (same Problem) and now we are using Paravirtuell controller and VMNEXT NIC. If there is just a very little load on the machine, it is going to get unusable, and just small word and excel files take forever to open and to save.

Now we tried to go back to HW-Level 10 for both Fileservers and it is working again, without any Problems. For testing purpose we tried to upgrade to HW-Level 11. One Fileserver is ok with that the other one we got the same problems again, hence we went back again to HW-Level 10.

But it seems more and more Machines are getting sluggish now, Medical-Application with an oracle Database (W2K8R2 Server) now also gets slower and slower, and the application takes a very long time to load.

Does anybody have any news or and helpful Tipps and Tricks?

Regards

Peter

[Moonduza]

I Would do the following:

- Add a VMXNet3 NIC
- Use Paravirtual Storage Controller
- Update VMware tools
- Check the host power management configuration and, if possible use high performance.

[peterriederer]

thanks but this is always our standard setup/config, and was already mentioned above.

- we only use VMXnet3 NIC,

- we only use Paravirtual Storage Controller (except these 2 filservers havnt been migrated so far, due to missing maintenance windows) but now they are and didnt change so much

- Tools are always up2date

- Host Power is high performance

we had no issues in the past, and didnt change anything in our infrastructure.

the only thing we changed was the disk size of the file server -> 1.8 TB to 2.5 TB but that could not be an issue in my opinion.

[controlsforall]

We are having the exactly same issue.

I had extended the driver from 4TB -> 6TB.

Windows 2012 memory utilization keeps going up. That VM locks and also it locks the whole vsphere server. I end up having to do hard reset on the whole ESXI.

Within the vspehere memory usage show 6GB and keeps increasing but on the Windows server it is stable around 2GB.

We have upgraded to the latest build. I was on tech support with dell. We upgraded perc and bios. I think we are still having the same issue.

[peterriederer]

you tried to go back to hardware compatiblity 10?

with more and more testing it seems other VMs and OSes are also affected, like WSUS Servers etc.

e.g. just start installing windows updates on Machines with HW13, and watch disk queue and disk latency in performance monitor (task-manager)

[adamsavagex]

I've witnessed the same thing and going back to hardware version 9 or 10 seems to fix the problem..

I was seeing high disk queue length on VMs  that are Windows Server 2012 R2 x64.

I'm going to bring up 2 VMs off the same template and downgrade one of them to hardware version 9/10 and do a comparison of the hardware loaded in the system and see if we can find the culprit.

[mvduijn]

We are experiencing the same issues with our file servers after upgrading to hardware level 13.

We have high disk queue length on VMs  that are Windows Server 2012 R2

We created a ticket with VMware but are still investigating the issue because support told me there are no known issues about hardware level 13.

The most problematic servers we already downgrade to hardware level 9 and i can't show VMware support the issue because other servers don't have the issue all the time.

Currently i am cloning a downgraded fileserver with hardware level 9 and upgrade to 13 hoping to reproduce the issue.

Does anyone have a vmware support call about this problem I can refer to?

[peterriederer]

We opened a ticket too, but after talking to them they dont know anything about problems with HW Level 13. So it just was an informational SR.

we both agreed to close the ticket, because i would like / should to do some more testing in our lab with a cloned fileserver for example etc. opening a new one then.

But at the moment i have not much time to investigate further in this problem :-(

But since i am back at HW10 with our 2012R2 Fileservers, everything is working fine. Other machines with HW13 are still lagging after there is some small load, but i dont want to downgrade them all at the moment.

[rblaas]

I am very interested in what VMWare will say about this issue.

I just installed a fresh Windows Server 2016 on our 6.5 vSphere platform using HW level 13 and it is not performing at all. I can't even update.

VMware tools are installed

Running on 2 CPU

4GB Ram

VMXNET3 Network

LSI Logic controller

(if important running on BIOS)

[peterriederer]

i am wondering, why this issue didnt show up earlier from more customers. vsphere 6.5 is already out some time now.

[adamsavagex]

I upgraded my ESXi hosts from 6.0 versus doing a clean install.. Is that the same for the rest of everybody else or were there some clean installs of ESXi hat are seeing the same issue?

[mvduijn]

All our ESXi hosts were upgraded from 6.0

[adamsavagex]

Hey mvduijn I need some confirmation.. I've been doing multiple tests today to try and resolve the issue while being on Hardware 13.. Can you install ALL of the features offered up in the VMware Tools install - specifically the VMCI drivers on a Server 2012 R2 system? i went to custom and installed from there.. I don't want to jump the gun but i did this on my 2012 R2 HW13 server and the queue length appears to be gone but i need confirmation from someone else before I can go back to my VMware ticket and point the finger at VMCI.

Thank you,

Adam

[peterriederer]

Hey Adam,

which VMware Tools Version did you use? i will try to test it on one of our machines. Currently we are are on 10.1.0 shipped with vSphere 6.5 but also tested 10.1.5.

You installed all VMCI-Options or just File and/or Network introspection driver ?

2 ESXi Hosts, where upgraded from 5-> 5.5->6->6.5

1 ESXi Host, was clean installed with 6 then upgraded to 6.5

Now 10.1.7 is out since a week which solved some issues with VMNEXT NIC Drivers:

Receive Side Scaling is not functional for vmxnet3 on Windows 8 and Windows 2012 Server or later

This issue is caused by an update for the vmxnet3 driver that addressed RSS features added in NDIS version 6.30 rendering the functionality unusable. It is observed in VMXNET3 driver versions from 1.6.6.0 to 1.7.3.0.

Regards

Peter

[mvduijn]

Hello adamsavagex,

I will install the vmware tools with VMCI driver tonight to one of our servers to see if there is any difference.

If the disk queue length is gone i will let you know.

peterriederer​

Currently we are are on 10.1.0 shipped with vSphere 6.5 but also tested 10.1.5.

I missed the release of VMware tools 10.1.7 and will give this a try to.

Most of our ESXi servers where upgraded from 5.5->6->6.5

What server brand and storage do you use?

We have HP Proliant BL460c Gen8 servers with HP 3par storage

Maybe i give it a shot to install a server from scratch with the latest HP Custom image.

Thank you,

Mike

[peterriederer]

Hello Mike,

i tried 10.1.5 and 10.1.7 with and without VMCI -> NSX File and Network Option in Setup for some 2016 Servers (Cannot Test with 2012R2 Fileservers at the moment) and it seems / "feels" a bit better with 10.1.7 but the issue still happens from time to time, didnt figured out exactly when. Getting load while Starting a full scan with Anti-Virus e.g. is no problem at the moment.

The Update from 10.1.0 to 10.1.5 or 10.1.7 took a very long time, Disk Queue up to 5 or 10 and more than 1000ms latency, 2 machines freezed completely and i had to hardreset them.

We have Fujitsu Primergy RX300 S8 Hosts, connected to a DataCore SANSymphony-V Storage System via iSCSI 10 GB.

BTW: We also have some 2008R2 Servers running with Tools 10.1.0 and HW13 there are not such Problems. Only 2012R and 2016 Servers.

Do you have still some 2008R2 Servers too, without any issues?

Regards Peter

[adamsavagex]

Hey Guys,

I've had this happen on 2012 R2 VMs only on Hardware 11 and 13 (we dont run any 2016 yet in our Datacenter)..

Our VM Tools versions vary between some of the 2012R2 VMs

They got from 9.10.5, 10.1.5, and 10.1.7 and they all have the same problem.. I was hoping the VMCI drive would have fixed the issue but alas like you said too it "looks" to perform better but in the end it's still crappy performance..

I have Cisco UCS B440s. 230s, 200s, HP DL980s, HP BL490C, Dell FC630, and Dell R610s on EMC VNX/CX/VMAX/Xtreme-IO Storage. The problem happens on all platforms and storage types. Although on the all SSD storage the performance is fine because it's being masked by the extreme performance of the SSDs.

This is the last response I received from VMware Support last night.

"We are still investigating this issue. We will keep you posted about this case.

I look forward to your response. Feel free to contact me if you have any questions or clarification

Thank you for choosing VMware!"

Hopefully they will come back with an answer soon. I'm running out of ideas here. I noticed that between a v9 and v11 2012 R2 VM there are multiple Generic Bus Devices under Other Devices in device manager on the v11 system versus just not being there on the v9 VM.

I'll keep you guys updated with anything new from VMware.

[controlsforall]

I found the problem to be crashPlan software, it speeds up a lot but the memory leak still exists with  vsphere 6.5 with latest patch.

[peterriederer]

Hello Adam,

just wanted to know if you already have any new information, or did some new tests which might be helpful?

Regards

Peter

[adamsavagex]

Hey Peter - I just got an email back from VMware today wanting to setup a webex. So nothing yet.. I've done some more testing specifically on my VSAN Cluster which is a Dell FX2 chassis running FC630s and i can't seem to re-produce the problem there - but I can on my Cisco UCS blades.. That may be because my VSAN is all-flash and wouldnt see the queue length issue because its so fast.

I'll keep you in the loop when and if I get a resolution out of VMware.

[adamsavagex]

Good afternoon everyone,

I just got off the phone with VMware and they have identified the problem and it will be fixed in 6.5 Update1. Update1 will be released sometime in July. I'm not sure of the exact day for the release.

Here is the summary of my call with VMware.

As per the current issue, we see that the disk Queueing is recorded once the hardware version for the VM is higher than the 10. The issue is only seen when the host is installed with vSphere 6.5 and the hardware version is higher than 10.

We already have the same reported with the engineering team and we are currently working on the same. The current workaround is to retain the hardware version to 10 or lower where the OS version is set to Win 2012 or higher. This issue is temporarily pointing to be caused due to the virtual disk polling enhancement. We do not have a definite solution yet and are currently working on the same.

As we have agreed, your support request is ready to be closed. If you come across any additional queries within 21 days, please contact me and I will be happy to reopen the ticket and continue working with you.

[peterriederer]

Thanks Adam

these are really good news, except for waiting so long till the update will arrive.

[mvduijn]

Hello Adam,

Good to hear there you have some progress on the issue.

Since last friday the problem is getting worse every day.

A simple copy job of 5GB takes about 9 minutes to complete.

Disk queue length on VM's are skyhigh.

Our storage provider and HP Enterprise have finished their investigating and confirming this is not a storage Hardware issue.

Can you share your call number of VMware so we can  communicate this with our VMware support engineer to confirm we have the same issue?

Thank you in advance.

Mike

[adamsavagex]

My support request number was 17467861205

[AdamViusaSerrat]

The same issue is happening to us. Vsphere 6.5, virtual hardware verssion 13 with windows 2016 and windows 2012 is not working well. In one w2012 R2 we have a file server and intermitent disconnections occurred... a big disaster as we had to downgrade the virtual hardware to all 2012 an 2016 vms.

After downgrade to virtual hardware 10 (w2012) and virtual hardware 11 (w2016) the issue disappears and all comes to life as usual...

[peterriederer]

and its hard doing a downgrade of the virtual Hardware.

at the moment i just create a new VM with HW10 (2012R2) or HW11 (2016) and connect the VMDKs in there, or is there an other more easier way?

[adamsavagex]

You could throw the VM through VMware Converter and select the Hardware Version from there.. Then delete the old VM once you get the new one online.

All ways are kind of a PITA.

[AdamViusaSerrat]

But this takes a lot of time if the vmdks are big and it's not an option if your users can't work and all enterprise is waiting your "repair" of the situation. Create a new machine and attach the vmdks is the faster solution I think... but I'm going to open a new ticket also for this bug.

[vSphere_Admin]

to downgrade the virtual hardware just download and edit the vmx file from the VM datastore folder and change the value from 13 to 10 or 11, save the file to your desktop and upload back into the VM datastore folder, see links below.

How to Downgrade VMware ESXi 5.5 Virtual Machine Hardware Version to 9 from 10

Editing configuration files in VMware ESXi and ESX (1017022) | VMware KB

Note: make a backup copy of the original .vmx file just in case of any problems!

Steve

[peterriederer]

Thank you so much Steve, thats much faster and more efficient. Works like a treat.

After testing with my 2016 Servers, going back to HW11 is not enough, i had to put them back to HW10 in order to get the performance back.

Peter

[mvduijn]

Just be aware changing the vmx file to downgrade a VM is not supported by VMware!

The only supported methods to downgrade a VM are:

• Revert to a snapshot created before upgrading the virtual machine hardware.
• Use VMware vCenter Converter Standalone and select the required virtual hardware version in the Specify Destination wizard.
• Create a new virtual machine with required hardware version and attach the existing disk from the virtual machine.

Mike

[mvduijn]

Vmware contacted us yesterday to inform us about the progress of the performance issue.

Patch 1 wich will resolve this issue is delayed to 27July.

It seems the issue is related to a bug with the UNMAP command.

This process reclaims deleted blocks on thin provisioned disks and luns.

Vmware also suggested a possible workaround for Windows VM's to disable the unmap command inside the windows OS.

You can change a registry key or use VMware PowerCLI.

You can make the changes in the registry by any of these methods.

• Using a registery editor, change this key to 1:
  
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\DisableDeleteNotification
  
• Use this PowerCLI command:
  
  fsutil behavior set DisableDeleteNotify 1

The perfomance seems a little bit better.

Mike

[peterriederer]

Good to know, Thanks Mike.

Nevertheless a delay of another 4 weeks for the patch is really really bad!!!

Going to test that immediatelly. Thats also why Windows Server 2008R2 and lower are not affected.

Peter

[peterriederer]

Works much better now, Machines are fast again.

BTW: Found another thread with the same issues:ESXi 6.5 Slow vms, High "average response time"

[E07256]

This issue seems to be directly addressed in VMware KB 2129176 https://kb.vmware.com/kb/2129176​

"Symptoms

• You notice performance degradation for some client/servers workloads.
• Packets have up to a 0.5 second delay over the expected arrival in the application.
  
  This issue is observed under these conditions:
  
  
  • The guest operating system is Windows Server 2012,  Windows 8 or later.
  • The virtual machine is on hardware version 11/ ESXi 6.0 compatibility.
  • The virtual NIC is vmxnet3 and the driver version is 1.6.6.0 and later.
  • The Receive Side Coalescing (RSC) feature is enabled globally and on the vmxnet3 adapter.
  • This issue is more prevalent when:
  • Running Microsoft SQL/TDS based workloads
  • Using Jumbo Frames
  • The client and server are on two different hosts"

[...]

Resolution

This issue is resolved in ESXi 6.0 Update 2, available at VMware Downloads.

Note: Applying ESXi 6.0 Update 2 may cause a Purple Diagnostic screen.  For more information, see ESXi 6.0 Update 2 host fails with a purple diagnostic screen containing the error: Vmxnet3VMKDevRxWithLock and Vmxnet3VMKDevRx (2144968). This issue is resolved in ESXi 6.0 Patch Release ESXi600-201605001.  For more information, see VMware ESXi 6.0, Patch Release ESXi600-201605001 (2136186).

If you are unable to upgrade, you can work around any performance degradation by disabling RSC on the affected guest operating system.

To disable RSC on the virtual machine, perform one of these options:

• Use the PowerShell command Disable-NetAdapterRsc.
  
  To disable for all interfaces:
  
  Disable-NetAdapterRsc *
  
  To disable for a specific interface:
  
  Disable-NetAdapterRsc -Name Ethernetx
  
  
• Disable RSC globally using the netsh int tcp set global rsc=disabled command.
  
  Note: This command requires an elevated command prompt.

To verify that RSC is disabled on the virtual machine:

• Use the PoweShell command Get-NetAdapterRsc. The IPv4OperationalState and IPv6OperationalState should both be set to FALSE.
• Use the netsh int tcp show global command.  The Receive Segment Coalescing State should be set to disabled.

Note: This change is immediate and does not require rebooting the guest operating system.

Alternatively, you can disable LRO/RSC for all virtual machines on an ESXi host using:

esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0

esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0

Note: This will disable LRO for all virtual machines on the ESXi host. Virtual machines must be powered off and then powered on or migrated using vMotion to another hosts for the changes to take effect.

"

[PGinhoux]

Hi,

I'm surprised by the detail of the fix provided. It talks about a fix in ESXi 6.0 Update 2, whereas this case is related to the 6.5 level.

I encounter the same issue on my ESXi hosts running the update-from-esxi6.5-6.5_update01 level.

On a first ESXi host newly installed I have created a first VM created from an OVF then when I want to log in, I experience a big delay about 20 minutes) to get in.

Can somebody explain me, if possible, more about the fix provided (in ESXi 6.0 Update 2) ?

And what can I look at on my side?

Thanks in advance.