windows server 2022 slow on vmware esxi 6.7

[EmersonBjj]

I have a Hypervisor server: VMware ESXi, 6.7.0, 19195723
Model: ProLiant DL380 Gen10 Processor Type: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz.
I installed a virtual machine with windows 2022 standard with 3 processors and 8 Gb of memory, but the machine is super slow, even the cpu and the memory without consumption, to open anything it takes time and some are stuck, the others come with 2012 and 2016 super fast, I already reset another 2022 the same thing, could anyone help?

[Maarten Van Driessen]

Is there any reason why you're running ESXi 6.7 still? That version is no longer supported and vSphere 7 and 8 offer a ton of enhancements that could resolve your issue without changing anything.

That being said, what does your storage system look like? Could be that that is causing latency or is not up to spec and could be causing your issues.

[NickDaGeekUK]

   

Not sure if this will help you but found this will researching the problem. I am not in a position to try this till the Christmas holidays so if you do try it first would appreciate you letting me know how it goes.

 

I would really like to hear if you have any advice on what appears to be a bug in the HPE driver for ESXi and if this also affects the DL380 from Gen 6 onward too. Do you have a list of recommended patched versions to match ESXi 6.0, 6.5, 6.7 please. 

https://woshub.com/esxi-slow-disk-performance-hpe-gen8/

Excerpt from the article below

Disk driver version: scsi-hpvsa-5.5.0.100-1OEM.550.0.0.1331820

As it turned out, HP has spoiled something in the disk subsystem driver for ESXi 5.5 and disk management became not as efficient as it should. Moreover, as I found out later, this problem also occurred in the HPE ESXi 6.0, 6.5 and 6.7 images.

After talking to my mates and searching the Web, I understood that the matter was in the driver that HPE integrated in its custom image with the installer of ESXi 5.5 or later.

However, this problem can be solved. The Internet community (https://homeservershow.com) has managed to find a driver that really increases the disk performance in HP Microserver Gen8.

Driver version: scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820

 

 

You can download the driver for free from the official HPE website:

https://support.hpe.com/hpsc/swd/…b1dfc5314e02bc01b1436b
Type: Driver — Storage Controller
Version: 5.5.0-88.0(9 Sep 2014)
Operating System(s): VMware vSphere 5.5
File name: scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib (707 KB)

Now we need to install it. The procedure is described below. First of all, check the version of the installed driver, and if it is different, replace it for the right one.

[NickDaGeekUK]

interesting result, I have just checked a 2016 server on the same datastore and ESXi host as the 2022 VM I defragged earlier. Massive difference in disk performance. 40 MB/s read and write.

[Kinnison]

Hello,

If I may interject, my experience with Microsoft Windows Server 2022 in the context of my small vSphere infrastructure is completely different, personally I find it much more responsive than previous versions. However, I must also tell you that the improvements, also in terms of "performance", made in subsequent product versions compared to the one you use have been substantial, then it also depends on how your virtual machines are configured, in relation to the intended workload.

In the end, IMHO, the efficiency / effectiveness of a system is the sum of a whole set of multiple interrelated factors, often small things that we sometimes risk putting off until another time and perhaps ending up forgetting.

Regards,
Ferdinando

[NickDaGeekUK]

 Hi Ferdinando,

thanks for sharing your experience and the useful reminder about the cumulative effect of postponing little things.

Nine times out of ten I would completely agree with you about the build up of little overlooked things. However, in this case I find that difficult.

The server in question is not the latest thing but its a new to me refurbished server. Maxed out RAM and Storage, CPU not maxed on cores though because the massive increase in MS server licences isn't justified.

Clean install of the latest HPE image for that model and generation followed by patching to latest patches then a clean install of 2k22 and 2k16 servers on that also fully patched including latest VMTools due to the security advisory last week. As you can see from the screen shots there is a massive difference between the 2k22 and the 2k16 in terms of disk I/O.

In terms of workload the 2k22 has a single role a Veeam B&R Server and that was dormant during the test. The 2k16 had four running workloads during the test. Configuration for the 2k22 was similar to the 2k16 server in terms of CPU/RAM and Disks. 

QED there is something going on to make this happen. 

I am looking into the HPE SCSI driver and the firmware for the HBA to rule out any underlying issues because the article I quoted says the version that was bundled with the HPE image is suspect all the way from ESXi 5.5. to 6.7 and VMWare's own troubleshooting KB says HBA firmware can also be an issue. Hence my request to Enrique  for any advice regarding compatible HPE firmware and compatible driver versions.

If you have any advice in that area too I would be more than wiling to accept that with my thanks.

[Kinnison]

Hello NickDaGeekUK,

I mean, we can discuss many things but objectively ESXi 6.5 (to which you are referring) as well as being a product that is no longer supported, does not benefit from those substantial improvements, large and small, introduced with subsequent versions, which your system, at least on paper (i.e. for HCL), should support up to the latest build of the 7.0U3 line. The system firmware released by the manufacturer has an equally important role in the iteration and usually when a more recent version is released it is because more or less substantial corrections have been applied to various types of inconveniences.

It's not clear to me what you mean by "maxed out" on RAM and STORAGE but not in CPU count, because in itself it doesn't say much.
And moreover, it is not entirely negligible how the hardware of your virtual machine and/or the operating system itself is set.

The point of my speech is that based on what I have read you can only formulate hypotheses but not go much further.

Regards,
Ferdinando

[NickDaGeekUK]

Hi Ferdinando,  

Thanks for the prompt response .

Maxed out meant maximum ram supported by that chassis and all HDD slots filled with large capacity drives.

Not maxed out for CPU meant I did fill both available sockets with CPUs but did not take the maximum CPU cores per processor that the chassis would support. This is what I mean about the Microsoft Core Tax. Keeping to a pair of 8 core CPUs, or below, means a single 16 core server licence would give two OSE instances per licence. Maximum bang for the licence buck but not maximum performance for the server. Sadly, if you are operating on very limited budgets this is just reality. Most SME IT departments like everyone else are struggling to make ends meet.

 

7.0 U3 would have been lovely too and I totally agree with your comments. Budgets again, I had to pay for new 2022 server licences this year for upgrades to all our VMs.  Having a spare perpetual 2 socket licence sitting there to use for ESXi but only for V6 it was a no brainer. Despite the HPE image dropping in 6.5 I think that licence is good up to 6.7 but not sure if that works for HCL on this hardware. Need to double check that again, so many things to think about when doing this. If the HCL doesn't list your exact model, you are down to checking individual components against the HCL (did this once before a while back for an earlier model, CLI to get hardware IDs out of ESXi) Time and money are always limited resources.

In an ideal world of unlimited budgets I would just buy new hardware, latest hypervisor and server licences all in one go. I get that but it isn't realistic. 

Ho Hum, Rant Over Hopefully next year I can leverage new ESXi licences and upgrade.

 

[Kinnison]

Hello,

One moment please, I use some systems from another manufacturer but of the same generation as yours but I haven't even thought about installing the 3 Terabytes of RAM nominally supported on paper not even when they were brand new. So you see that "maxed out" on RAM can mean many things. So explain for a moment how you achieved the result, for example you won't be using a configuration called 3DPC or just part of the four available memory controllers of your CPU. The issue of ESXi licenses is also a topic of relative value if you don't specify for a moment what license level you refer to.

If we want to talk about this and that, you plan to procure new licenses related to the "Microsoft Windows Server 2022" product because perhaps you use versions that are no longer supported but then as a basis you plan to use an equally unsupported version of ESXi and not without vulnerability, don't you find it a sort of contradiction in terms?

In short, IMHO, if you want some less far-fetched advice, in my opinion you should provide some details, going a little more specifically into how your system is actually configured.

Regards,
Ferdinando

[NickDaGeekUK]

Hi Ferdinando,

I appreciate your offer of assistance that isn't far fetched and am providing the details you requested. 

Your subsequent comments had me really confused at first, until I realised my initial post had a typo in it. Sorry for the confusion but it's a Gen 8 not a Gen 9 (have edited it now) and won't take 3TB of RAM. It will only hold a max of 512 GB but 256 GB was all I could afford.  I believe this is 16 x 16GB which I believe is correct for that DIMM type and size which is what I meant by "maxed out". So with four memory controllers that is 4 x 16GB or 64GB per channel I believe.

Storage 12TB RAID (8 2TB disks in RAID 6) as local HDD storage in a ESXi single datastore. Not maxed out capacity wise but all hdd slots filled for better redundancy (I think I can lose two drives before it goes down) plenty big enough for what we do here.

CPU Intel(R) Xeon(R) CPU E5-2609 x 2. Not maxed out for the reasons specified before cost of MS Core Licencing.

ESXi  licence is 2 CPU vSphere Essentials V6.

There is no contradiction in terms regarding older software versions merely the reality of not having an unlimited budget. I  explained that the upgrade is happening in stages, due to budget constraints. Server licences first, ESXi next then hopefully migration to a newer server after that.

So there you have the configuration that you asked for, and the logic for the legacy software and hardware. Does it shed any light on the situation for you? I hope so.

However, I think we are drifting off topic a little bit. Whilst hardware and drivers may be the answer in the end the initial problem appears to be guest OS based and specific to one OS version in particular.

The OP   stated that his newer Gen 10 server running ESXi V6.7 was having performance issues with 2k22 servers that he was not experiencing with 2k16 or 2k12 on the same server. I corroborated that by stating that on a Gen 8 with 6.5 I was seeing exactly the same thing. 

So the point I think we need to address is what, if anything is different with a 2k22 VM under ESXi 6.x. that causes this when 2k12 and 2k16 don't have the problem. What do you think?

[Kinnison]

Hello,

Now it's a little clearer, so your E5-2609 (V1/V2) has 4 physical cores without hyperthreading and limited to DDR3 memory modules with 800/1066 Mhz speeds, so you have to be careful about how you configure your virtual machines, because depending on your workloads, if you have too unfavorable a ratio between physical cores and virtual cores, performance suffers, with the understanding that you can't expect much anyway.
Your storage is made up of 8 disk drives in a RAID 6 array but it is not clear whether mechanical or solid state and what type, this also affects the final result to a more or less significant extent.

So, to try to narrow down the causes of your problems, in my opinion I would start by consulting the contents of this KB article:
https://kb.vmware.com/s/article/2001003

Forget the experience of the OP, each case is a story in itself as is the concept of performance if there is no objectively reproducible and/or measurable feedback.

Regards,
Ferdinando

[NickDaGeekUK]

Hi Ferinando, 

HDD old fashioned spinning iron 6G SAS 7.2k rpm.
Yes that was the KB I was starting with to troubleshoot. Will let you know what I find with ESXTOP later.

Your point on configuration is well made. I originally came from a Hyper V environment and one thing I did notice then, and assume it may apply to ESXi too, is memory configuration and the NUMA node problem. Found that beyond a certain point adding more RAM to a VM caused a decrease in performance. I found a couple of articles just now about it and ESXi

https://blogs.vmware.com/performance/2017/03/virtual-machine-vcpu-and-vnuma-rightsizing-rules-of-thumb.html

https://shuttletitan.com/vsphere/numa-vnuma-should-we-consider-cores-per-socket-vm-configuration-in-vsphere/

given that I suspect I also need to check the CPU and Memory Allocation on the VM and the Bios of the host for affected 2k22 machines.

[Kinnison]

Hello,

I understand, from a RAID 6 array made up of mechanical ones, perhaps of an old generation, the expectation of adequate performance, especially in writing, in conditions of high IOPS is objectively unrealistic, perhaps the possible cache of the disk controller, assuming in "write back" mode (in case check the battery status) it does something but as they say where I come from, "you can't get blood from turnips".

Regards,
Ferdinando

[NickDaGeekUK]

Hi Ferdinando,

yep similar sayings here about not being able to make a silk purse out of a sow's ear or get blood from a stone 

[Kinnison]

Hello,

In a time of hardship which is not unusual in the micro-business sector it can be a problem but, in my opinion, you should consider getting a more modern system, otherwise having licenses for the most recent versions of the ESXi product isn't of much use if for a reason you may not be able to use them due to lack of support / incompatible hardware.

Regards,
Ferdinando

[NickDaGeekUK]

Hi Ferdinando,

That is a very good point about the lack of support and incompatible hardware. That is the situation we were in when I got here years ago, the hardware wasn't supported for V6 although it was running perfectly. Dropped the support contract with VMWare because we were told it was unsupported hardware. Might have to use this argument with management over budgets and the HCL when specifying a new server.

[E_Espinel]

Hi, my suggestion would be to upgrade to the latest patch level 20497097 and check or install the latest VMtools version for windows 2022.

links:

https://customerconnect.vmware.com/patch

https://docs.vmware.com/en/VMware-Tools/12.1/rn/VMware-Tools-1210-Release-Notes.html#guestop

https://customerconnect.vmware.com/downloads/details?downloadGroup=VMTOOLS1215&productId=742

 

[Maks Roshchyna]

So ideally would be to buy new Servers for vSphere8 or you should convince your director to buy new ESXi licenses if your hardware allowed to upgrade of course. 

[NickDaGeekUK]

Facing a similar problem and no latest tools and patches didn't help me.

Admittedly I am running an older setup, a DL 380 Gen 8 and ESXi 6.5 but the similarity of the problem leads me to suspect that it is Windows 2022 not ESXi that is at fault. I am currently running a defrag of the C drive of a clean install of 2022 because disk access and performance have been so bad. It is managing a best effort of 100% active time and read and write access of 7 MB/s each. The average is more like 4 MB/s. This is ridiculous.

I have a pair of really old DL380 Gen 6 server with ESXi 6 and Server 2012 R2 VMs which is about to be retired as EOL. So I ran them up and did the same defrag of C: on two different VMs.  The are managing 99% active, time albeit with slower average response times, at best efforts of  35 - 60 MB/s and an average of 14 - 28 MB/s. I strongly suspect it isn't ESXI or the Hardware that is the problem, I think it is 2022 itself. Screenshots of the Task Manager Performance tab are attached the 2012 VMs all have higher transfer rates but also a higher average response time (around 500ms). Compare that to the 2022 VM which as an average response time of under 20ms but can't do much better than 4MB/s for transfer rates.

I would love to know if the problem goes away when running under Hyper-V. Can you run a test lab on a server running Hyper V and install a 2022 VM in that?  I am going to try a 2022 install on one of the EOL servers and will let you know.

[NickDaGeekUK]

Hi Enrique,

I have already patched to the very latest VMTools because of the recent security notice, for which I am grateful. Always good to know that our suppliers are on the ball regarding CVEs and patching. 

I have also just tested a much older setup in terms of hardware and ESXi version and with an older MS OS too. It out performs the 2022 VM on all levels.

I am going to run up a 2022 VM on the old hardware and ESXi and test the theory that the problem is within 2022 not ESXi and would really like to know if your team have done, or could do, the same.

Will let you know what I find.

[NickDaGeekUK]

I found a problem that links to an old bug in ESXi that some reports say was not fixed in later releases of 6.7 either. It has helped my issue. I would recommend it as a solution.

The problem is with Windows Server and NTFS having DelteNotify set on by default. It can be disabled with either fsutil or a registry key see below

https://knowledge.broadcom.com/external/article/317995/performance-issues-on-windows-virtual-ma.html

confirmation that this remained a bug until late release of 6.7 and a hotfix but cured in 7.0 below

https://www.reddit.com/r/vmware/comments/fect1t/esxi_67u2_vvol_vmfs_unmap_bug_causing_guest_io/

------------------------------
IT Manager. That person just about managing to keep the IT going with no budget or staff despite the effects of Senior Management and End Users.

IT Worker. A biological system for turning caffeine into code.
------------------------------