Problem with memory tiering and client VMs

[fehret]

Hi there,

It's in my homelab, so YES, it is unsupported hardware! But I'll be happy if somebody has some new ideas to investigate. 😊

The context : 3 Minisforum MS-01 cluster with ESXi 8.0 U3g, all cores active (performance & effiency) and 96GB of RAM - All identical - No VSAN

I purchased 3 dedicated NVME disks and I wanted to activate memory tiering.
Starting that point, I started to have issues with some VMS with one common thing : all are Windows 11 VMs.
All other VMs didn't show any sign of issue.

Symptoms:
- not available anymore, no network connectivity. Can happen when using it.
- black screen in vCenter console
- no VMtools reporting anymore
- very high CPU consumption, which turns the cluster in a very bad state (DRS score below 30% when usually >95%)
- comes back like nothing happened when migrated to another host, until it starts again. There is also no trace in VM's event viewer.

What I've tried so far but not working:
- host afinity rule (VM will become unavailable at some point, even if not moving, even if powered on on host where it should)
- changed CPU allocation at start (both "assigned at power on" or else)
- different memory tiering ratios 
- changed scheduler settings on VM :  bcdedit /set hypervisorschedulertype classic

As soon as I turned the tiering feature off, it is OK again (and it has been rock stable for months before)

Any idea I could try out? Did somebody achieve proper memory tiering in a mini-PCs lab?

Thanks in advance! 😉

-------------------------------------------

[Duncan Epping]

What kind of NVMe devices are you using? On Reddit a while back someone reported issues as well, and in their case the NVMe were thermal throttling as a result of the system overheating. 

-------------------------------------------

[fehret]

Hi Duncan,

I'm using Samsung 990 Pro disks (1 TB).
There were installed on the fastest port on MS-01s (the Gen 4 4x)

I don't see why only Win11 VMs only would be impacted os much by thermal throttling, but for sure, they are reacting differently compared to my other workloads (mostly Almalinux + Win servers).
Would you have any clue about the architecture difference that could lead to such?
Also: is there any way to check if ithermal throttling is happening on the host?

What I will try next on my side is to disable effiency cores on one host, put Win11 VMs on this host and see if it is still making such issue.

But of course, if you have any other idea that I could try, I'm open to any suggestion! 😊

-------------------------------------------

[Duncan Epping]

Ah, you got P and E cores, ESXi doesn't support that, that could indeed also be the problem. Sorry completely read over that. The 990 pro normally shouldn't be an issue. Not sure if you can detect thermal throttling with your hosts, but in ESXi you may be able to check the temp on the system. The reddit post on the issue is here by the way: https://www.reddit.com/r/vmware/comments/1mlutso/esx_9_nvme_tiering_literally_unusable_performance/

-------------------------------------------

[fehret]

Hi Duncan,

Yesterday, I tested deactivating the E cores, but unfortunately, it didn't change anything. My client VMs crashed after a while.

I think it's a sinking ship because one of the options I still have is to upgrade to VCF9 and hope it handles it better, but since my employer is currently unwilling to invest in VMware until they have final pricing for the licences and an overview of the alternatives, I was unable to renew my certification in time and, as a result, I am not eligible for VMUG...

In short, this is not the subject at hand, but apart from the option of reducing the ratio (as suggested by Dave) or leaving a host with memory tiering disabled on which I will run these VMs, there is not much else that can be done.

But in the process, I found something interesting:
- when E cores are enabled, my host displays 14 logical processors, so like each actual physical core
- but when E cores are disabled, there are 12, which corresponds to the 6 P cores with hyperthreading

So, I'm really wondering which settings will perform best! 😊

-------------------------------------------

[fehret]

Hi Duncan,

Some news about P & E cores : so I did disable all E cores on one host and created afinity rules just for this host and enables memory tiering just for that host.

Unfortunatelly, this did solve the issue and all 3 client VMs I have were not available after a while.

So I guess this is a sinking ship as, except if somebody comes up with some "magic parameter" for Windows client VMs, I'll remain stuck.

And I'm unfortunatelly not v9 "elligible" as my employer didn't want to invest money to renew my certification, so I can't apply to VMUG licences anymore. Maybe next year but if alternatives are more ready, we might migrate within 2-3 years, which is exciting but also sad at the same time considering what I dedicated to VMware for almost 2 decades.

BUT, I learned something interresting switching of the E cores :

 host looks when all cores are activated, my host looks like a 14 cores server:

• But when I disable E cores, it looks like a 12 cores???

Which is interresting because according to Intel website, with Hyperthreading, the total amount should give 20 threads (6x2 + 8) meaning that P cores are not seen as Hyperthreaded IF E cores are enabled. I'm wondering what would be the perf difference between the 2 settings.

https://www.intel.com/content/www/us/en/products/sku/232135/intel-core-i913900h-processor-24m-cache-up-to-5-40-ghz/specifications.html

I'll try to reduce the ration as suggest by Dave but I might ending up with one host out of 3 not using memory tiering and run Win11 VMs on that one host.

-------------------------------------------

[David Morera]

What ratio are you using? The differences between the tech preview version (8.0U3) and 9.0 are night and day in many areas including performance. The recommended DRAM:NVMe ratio for 8.0U3 is 4:1 so only 25% comes from NVMe. 
Are you actually seeing pages move to NVMe? Is the NVMe device showing r/w? 

I would start by putting the ratio back to 25% or lower even. Changing this ratio higher in 8.0U3 could be the culprit, in addition to unsupported devices.

-------------------------------------------

[fehret]

Hi Dave,

I was using 100% but my current workloads are really not high on those hosts.

I don't even have enough workload to fll my 96 GB of RAM yet, I wanted memory tiering to spin up labs and nested VMs (which is also NOT recommended, I know)

I still have an old Dell server with enough RAM, but in Europe, electricity bill hurts a little and it is really not practical to work on labs when you need 15-20 minutes to spin everything up.

Concerning checking if pages are moving to NVME, do you have any recommandations/docs how to do that?

But I'll try the ratio down and let you know, thanks for the suggestion! 😉

-------------------------------------------

[fehret]

Update ratio : not working either with 25%... below, there is no interrest! 🤣

-------------------------------------------

[Duncan Epping]

I have not tried this on 8.x, but on the commandline you can look at "memstats -r vmtier-stats" to see if the tier is actively being used or not. Note, and this is often forgotten, we tier-out memory when there's host pressure, not just because we can. There needs to be a reason, so if there's no reason, you won't see tiering.

-------------------------------------------

[Duncan Epping]

I just tested it, and if you go to the command line you can indeed see the stats. Just as an example, I powered on a lab in my own environment with Memory Tiering, and I deployed some VMs and overloaded to host to ensure there was memory pressure, and below you can see that there are memory pages stored in Tier1 (NVMe).

memstats -r vmtier-stats -u mb -s name:memSize:isTiered:active:tier1Target:tier1Alloc:consumed:tier1Consumed:tier1ConsumedPeak

 VIRTUAL MACHINE MEMORY TIER STATS: Wed Dec 17 15:27:27 2025
 -----------------------------------------------
   Start Group ID   : 0
   No. of levels    : 12
   Unit             : MB
   Selected columns : name:memSize:active:tier1Target:consumed:tier1Consumed

--------------------------------------------------------------------------
           name    memSize     active tier1Target   consumed tier1Consumed
--------------------------------------------------------------------------
      vm.533611       4096        384           0        371             5
      vm.533612       4096        382           0        368             4
      vm.533613       4096        379           0        365             4
      vm.533614       4096        353           0        336             1
      vm.533615       4096        386           0        374             5
--------------------------------------------------------------------------
          Total      20480       1883           0       1812            18
--------------------------------------------------------------------------

-------------------------------------------

[fehret]

Hi Duncan,

I tested the command and I can confirm tiering works if I put "pressure" on it.

In the example below, the first command is when all 3 hosts are available and the second one is when one is in maintenance. We can clearly see the difference.

BTW: nice article on your blog! 👌😉

But regardless if I'm in situation with tiring occuring or not, my Win11 VM will start getting crazy (of course, except when memory tiering is completelly off)

But chat GPT gave me some interresting thoughts (no real solution) that it might be related to vTPM but I can turn this issue in all directions, I don't understand why ONLY Win11 VMs will have such issue. I've a couple of Win2019, Win2022 VMs and none of them had issues, despite they also encrypted and have vTPM. Also GPOs are almost all similar as I use CIS benchmarks for all.

I'll try a brand fresh VM soon to see if it has something to do with legacy stuff (the current VMs were upgraded from Win10 to Win11) but still no clue why for the moment! 😊

PS: the fact it is crashing when all 3 hosts are on and no memory tiering is occuring rules out thermal throttling IMHO.

-------------------------------------------

[Duncan Epping]

I am just happy it rules out issues with memory tiering :-) 

-------------------------------------------

[fehret]

Well, without tiering active: no problem and with memory tiering active : PROBLEMS ! 🤣

I still need to test the fresh VM thing... I'll let you know

-------------------------------------------

[fehret]

Spoiler: new fresh Win 11 VM didn't solve the issue 🙃

-------------------------------------------

[fehret]

Hi there,

So return of experiece: frustrated! 😑

I've configured 2 out of 3 hosts with memory tiering and put my Win 11 with affinity rules on the 3rd.

When I start labs and stuff, you can see that vSphere will "fill up" the physical memory on all 3 hosts instead of using the memory tiering, resulting having the 3rd host's RAM completely full and the 2 with memory tiering not really used (even after the labs bootup/firsrt rush)

Not what I expected... 😁

-------------------------------------------

[Tony Fahlstedt]

Hello

@Duncan Epping
I'm taking a chance here and hijacking the thread as I have ran into somewhat the same issues as @fehret
I have a homelab with below setup.
2x Intel NUC 11  Core i7, 64GB RAM
ESXi 8.0 U3i
For tiering I use a dedicated  Samsung_SSD_970_EVO_Plus_1TB nVME

vCenter 8.0 U3i
An amount of Windows 2025 / Linux VMs, DCs, Fileservers, nested labs and so on.

Following the nVME tiering preview guidelines I have set below advanced settings on the hosts.

This gives me 318.33 GB RAM / Host which has worked just great from day 1.

Yesterday I deployed an new vCenter 9.0.2 and joined the ESXi hosts which worked fine.
Decomissioned vCenter 8.0 U3i
Now when I try to migrate Windows 2025 Servers with secure boot / VBS activated I get this error.

The operation is not supported on the object. Placing or Powering on a VM DC02 that does not support user level monitor on a host esxi01 with Software Memory Tiering enabled is not allowed.

I also got this error when trying to start a VM that has secure boot / VBS activated
No host is compatible with the virtual machine.

If I disable VBS of course I need to reboot it and disable VBS in the OS also.
If I disconnect and remove the host from vCenter 9.0.2 the VM starts without issues.
However with it started now I could not join the host to vCenter 9.0.2
There seems to be some configuration on the cluster that is causing the problems.
I wanät the Windows Server Vms to keep using VBS.
Do you have any suggestions on how I can solve this without the need for rebooting the hosts....if possible at all ?
I am planning on upgrading the ESXi hosts to 9.0.2 also but have not dared to try it yet.
Perhaps my only solution is to disable the tiering before I try to upgrade and prey the VMs will start :)

-------------------------------------------

[Duncan Epping]

VBS and Memory Tiering is an unsupported combination unfortunately. 

-------------------------------------------

[fehret]

That is key information, I don't know why it didn't came up before! Thanks! 😮

But that makes it weirder as most of my Windows VMs have VBS activated, but as I didn't really power them off before, they run without any issue, at least Windows SErver so to say.

I completelly switched off Memory Tiering for a while, but I'll give it another go. I'll keep you posted. 😉

-------------------------------------------

[Tony Fahlstedt]

Is there any information you can share here regarding improved VBS / Nested support in upcoming v 9.1 ?

-------------------------------------------

[Duncan Epping]

No, I cannot comment on any future improvements on a public forum. If you still are a Broadcom employee it is best to ask these questions via Google Chat :-)

-------------------------------------------

[Tony Fahlstedt]

Understood :)

Just as @fehret says, it does not help to switch off VBS / Credential Guard following Microsofts guidance.
The VMs does not start, below error message persists.

The operation is not supported on the object
Placing or Powering on a VM "vmname" that does not support user level monitor on a host "Hostname" with Software Memory Tiering enabled is not allowed.

I have gone through the .vmx file trying to find any of the not supported parameters without any luck.
I'll ask for further advise in Google Chat :)

Br / Tony

-------------------------------------------

[fehret]

@Duncan Epping, is there any logs that we can look for specific memory tierings issues?

I wanted to try this afterrnoon to power off all client VMs, activate memory tiering on all hosts, but I might end-up at the same stage as @Tony Fahlstedt IMHO.

-------------------------------------------

[fehret]

Well, switching off VBS did not solve the issue with Windows client VMs, bummer, I had some hope! 😒

Any idea left? 😊

-------------------------------------------

[Duncan Epping]

Not sure which problem you are experiencing now Fehret.

-------------------------------------------