Need help with esxtop memory values

[Raudi]

Hello,

only a short question, i'm confused regarding the TCHD value of esxtop in a vSphere 8 NVIDIA vGPU environment.

We have several vGPU VM's and all VM's have a fix TCHD value which is 75% of the configured memory, even when i use a tool inside the guest OS to fill up the memory to nearly 100%.

All VM's shows in the vCenter the same fix value for active memory. For example a VM with 80 GB RAM, it shows 60 GB active memory.

Why do i see every time a fix value of 75%?

Kind regards
Stefan

-------------------------------------------

[snapfriend]

Hello  @Raudi

Please check if VMs are having reservation set on VM Resource Pool then it might lock on 75%. I didn't find any articles which relates to such lock system to vGPU VMs (I might be wrong but I didn't find it). Please check below VMware reference article for esxtop query.

https://knowledge.broadcom.com/external/article/382249/using-esxtop-and-interpreting-esxtop-sta.html

https://knowledge.broadcom.com/external/article/375894/understanding-esxtop-memory-parameters.html

-------------------------------------------

[Raudi]

Thank you for the answer, we have no resource pools configured and the VM has 100% memory reservation because of the vGPU. This it looks like in vCenter:


And a esxtop example:


Strange behavior...

-------------------------------------------

[snapfriend]

Hello Again,

Kindly look into VM Edit settings -> Click on Memory -> See if there is any value in field "Limit". I think that is set to VM level to limit the VM on 60GB. Limit means it won't go beyond that. But it is strange. Even, you can see OS level as well if there is any limit set on OS level.

Please find keyword "limit" in below VMware article to know limit value

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-monitoring-and-performance/performance-monitoring-utilities-resxtop-and-esxtop/using-esxtop-or-resxtop-in-interactive-mode/memory-panel.html

-------------------------------------------

[Raudi]

Hi,

yes that was my first idea, here is nothing configured and i just checkt this in esxtop too:

"-1" -> no limit

And even if there was a limit i must see this inside the OS and in esxtop that the ballon driver will consume memory.

The confusing thing is, i used the tool "Not My Fault" to fill the memory of the VM nearly to 100%, no problems, but the reporting in esxtop is every time the same.

-------------------------------------------

[StephenMoll]

For memory to have been 'touched' this has to have happened within a definable time frame doesn't it?

There must be a number of pages of memory that once populated, get touched infrequently. So even though you might run a tool to 'touch' all unused memory to try and get 100% utilisation inside the guest, because in your case 25% of the memory pages are occupied by stuff that isn't doing much, you won't ever get to 100% active.

I think a memory page to remain marked as 100% active it has to have been accessed in each world cycle. Pages that start not being accessed in each cycle see a decay in its activity score. So all we can say is that your tool can maintain 100% activity in the memory pages that were not previously occupied. The other 25% is in use, but not sufficiently active to rise to 100% active, i.e. being touched every world cycle.

The nice 75% number is probably coincidental, and I would expect to see a different value depending on OS and application/feature load.

-------------------------------------------

[Raudi]

I tryed to get more memory used to jump over that 75% border, which is for the 50GB VM at 37,5GB. I used the tool NotMyFault from sysinternals.

And the memory looks like this:

But the TCHD value was still fix at 37,5GB.

-------------------------------------------

[StephenMoll]

I don't think "Not My Fault" is doing what you think it does.

I think it just reserves page pools by allocating some sort of pointer, and then not releasing the pointer in order to simulate memory leaks in simulated 'bad' kernel-mode driver. Having a memory page reserved this way marks it as being in use, and does remove the page from the available pool, but does NOT mean the page is active. In order for that memory page to appear active to ESX, it needs something to be happening frequently enough that it is captured as an accessed memory page at every world cycle in the hypervisor. The moment a page stops being seen in each cycle, the statistics will show a decline in the 'activity' stats for the page, and when they drop far enough the page will be marked as 'inactive' or 'cold'. You see this in your stats in that you are only able to keep 75% memory pages hot, 25% are cold and therefore inactive.

It looks like you should be using Mark Russinovich's "TestLimit" app instead. Testlimit - Sysinternals | Microsoft Learn

This will leak and more importantly 'touch' memory, and I think that is what you are really wanting to do.

-------------------------------------------

[Raudi]

I tested this tool:

Testlimit64.exe -d 44000 -c 1

Leaking private bytes with touch 44000 MB at a time...
Leaked 44000 MB of private memory (44000 MB total leaked). Lasterror: 0

But esxtop still shows a TCHD of 38590.72

-------------------------------------------

[StephenMoll]

That's a small increase, and implies that TestLimit has only managed to touch a relatively small number of memory pages more than Not-My-Fault was able to. It would also suggest that Not-My-Fault is more 'active' in its reserved pages than I would have imagined. 

The rest of the pages are beyond the reach of theses tools, and are simply pages being used the OS and applications at a low enough rate to keep them cold and inactive from ESX statistical view.

-------------------------------------------

[Raudi]

I used notepad and created a 900MB text file. Then i opened the textfile again and again until the physical memory was full, then windows has started swapping. I still continue opening this file until i got an error that the file can't be opened.

TCHD is the whole time at the same value...

-------------------------------------------

[MORACY FERRARI]

I was reading all the previous comments and tests and results, and considering that and reading this KB article: "Using ESXTop and Interpreting ESXTop Statistics" I started to think we should question how the TCHD value is calculated instead of if the tool used to increase the usage/activeness of the memory is doing its job or not. 

In the above KB it says under Groups Statistics - TCHD: ..."estimated by VMKernel statical sampling. VMKernel estimates active memory usage for a VM by sampling a random subset of the VM's memory..."

So, in order to calculate the TCHD value, vmkernel uses methods like "estimated", "sampling", "random subset of VM's memory", all these indicate vmkernel does not observe all pages directly at each interval; instead they sample a random subset and extrapolate to estimate overall active memory, and also, it "won't be exact, but becomes more accurate over time".

In statistics, we don't survey all people in a country to estimate opinions in a voting, but a good sample may be 0.1%, depending of how the sample as selected, and then extrapolate to the rest of the country and we get accurate results with low error margins.

Suppose the tools you used are indeed touching 100% of memory pages, not all pages are repeatedly accessed within the sampling and averaging window the vmkernel uses to calculate the TCHD value. In that sense, one-time touch ≠ active, sequential scan ≠ active, meaning temporal locality matters.

And, even if 100% of pages are touched by the tools, only a subset may qualify as active under vmkernel's definition (frequently accessed within a time window)."

The fixed 75% you found seems to be an emerged empirical pattern rather than a defined algorithmic or configuration limit. This strongly suggests the influence of:

• Sampling bias (not all pages are observed in one cycle since random page sampling occurs)
• Temporal filtering (only recently reused pages count) and 
• Moving averages (peaks are smoothed down) 

-------------------------------------------

[Raudi]

Now, after i tryed to explain my problem for the x time to the support i got a helping answer from them, it goes in the direction of your answer.

In short:

This is a cosmetic reporting discrepancy in how the Active Memory heuristic tracks "touched" pages for VMs with 100% reservations. Because your GRANT and CNSM values are correct, and your internal OS testing confirms the memory is addressable, there is no loss of performance or available RAM.

More detail:

TCHD in esxtop represents VMware's estimate of Active Memory. Unlike GRANT or CNSM (Consumed), which reflect actual physical pages allocated to the VM, TCHD uses a statistical sampling algorithm to estimate how much memory the Guest OS is actively reading or writing.

• Sampling Bias: To maintain host performance, ESXi does not monitor every memory page. Instead, it samples random pages to estimate activity.
• The vGPU Factor: vGPU-enabled VMs require a 100% Memory Reservation. This "pins" the memory to physical RAM. In many ESXi versions, the heuristic used to calculate "Active Memory" for pinned, high-performance I/O workloads tends to plateau at a 75% estimate. This is a reporting artifact of the sampling algorithm and not a physical limit on the RAM available to your VM.

My problem was, i had the fear that here is something misconfigured, because why i don't see more meomry active than 75%? At the time we had 60GB for each vm, we had more times a week a bue screen inside the VM, Most something with directx, and then we found an event log entry with out of memory, so we added some memory to the vm's and the problem was gone.

But even with only 60 GB the active memory, the active memory was never above 75%, so i was searching for problems, like reservations, limitations etc. something that helps me to understand, why i have this 75% static value here...

Thanks to all for thrying to help me!

-------------------------------------------