vsphere 7 and esxi memory

[billdossett]

Hi,  I've recently started working with vsphere 7 on my lab cluster at home.  I have 4 dell optiplex 9010 workstations and they each have 32GB of memory... on vsphere 6.7, this was fine and I never had any memory issues in particular and ran loads of development vms on it for testing and development.

This is an HCI cluster and I recently upgraded to vsphere 7... all good, but, all of my hosts are showing around 17GB memory in use and under perf memory it says that is the amount of host physical memory consumed for backing up guest physical memory pages. This is on a bare cluster with just the  vcenter appliance loaded on it.  When I start loading up components on it, I start getting memory warnings immediately.  Now I know 32GB of mem isn't much for esxi hosts, but these nodes are maxed out and I can't add more and its just a lab... is there anyway I how much memory is consumed for backing up guest physical memory pages?  I've been googling this and not finding much.

Thanks

Bill

 

[a_p_]

When you say HCI cluster, are you using vSAN?
In this case https://kb.vmware.com/s/article/2113954 may explain the memory usage.

André

[billdossett]

Yes, Hyper-converged Infrastructure is VSAN.  The cluster has always been VSAN, under 6.7,  it didn't use this much memory, but this has given me a clue to look for, thanks, will report back if follow-up gives results.

[billdossett]

So, I did a full reinstall of my cluster, however this time I did not turn on dedupe/compression - my 6.7 VSAN did not use dedupe/compression.

So that was not the issue...   6.7 VSAN used a fraction of the memory that 7 VSAN cluster is using on esxi hosts...  I had that running for like 3 years I think and I would have definitely noticed if it was taking that much memory... anyone?  any ideas on how I can reduce that?  I'm working on a project that requires vsphere 7, but if this is the case, when I am done I am going back to 6.7 as this is crazy that it eats more than half the memory on my hosts.

[Valentin Bondzio]

How much of that is vSAN? See how to check e.g. via: https://cormachogan.com/2020/03/13/track-vsan-memory-consumption-in-vsan-7/
Can you connect via SSH and run the following 3 commands and post the output here? (mono spaced / e.g. Courier New)

memstats -r vm-stats -s name:b:memSize:max:consumed:ballooned:balloonTgt:swapTgt:swapped:active:touched:zipped:zipSaved:shared:zero:sharedSaved -u mb 2> /dev/null | sed -n '/  \+name/,/ \+Total/p'

memstats -r comp-stats -s total:minFree:free:numHigh:numClear:numSoft:numHard:numLow:memState -u mb 2> /dev/null | sed -n '/^-\+/,/.*/p'

memstats -r group-stats -g0 -l2 -s gid:name:parGid:nChild:min:max:conResv:availResv:memSize -u mb 2> /dev/null | sed -n '/^-\+/,/.*\n/p'

[billdossett]

sorry, was on different project - finally getting to this

I have no VMs other than the vcenter on this cluster... the esxi hosts went from like 1-2GB consumed to 16+ GB consumed when I configured VSAN, so **bleep** near all of it.  I was reading in the other posters link I think, that it reduces memory if you have 32 GB or less... I am SURE that this was not the case in 6.7.   Like I say I had a lot of dev stuff running on it and never had memory issues.

would sure appreciate any help on getting this down as I can't get much running on it as is.

 can't find a great way to format this output! sorry, but doesn't want to monospace.

name b memSize max consumed balloonTgt ballooned swapTgt swapped touched zipped zipSaved shared zero sharedSaved
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
vm.535188 y 128 -1 128 0 0 0 0 22 0 0 1 1 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 128 128 0 0 0 0 22 0 0 1 1 1

------------------------------------------------------------------------------------------------
total minFree free numHigh numClear numSoft numHard numLow memState
------------------------------------------------------------------------------------------------
32654 940 15803 1 0 0 0 0 High
------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------
gid name parGid nChild min max conResv availResv memSize
--------------------------------------------------------------------------------------------------------------
0 host -1 4 32492 32492 21515 10977 18126
1 system 0 10 17661 -1 17645 10993 17118
2 vim 0 4 0 -1 3802 10977 822
3 iofilters 0 3 0 -1 25 10977 12
4 user 0 1 0 -1 28 10977 176
--------------------------------------------------------------------------------------------------------------

 

 

[Valentin Bondzio]

This should have been similar on 6.7 ...

Does the current consumption match what you calculate here? https://kb.vmware.com/s/article/2113954

Maybe somehow the ESXi "perceived" memory was below 32 GB when you had 6.7 installed? Can you boot with the following kernel option set?

memmapMaxRAMMB = 31744

If that didn't work, any chance you could _easily_ re-install 6.7 and run the same commands? (but share the output via e.g. pastbin)

Test monospace formating, manually added pre tags:

 _______  _______  _______  _______ 
|       ||       ||       ||       |
|_     _||    ___||  _____||_     _|
  |   |  |   |___ | |_____   |   |  
  |   |  |    ___||_____  |  |   |  
  |   |  |   |___  _____| |  |   |  
  |___|  |_______||_______|  |___|  

 

[billdossett]

definitely was not on 6.7, will look at the calculations... I can actually load 6.7 fairly easily now as I have a complete pxe imaging appliance setup so it only takes me about 30 minutes, but I need to finish testing some stuff under 7.x if I can get it to work with this little memory... and I can check the kernel with max mem adjusted under 7.x first... and ok will use tags to format thanks!

[billdossett]

ok, finally got a chance to look at this again... I did the math and the consumption conforms to that, 14GB

and I tried reducing maxmem value to less that 32GB and as expected it had no affect - as the math works out correctly...

So, I will eat my left arm if 6.7 used that much memory - I know it couldn't have as I would have noticed it, the same as I am under 7.x... I will hopefully have time to load up 6.7 later today or tomorrow.  I see that the equations refer to pretty much any version of vsan...  so it should be the same...  will see if I have to get some mustard for my left arm.

[Kahonu84]

I hope you're right handed.

[Valentin Bondzio]

Reading up on it, reducing the memory should have an effect, it is just proportional.

https://blogs.vmware.com/virtualblocks/2015/11/11/vsan-design-sizing-memory-overhead-considerations/ Scenario 3 explains it well:

"In systems with less than 32GB of RAM, the amount of memory used will be scaled down linearly according to the formula ( 32 / SystemMemory ) where SystemMemory is the amount of memory in the system in GB. Thus, if the system has 16 GB of RAM, the amount of memory consumed will be 1/2 of the output given the formula use to compute memory consumption. If the system has 8 GB it will be scaled down by 1/4."

If you want to compare the difference in consumption based on individual components, you might want to compare the output of sth. like:

memstats -r group-stats -g0 -l7 -s gid:name:min:max:minlimit:shares:conResv:availResv:memSize -u mb | sed -n '/^-\+/,/.*\n/p' | awk 'NR == 3 || $3 !~ /^0/ {print $0}'

This will only print worlds with a memory reservation, change the "$3" to "$9" to match any world that consumes more than 1 MB (or adjust the match expression to whatever you like).

[billdossett]

I tried reducing the kernel memory - I made sure I saw the memory go down in the host accordingly - it didn't make any difference in the esxi host.  The default value for maxmem was huge, so I thought maybe that was going to make a difference but it did not.

I finished deploying esxi 6.7 last night and the vcenter and I'm just about to build the vsan now...  I have added the hosts.

host01 is using 25.85 GB out of 31.89 GB it is running the vcenter and has vsan enable as the first host in the vsan deployment.

host02 - 03  is using 7.05GB our of 31.89 GB  VSAN not configured

and after VSAN is configured they are all using 16GB out of 31.89GB, so I was wrong - time to butter up my left arm I guess... hard to believe I never noticed that memory use over 2 or more years of running it... I did have a lot of VMs on it and I guess I just always assumed they were using a the memory.  It only now when I am trying to deply NSX-T and the appliances in NSX-T are very demanding memory wise, that I noticed I guess.

Was good to do the calculations and see anyway.... thanks.