ESXi 7 max reservable RAM

[meinrad_]

Hi

I'm new to VMware and EsXi, so far I only used Linux KVM.
I've got a HPE Microserver GEN10 with only 8GB of RAM. I'm using ESXi 7 as a hypervisor. Most of the time I'm running a small single Linux guest on it. Because I'm using hardware pass through all the memory of the guest has to be reserved. Since it's in most cases the only guest, I'd like to assign 4GB of RAM to it. But for some unknown reason ESXi lets me only reserve 2GB of RAM for it. If I run other guests without memory reservation and hardware pass through, ESXi lets them use up to 7GB of the 8GB of physical memory.

How can I reserve more memory to a guest?
What are the limitations and where can I find the corresponding documentation?
Is there a way around reserving all of the memory for a guest with hardware pass through?

Thanks for your help!

Meinrad

[jburen]

The best place to start if you are looking for documentation is https://docs.vmware.com/en/VMware-vSphere/index.html When it comes to reservations there is no absolute limit. The only limit is what you have available in your ESXi host: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-117972E3-F5D3-4641-9EAC-F9DD2B0761C3.html

In this example, I have configured 8 GB memory for this VM. When I want to reserve memory, the maximum for this reservation is 8 GB. If I want to reserve more I must increase the amount of assigned memory first.

 

[Duncan Epping]

it probably will not allow you to reserve more as there are a couple of other things that require memory, like the hypervisor itself, and the overhead reservation for the VMs. 

Why would you set a reservation? ESXi is pretty good at memory management, just let it do it's thing!

[meinrad_]

Thanks for the detailed answer.

Your commands showed me that the host and the system alone use about 5.5GB. And that I technically could increase the reserved momory of my VM to almost 3GB.

$ vsish -e get /sched/groups/4/stats/capacity | awk -F '[ :]' '$4 ~ /mem-unreserved/ {print $5/1024" MB"}'
1007.29MB

$ memstats -r group-stats -s gid:name:min:conResv:availResv:memSize -u mb | sed -n '/^-\+/,/.*\n/p' | awk 'NR == 3 || length($3) !~ 1 {print $0}'
---------------------------------------------------------------------------------
     gid                         name        min    conResv  availResv    memSize
---------------------------------------------------------------------------------
       0                         host       7606       6599       1007       4149
       1                       system       1586       1570       1023       1431
...
  627625                   vmm.206715       2048          0       3055       2048
  627626               vmmanon.206715         35          0       1041         18
---------------------------------------------------------------------------------

 

I guess there is no way I can degrease that. Therefore I think I'm going to invest in some RAM. Does the amount of memory reserved by the system
and host stay more or less constanst if I double the RAM or does it also increas dramatically?

 

I know this is not the typical usecase for ESXi and that it's ment for way bigger servers, but I decied to use ESXi because it is well supported by the
hardware (HPE Microserver GEN8) I use and I wanted to learn about it.

[Valentin Bondzio]

I just looked over the code (very briefly and very likely very incompetently) and I don't see how you could disable the need for pre-allocation on pass through. You could try to manually set "sched.mem.prealloc" and "sched.mem.prealloc.pinnedMainMem" to false in the vmx, I'm not sure if that will be honored. In any case, it would be "sowas von unsupported", if it works at all.

There is also a way to reduce the reservation of certain worlds, we do mention the use of "localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig" in a bunch of KBs, it only takes a bit of trial and error and interest to deduce the usage and reduce the reservations of worlds where the memSize is waaaaay below the min.

Again, in no way recommended and you should really get two DIMMs of 8 GB instead of doing that, my answer is really just to satisfy your curiosity.

[Valentin Bondzio]

A reservation is a promise, i.e. that while the memory might not be consumed / resident for the "reserver" right now, any allocation would be granted and if necessary forcefully taken away (reclaimed) from someone else. ESXi has a certain amount of default reservations for the purpose of stable operations, you are of course more likely to collide with those on a host that has as much memory as my phone.

The max you can reserve, is whatever is currently not reserved in the user pool (where the VMs go):

vsish -e get /sched/groups/4/stats/capacity | awk -F '[ :]' '$4 ~ /mem-unreserved/ {print $5/1024" MB"}'

Note that you _shouldn't_ reserve all of that in any environment that you care about because under some circumstances ESXi might need to reserve more (e.g. heavy IO workloads) and if it can't, will PSOD.

To list all current reservations > 9 MB:

memstats -r group-stats -s gid:name:min:conResv:availResv:memSize -u mb | sed -n '/^-\+/,/.*\n/p' | awk 'NR == 3 || length($3) !~ 1 {print $0}'

any form of passthrough requires memoy preallocation (since 6.5) which in turn requires a full reservation.

[Duncan Epping]

Ah, missed the "passthrough" part of the post

[Duncan Epping]

Just to be open and honest, 8GB does not give you a lot of memory to work with in general to be fair...

[Valentin Bondzio]

Definitely agree but there is an argument to be made that for corner (or edge, wink wink) cases like these we could at least look at our default reservations and potentially reduce the large consumers (hostd / vpxa / logger) based on actual estimated consumption or fully disable other services if not in use (e.g. iSCSI). A fine line to walk for sure and should definitely not be done on servers that don't fit into a shoe box.

[meinrad_]

I know, but since I only need very limited amount of memory I thought 8GB is going to be enough. I was not expecting ESXi to reserver 6GB RAM for it self.

[Duncan Epping]

Keep in mind that ESXi is designed and intended to be used in datacenters, where memory/cpu resources are typically not a problem. Hence the reservation of resources etc.