Resource Pool - Insufficient Memory Resources error

[tjneese]

We are running a VI3 environment (VC 2.0.2 and ESX 3.0.2). We recently enabled DRS and HA on our cluster (we have 4 ESX servers). I created my first resource pool and moved (3) guests into the pool. There are no other resource pools in the DRS cluster and we had none to migrate when we enabled DRS. The guests are test systems running Windows Server 2003 with a high RAM footprint (3GB each). They rarely see anywhere near that much usage, but "the vendor requires 3GB of RAM...".

My thought was to put them in a resource pool and give them 4GB of RAM as a limit for the entire resource pool (and lower this or raise this as needed). This worked fine when I enabled it. The problem I realized is that I set the Resource Pool with Expandable Reservation enabled. So even if I limit the pool to 4GB, the guests could take away from the parent (in this case the root of our DRS cluster). So I thought I would disable Expandable Reservation. When I do this, it fails and gives me an error "insufficient memory resources". I even tried setting the resource pool to Unlimited and then removing the Expandable Reservation. Same error message.

I'm not sure what is causing the problem as we have enough memory available and the (3) guests in the resource pool are not in testing right now so they are using minimal RAM. Any ideas? Thanks!!

TJ

[masaki]

If I well understand your problem you have created a Resource Pool with 4GB limit with 3 guests inside with 3 GB each.

I can be wrong but you should set the rsource pool's limit to 9 GB.

Try and let me know

[jhicks]

I think this is a brilliant idea. Provide the vendor what they think they need, but restrict them to one overall usage profile. "You need 3 servers all with 3GB RAM, sure, here you go." You're pretty sure they won't need all that memory at once, but you won't have development servers hogging all your resources if they do. Of course, if things don't work well for the vendor, someone WILL complain - but at least they can't complain at provisioning because in the OS they see that you've allocated 3GB per as requested.

However, with "Expandable Reservations," they really aren't being limited. Since you're having problems resetting this, I would just remove the VMs from the resouce pool, destroy it, and then recreate it as you wish. I didn't know there was one, but perhaps there's a restriction on going from expandable to non-expandable. Thought, if your VMs have memory reservations that total more than the Resource Pool they're a member of, it should throw an error. In your VM's resource allocation setup, make sure the total "reserved" memory is under your RP limit. (i.e. If you've allocated 3GB each, and set the resource reservation to 50% of that, you'll need a RP with at least 4.5GB reserved. But if they're set to the default of "0" reserved, then there shouldn't be any conflict.)

[tjneese]

Thanks for the idea. I moved the (3) VMs out of the resource pool (RP) and deleted the RP. I recreated it with expandable reservation box cleared. I then set the RP to a memory limit of 4096MB. I tried to move (1) VM into the pool, but got the all too familiar "insufficient memory resources error" even though the VM I moved had NO reservation and a limit of 3GB for memory. Seems like the RP should have been able to accommodate that. I then set the RP limit to unlimited, but kept the expandable reservation box cleared. Still could not even move 1 VM into the RP. Doesn't make sense to me.

Since we have a support contract, I went ahead and opened a case with VMware support. I post their response after I get this one figured out.

Thanks,

TJ

[tjneese]

After talking with a VMware support engineer, I realize more how this works. Let me try to explain. When you create a resource pool and do NOT allow for expandable reservation, the limit has no meaning. All the resource pool can utilize is the reservation amount. If you DO set expandable reservation, the resource pool will allocate the reservation amount and it will be able to allocate resources from the parent resource pool - provided the parent has unallocated resources. If you then set a limit for the resource pool, it will only get resources from the parent that allow the resource pool to hit the limit.

So in my case, I changed my RP to allow expandable reservations, set a reservation of 1GB and a limit of 4GB. Even though the 3 VMs are each configured for 3GB of memory, the group of VMs will only be allowed (limit) 4GB of memory. Since I can chance RP settings while the VMs are running, I have the option to increase or decrease the RP limit as needed for testing.

Hopefully this will allow me to please the customer who "requires 3GB of RAM" and yet manage the cluster resources better when the memory gets short.

[markt999]

Old thread I know, but I stumbled across the statement "When you create a resource pool and do NOT allow for expandable reservation, the limit has no meaning." and thought that I've never seen that said before. My own testing would imply that the support engineer was in error - if you disable expandable reservation at the RP level, the limit does still have meaning. My test: create a VM with 4GB memory. Reserve a small amount (or indeed none - but I did 512MB) on the VM. In the RP (no other VMs) create a reservation of, say, 610 MB. This will allow the VM to start. Now try with no limit in the RP and monitor the host memory for that VM. You might want to run a memory grabbing tool in the VM to force the issue. In my test, the memory went to about 3900 MB and the system ran fine. Now reboot the VM but with a limit set to 610 MB. It'll be obvious that there's a problem when the machine crawls! I ran my memory grabbing applet which still worked, but the host memory never went above 855 MB (and it took an age to run). My conclusion: the limit is used whether or not you allow expandable reservation. (Either that or the test was flawed, though I have to say the difference in the behaviour of the test VM was palpable.)

http://www.vmware.com/pdf/vmware_drs_wp.pdf (a document I know well!)

This states that "If expandable reservation is not checked, the system considers only the resources available in the selected resource pool". Not the same thing as "the limit has no meaning".