VMware vSphere

 View Only
Expand all | Collapse all

Memory Resource Allocation - Serious Bug?

  • 1.  Memory Resource Allocation - Serious Bug?

    Posted Jan 04, 2008 04:04 PM

    So I'm looking over Memory Resource Allocation, and I notice most VMs have the Limit set to the memory assigned to the VM (i.e. 1024 MB or 2048 MB). A few apparently random VMs have the limit set to Unlimited though. Anyone know what causes this? And if it would cause any problems?

    Second thing I notice, and this could be a real problem... There are a couple of VMs that I built with 1024 MB of memory, and then later I increased their memory to 2048 MB. Well, the Resource Allocation number didn't change! So the limit is still set 1024 MB...

    What happens when a VM that has been assigned 2048 MB (and that's what the VM's OS thinks it has), has a Resource Allocation limit of 1024 MB? I'm thinking some serious swapping is going on (which could explain some performance issues I've been having). Anyone aware of this bug? I'm using VC 2.0.2 50618



  • 2.  RE: Memory Resource Allocation - Serious Bug?

    Posted Jan 04, 2008 04:27 PM

    My understanding is that the limit is the maximum that the VM will be allowed to use. Therefore, even though the guest OS thinks it has 2048, what it was allocated. ESX will only allow that machine to use 1024, if that is what the limit is set at. That means less RAM thus perhaps creating a performance issue.

    Dallas



  • 3.  RE: Memory Resource Allocation - Serious Bug?

    Posted Jan 04, 2008 05:07 PM

    Yeah as the other poster pointed out, that "unlimited" is a bit of a misnomer. It allows you to adjust the memory on the fly within the VM limits

    So if you have a guest with 2Gb of RAM allocation, and you want to "throttle" that memory back to 1Gb, you can do it without changing the settings of the VM (which requires you to power off). So "unlimited" means there basically is no such throttling and the memory is allowed to use as much as the memory allocation assigned to the VM.

    The guest swapping depends on the OS, and what you have inside the VM. You shoud monitor the VM with the meters to see if you really need that much RAM. We typically only allocated 768 Meg of RAM at most, except for special circumstance. Some of my VM's are at 512 Meg, rarely do I assign more than 1G, and I watch the swap file to see if it grows. Server 2003 will and must use a swap file. So even the existence of a swap file doesn't necessarily mean it's in use, you have to watch the growth of the file, if it significantly increases, then you can bump the RAM, but the swap file is always used by Windows 2003 Server, XP and Vista, you can turn it off.. but you could have unexpected results doing so.



  • 4.  RE: Memory Resource Allocation - Serious Bug?

    Posted Apr 28, 2008 07:38 PM

    I am experiencing a very similar problem where various vms have cpu resources set to 23100 whereas the others are just unlimited. I agree and do think this is a problem for I had never set any limit or reservation on these vms. At first I thought it was due to how the vms were converted from physical because a portion were done manually and the others using VMware Converter, but I found that there is no rhyme or reason here.

    Hopefully someone who's experienced this before will know the cause.



  • 5.  RE: Memory Resource Allocation - Serious Bug?

    Posted Apr 01, 2009 11:05 PM

    Anyone ever find out what was causing this?

    I have about of 50 hosts that we see this on all the time. We'll spin out a VM form a template, check the VM to ensure that limits are not set for CPU and Mem and let DRS handle the rest after that.

    After a day or sometimes weeks to months, we'll get calls saying the VM is very slow. Check the perf graphs, sure enough ballooning about 1/2 the memory. Check the CPU or Memory limits and sure enough, configued to some value vs unlimited like we origianlly configured the VM. Usually, memory is the one that causes problems. It usually seems to get scaled back to about 1/2 of what you have provisioned for the VM. CPU will get limited by a value, but it is usually the sum of all the CPUs in MHz of the host.

    The host hardware is a mix of BL685c G1 w/64GB or 128GB or RAM. All running verion 3.5.0 130756

    Thanks,

    KC



  • 6.  RE: Memory Resource Allocation - Serious Bug?

    Posted Sep 19, 2009 01:25 AM

    Still seeing this issue. Anyone come up with a resolution?



  • 7.  RE: Memory Resource Allocation - Serious Bug?

    Posted Jun 07, 2010 06:36 PM

    I am having this very problem now. Running vSphere 4 Update 1 on my templates. Any news on why this is still happening?

    If you find this information useful, please award points for

    "correct"

    or "helpful".

    Wes Hinshaw

    www.myvmland.com



  • 8.  RE: Memory Resource Allocation - Serious Bug?

    Posted Jun 07, 2010 06:43 PM

    typically you will see these limits set if you have your templates configured as such. Convert the template(s) you use to deploy your guest with to a VM, and check to see there are any limits set.



  • 9.  RE: Memory Resource Allocation - Serious Bug?

    Posted Jun 07, 2010 06:52 PM

    I built all my server 2008 templates and I never set a a memory allocation. Also, when I deploy servers from templates it may or may not have the allocation set. I converted all my templates and they "all" had a reservation set. So if I know I never set it, then when did this happen? It looks like some of my 2003 templates are in the same boat. Still looking into it.

    If you find this information useful, please award points for

    "correct"

    or "helpful".

    Wes Hinshaw

    www.myvmland.com



  • 10.  RE: Memory Resource Allocation - Serious Bug?

    Posted Jun 08, 2010 01:28 AM

    Ok, well it looks like I have figured it out on my own. From what I have been reading this can happen when you upgrade hardware (for instance add memory). We added memory to our hosts and doubled it. This and upgrade to a new version of ESX can likely cause the issue. If you have a hunch you are having memory balooning issues, you may want to consort vFoglight or any other troubleshooting software that has predefined triggers.

    If you find this information useful, please award points for

    "correct"

    or "helpful".

    Wes Hinshaw

    www.myvmland.com



  • 11.  RE: Memory Resource Allocation - Serious Bug?

    Posted Nov 15, 2010 07:59 PM

    This and upgrade to a new version of ESX can likely cause the issue. If you have a hunch you are having memory balooning issues, you may want to consort vFoglight or any other troubleshooting software that has predefined triggers.

    It's NOT a bug, you didn't read the part about.. it's not unlimited from the HOST its unlimited from the range you defined in the VM configuration.

    2GB is the MOST that VM can be allocated (setting the memory limit). If you see "unlimited" that means the VM can use ALL of it's configured memory. This is because if you change the memory config later to something like 4G you don't need to change it in 2 places, since it's already "unlimited".

    It just means the VM is "unlimited" with respect to the configured RAM, not ALL the memory the host can provide. So there is not problem, there is no bug, and there is NO issue.



  • 12.  RE: Memory Resource Allocation - Serious Bug?

    Posted Nov 15, 2010 08:18 PM

    Well, except for the part where something is causing this value to be changed on lots of VMs...



  • 13.  RE: Memory Resource Allocation - Serious Bug?

    Posted Nov 15, 2010 08:27 PM

    Well, except for the part where something is causing this value to be changed on lots of VMs...

    I have noticed myself, it's triggered by a clone or sVmotion, that is not a mystery, the original question wasn't about WHAT was causing it to change, but WHY the value is unlimited.. and NOT set the value in the memory configuration.

    THAT part is NOT a bug, it's by design. Could be a pre-built limit so that VM's don't start grabbing memory just because someone changed the memory amount, that could pose a lot of issues, so by setting a limit it keeps this from happening.

    Besides that slider CAN be adjusted while the VM is running... so yes maybe it's an annoyance, but it doesn't cause downtime.



  • 14.  RE: Memory Resource Allocation - Serious Bug?

    Posted Nov 15, 2010 08:36 PM

    Hmmm, I'll have to check it after my next svMotion. I've been doing a lot of them recently as we work on getting off an old SAN. However, I'd still consider it a mystery. Why would I want my memory limit to change after a clone or an svMotion? Or really ANY time where I wasn't the one making the change?

    But, the original poster sure seems to be talking about WHAT causes the values to seemingly randomly change to me (even re-read it). It later morphed into a discussion as to WHY and HOW memory limits work (the part you correctly replied to).



  • 15.  RE: Memory Resource Allocation - Serious Bug?

    Posted Nov 16, 2010 10:16 PM

    I've been having this problem since before my original reply to this post back in 4/09

    We've been bit by the template being configured with a limit in the past..a limit that we didn't intentionally set. Fixed that.

    Once again, a random sampling, very large sampling at this point, have some value set as something other than unlimited for memory limit.

    I read somewhere that removing a host and adding it back in will cause this value to change.

    This could be true but I wasn't able to reproduce it this afternoon but much like allencrawford states, logging directly into the host and looking at the resource allocation for memory shows a value other than unlimited on all the VMs.

    Looking at some esx3.5 hosts, i see the VMs are set as unlimited for memory limit

    Looking at esx4 and 4.1, they are set to what the RAM is/was configured at time of provisioning.

    We've done rolling upgrades through our clusters so a lot of the VMs were once on esx 3.5 and are now on 4.1. But in some case, the VMs were provisioned just a few days ago, with no limit set, but now have a limit set.

    I too run a powershell script to change the values back to unlimited because too many times, the VM has been changed to a value that is less than the allocated amount of RAM and it starts to balloon as it should. This causes lots of performance issues.

    What else could be causing this value to be set?



  • 16.  RE: Memory Resource Allocation - Serious Bug?

    Posted Dec 16, 2010 04:12 PM

    I too have seen this issue. We had a VM with poor performance and increased RAM, twice. However today we discovered the limit has not changed since the first time we increased RAM. This VM came over from 2.5 to 4.1 and was vmotioned many times.



  • 17.  RE: Memory Resource Allocation - Serious Bug?

    Posted Feb 25, 2011 04:37 PM

    I now have my second case open with VMWare support over this issue. (They didn't believe me the first time..) The case has been sent to engineering, along with about 1GB of gzipped trivia logs.  I will be happy to post whatever VMWare Support comes up with.



  • 18.  RE: Memory Resource Allocation - Serious Bug?

    Broadcom Employee
    Posted Nov 15, 2010 09:06 AM

    So I'm looking over Memory Resource Allocation, and I notice most VMs have the Limit set to the memory assigned to the VM (i.e. 1024 MB or 2048

    MB). A few apparently random VMs have the limit set to Unlimited though. Anyone know what causes this? And if it would cause any problems?

    Limit will not cause any problmes, this is the feature in the ESX resource management to limit the VMs not to cross the assigned resource. Unlimited also will not cause any problems, guest does not know about the 'unlimited' and it is really limited to the memsize for that VM. Unlimited helps in adding the memory to the VM in feautre like hot add, etc.

    Second thing I notice, and this could be a real problem... There are a couple of VMs that I built with 1024 MB of memory, and then later I increased

    their memory to 2048 MB. Well, the Resource Allocation number didn't change! So the limit is still set 1024 MB...

    Yes, if you are limiting memory(Ex: 1024 MB in your case) , you need to increase the limit also along with memsize. So it is better to use unlimited. Unlimited is just terminology, but guest will never consume resource more than is is assigned.

    What happens when a VM that has been assigned 2048 MB (and that's what the VM's OS thinks it has), has a Resource Allocation limit of 1024 MB?

    I'm thinking some serious swapping is going on (which could explain some performance issues I've been having). Anyone aware of this bug? I'm using

    VC 2.0.2 50618

    Yes, and this not the bug. As you are limiting 1024MB , more than 1024 MB RAM memory will not be given to that VM. So rest will be reclaimed by memory swap. This is expected. If you dont want to make swapping of that VM you need to remove limits.

    Thanks.



  • 19.  RE: Memory Resource Allocation - Serious Bug?

    Posted Nov 15, 2010 07:46 PM

    Well, according to this KB article (http://kb.vmware.com/kb/1003470) which I've experienced on more than one occasion, you certainly can have issues by setting a memory limit less than what you've got allocated to the VM.

    But, whatever causes this buggy behavior seems to still exist. It isn't being propagated via templates in our environment (already checked that) but I'm still not clear on what causes it. I've got a PowerCLI script that I run occasionally to reset all the VMs back to default "Normal" share levels and "unlimited" on the memory limits. Yet, after moving to a 4.1 vCenter Server, just about every single VM is set to have a memory limit equal to its assigned memory. Obviously that isn't going to hurt anything, until you go to upgrade the memory of course because you have to remember to change the memory limit too. But who's going to do that when they think everything is already set to "unlimited"?

    Odd issue #2 that I see is that even after resetting all the limits (again) if I check the "Resource Allocation" on a cluster within vCenter, everything looks great (unlimited). But check the "Resource Allocation" tab when connected directly to a host in that same cluster, and all powered on VMs show a memory limit matching their assigned memory. Only the powered off VMs show "unlimited."

    Oh, and someone on this thread said "just change the memory limit when you upgrade the memory on a VM." That's easy to say, but the other problem I had was when I was upgrading the memory in our MS-SQL VM that hosts the vCenter database. Since that VM was down, so was vCenter, so I had to connect directly to an ESXi host. I was able to bump the memory up just fine, but the limit would not change. I went from 8GB to 12GB and the memory just stayed at 8GB. The only time I was able to change it was once vCenter was back up and I did it via the vCenter connection (i.e. not directly attached to the host). It accepted "Unlimited" when i did that. Something is clearly not right years later...

    Two notes:

    1) I have opened an SR for this (finally, as we've had the issue since the vCenter 2.5.x days at least)

    2) We did not upgrade to vCenter 4.1, we literally installed a fresh instance of vCenter 4.1 and migrated hosts to it via some PowerCLI scripts. And all of our hosts are still ESXi 4.0 U1 (except for a few ESX 3.5 U4 hosts that can't be upgraded yet).



  • 20.  RE: Memory Resource Allocation - Serious Bug?

    Posted Feb 24, 2011 10:48 PM

    AllenCrawford,

    Any chance you could post the PowerCLI scirpt?

    Gene



  • 21.  RE: Memory Resource Allocation - Serious Bug?

    Posted Apr 18, 2011 08:39 PM

    Sorry for the delay, just now getting caught up on some old emails and found the one for this thread.  The script below is meant to be run interactively after you've already connected to vCenter and it expects you to provide a cluster name to it.  I did that because our environment is pretty large and I didn't want to hit it all at once, though it really shouldn't be an issue (other than the amount of time it takes).  You can easily tweak it to just run on every VM.  The script not only sets the memory limit on all VMs to the default (unlimited) but also resets the shares and reservations back to default as well as the same three values for CPUs.  The script isn't as optimized as it could be for speed (this is one of my first PowerCLI scripts, so it is old) and could probably have some disk I/O resource/share items added to it.  Regardless, here it is:

    Param([string]$strClusterName)
    if (! $strClusterName) {
    Write-Output ("")
    Write-Output ("Hey, you need to give me a parameter!")
    Write-Output ("  Example: " + $MyInvocation.MyCommand.Name + " <ClusterName>")
    Write-Output ("    - where <ClusterName> is a string that matches the name of your VM cluster")
    Write-Output ("")
    }
    else {
    Write-Output ("")
    Write-Output ("Thinking...")
    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec;
    $spec.cpuAllocation = New-Object VMware.Vim.ResourceAllocationInfo;
    $spec.cpuAllocation.Shares = New-Object VMware.Vim.SharesInfo;
    $spec.memoryAllocation = New-Object VMware.Vim.ResourceAllocationInfo;
    $spec.memoryAllocation.Shares = New-Object VMware.Vim.SharesInfo;
    $spec.cpuAllocation.Shares.Level = "normal";
    $spec.cpuAllocation.Reservation = 0;
    $spec.cpuAllocation.Limit = -1;
    $spec.memoryAllocation.Shares.Level = "normal";
    $spec.memoryAllocation.Reservation = 0;
    $spec.memoryAllocation.Limit = -1;
    #Get-Cluster $strClusterName | Get-VM | % {Get-View $_.ID} | % {Get-View($_.ReconfigVM_Task($spec))}
    Get-VM | % {Get-View $_.ID} | % {Get-View($_.ReconfigVM_Task($spec))}
    }

    EDIT:

    1) I have no idea why the above script is double-spaced, but I'm too lazy to fix it.

    2) I just now noticed that this is my modified version that was set to run on ALL VMs in the vCenter you are connected to.  The second to last line (Get-VM ...) really should be commented out and the one above it should be uncommented to run as I originally intended it.



  • 22.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 15, 2012 03:19 PM

    Any update on this?

    I am seeing this at alot of clients and it appears to be upgrade related.   The Memory Limits get set to the size of the granted memory of the VM.  This isn't a problem until they grant the VM more memory and the limit stays at the old value.

    -MattG



  • 23.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 03:20 AM

    I too have seen this behaviour earlier in my environment running on vSphere 4.1. Some of the VMs had memory limit set to 4 GB causing severe performance issues. I have not been able to reproduce the issue therefore refrained from opening the support ticket. It wasn't a upgrade but a fresh 4.1 install, still no clue what had caused that behaviour..



  • 24.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 04:12 PM

    Just to make things much clearer.
    There are three cases here:
    1) When Memory limit is less than the Memory size --> Memory Limit is the Hard value that you assign to the VM which means that VM should  not grow beyond this limit.This is being set by VMkernerl.Since this hard limit is not known by the GOS and the application running on it, so they think  that complete Memory size(assigned) is at their discretion.Over the period when once the limit is hit VMkernel will state that you have reached the memory limit and no more memory can be assigned to you.
    Suppose your VM memory size is 2GB and you have set the memory limit to 1 GB in that case VM will only be able to use 1GB and the remaining 1GB will be swapped out
    The guest simply doesn't know that it is being rejected, it just knows it cannot get access to more memory.  As far as it is concerned, it still sees another 1GB sitting there unused.  This is naturally going to cause a performance hit to your applications, as they are being deprived of memory resources.
    The VMkernel simply will not allocate any more memory due to the hard limit.That's when Memory reclamation techniques will come in picture which will break the limit set by the VMkernel like Ballooning.

    2) When Memory limit option is checked as unlimited or Memory limit set is more than Memsize --> By default  Memory limit is unlimited for  VMs.Here   limit will be  equal to Memsize size of the VM.



  • 25.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 04:18 PM

    The problem is that the memory limit is being set by some bug with vSphere.   I am pretty sure it has to do with the upgrade process.

    This is a serious bug as it affecting alot of clients that I have seen and they did not change the memory limit settings on their own.

    VMware needs to address what is causing this,  because it can causing Guest OS memory paging even though memory is not an issue on the host.

    -MattG



  • 26.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 04:42 PM

    The only way VMware is going to address it is if you file an SR.  A forum post (while helpful) is not something VMware uses to build bug reports.



  • 27.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 04:53 PM

    Cases have been opened and subsequently closed when the issue cannot be easily reproduced. (11049431803)  I'm not so sure that the issue has gained the attention of the correct people in support, ie an escalation engineer.

    In my case, the problem completely went away after upgrading to 4.1U1.



  • 28.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 05:17 PM

    Thats wierd? Upgrading to 4.1 U1 removes the limits? It either would need to remove all memory hard limits (bad) or know which VMs had been mistakenly set to a hard limit and change only them?

    -MattG



  • 29.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 08:58 PM

    Upgrading did not remove the limits.  We wrote a script to check for and remove it.(set it to unlimited)  We just noticed that the problem stopped happening after the upgrade.



  • 30.  RE: Memory Resource Allocation - Serious Bug?

    Posted Oct 22, 2012 08:56 PM

    All,

    I've seen this issue from ESX on 3.5 all the way up through and including the version that I'm on today (5.0.0).  I will open up a case to see where VMware is with this.  I assume this is a bug that's been an issue from long ago and that VMware admins just have to deal with it.  That said, it did cause an issue recently whereby I had a VM ballooning after RAM increases....



  • 31.  RE: Memory Resource Allocation - Serious Bug?

    Posted Oct 23, 2012 04:43 PM

    For additional emphasis, I have worked for a vendor for four years where every month I look at dozens of different customer sites. After looking at 100s of virtual environments, I count on seeing these "stealth" memory limits plaguing at least a few virtual environments every week and it's a part of my regular demos. Admins are regularly stunned and confused when I show this to them and is often a source of performance issues.

    I'm seriously suprised that there hasn't been enough support tickets to warrant a witch hunt at any point in the past four years.



  • 32.  RE: Memory Resource Allocation - Serious Bug?

    Posted May 29, 2013 03:13 PM

    I saw hard memory limits yesterday at a client who is running 5.1.

    If VMware won't fix this bug,  they should at least change vCenter to have it put a warning icon next to VMs with limits (like SSH enabled hosts) so that you know something is wrong.

    -MattG



  • 33.  RE: Memory Resource Allocation - Serious Bug?

    Posted Mar 16, 2012 04:57 PM

    Unfortunately, this is one of those bugs that is hard to reproduce (you don't know when it happened). Calling support to say "somewhere in the past year a VMware related process changed my memory limits from unlimited to hard set" is probably not going to get me very far.

    If enough people chime in on this thread maybe VMware can look into why this is happening.

    -MattG



  • 34.  RE: Memory Resource Allocation - Serious Bug?

    Posted Aug 12, 2013 03:35 PM

    Chiming in- I've seen this in two environments and found this to be the source of performance issues.  VM memory size grown, limit set to the old size, swapping (etc) kicks in.

    I first noticed this years ago, in the probably 3.x versions.  At the time, from the research I did, I came to the conclusion that these limits were set by DRS during a DRS-initiated VMotion operation.  Though that could be incorrect.