vpxd consuming all memory -- vCenter slooooooow and cannot connect

[PeterBlatherwic]

As reported in http://communities.vmware.com/message/2020222 we are still experiencing the issue with vpxd consuming huge amounts of RAM and causing severe vCenter slowness.  This has gotten much worse recently.  We are hoping to renew the conversation on this, as it is becoming intollerable, and our efforts have so far failed. 

Configuratoin: VMware vCenter Server 5.0, update 1, version 5.0.0 build 623373.  Running on Windows Server 2008 R2 Standard SP1, in a virtual machine with 4 vCPUs and 8 GB of memory, SQL database on same machine.

When the issue opccurs, the main vCenter executable, vpxd.exe uses up all available memory, causing severe vCenter slowness, to the point where vSphere Client cannot connect, tasks fail due to timeouts, remote desktop to the machine are extremely slow and almost completely unusable.  After stop/start of vpxd.exe service, or restart the Windows VM, the issue initially clears, but then returns later. We are now seeing this more than once per day, sometime 3 - 4 times in a single day.

In more detail:

- Windows task manager reports 99% of  memory is consumed.  Typically, this is around 6.5 GB of memory consumed  by vpxd.exe.

- Before the issue occurs, vpxd is typically consuming around 330 MB memory (reasonable)

- Other large consumers are sqlservr.exe around 1.0 GB, java.exe around 820 MB + 380 MB (there are two processes), tomecat6.exe 690 MB.  We think these are probably normal, and they do not grow out of control like vpxd does.

- During the issue, CPU also becomes very high, near 100%, as seen through vSphere Client performance

- VM is running current VMware tools (8.6.5, build 652272)

Things we have tried:

- Upgraded to vCenter Server 5.0u1

- Rebalanced vCPUs (as suggested in other thread).  Initially was 4 CPUs on 1 socket, changed to 2 CPUs on 1 socket, now 4 CPUs split across 2 sockets.

- Moved to VM version 8

- Set service to autostart, delayed

- General cleanup -- removed a bunch of servies we are not using (Orchestrator, Update Manger...)

Any further suggestions, known issues, FIXES, whatever would be helpful! 

-- PeterB

[Britz]

same issue!!!

[jizaymes]

Also have the same issues. In our deployment, the vCenter server is on a standalone box so vmware tools things dont apply. Its using 6GB of RAM where it shouldn't really be.

[Britz]

We fixed it by installing java from java.com<http://java.com>

We installed it to ship logs to VMware though their website and while doing so it fixed it. Let me know of this was a fluke or an actual fix.

[Britz]

nevermind rebooted after a few weeks problem is back!!!

[Britz]

by chance does anyone have the vsphere web client installed???

disabled the service and seemed to stabalize

[illvilja]

Hi!

vCenter 5 Update1A was released today which resolves an issue with tomcat6.exe allocating alot of memory.

vCenter Server Web services might consume all the memory assigned to it
vCenter Server Webservices (tomcat6.exe) might consume all the memory assigned it. Increasing the Tomcat maximum memory pool does not resolve the issue.
This affects any vCenter web services related functionality.
This issue is resolved in this release.

http://www.vmware.com/support/vsphere5/doc/vsp_vc50_u1a_rel_notes.html

I'm not sure if this has anything to do with vpxd.exe, but it's worth a shot.

--- Martin

[PeterBlatherwic]

Based on responses, it appears multiple others are also having the same issue, or at least same symptoms. 

We worked through our issues with VMware support, and they also recommended applying 5.0u1a when available.  We will apply it and see.  Read on.

Things we did:

- Increased memory reservation to 16 GB / Unlimited  (YES 16 GB .. hideously large !!)

- Set CPU reservation to 8 Ghz / Unlimited (high, but not as insane as the memory one)

- Disabled web services (tomcat).  This was reenabled later on in the process, after the memory / CPU reservartions changed.

- Increased tomcat memory pool (had no effect)

- Rebalanced vCPUs, as above (no effect on the runaway memory issue)

- Cleaned up dB -- SQL was *very* fragmented (marginal effect on performance, no effect on run-away memory consumption)

- Cleaned up some services we are not actually using daily (Orchestrator, Update Manager)

Lookin at the fix in 5.0u1a release notes, I am personally not very convinced this is really the same thing.  For us, it was vpxd that was consuming all memory, as seen through Task Manager.  We still see memory spikes occasionally, up to about 70+% of total RAM with the new reservation.  This would have exceeded our previous reservation -- seems likely (to me) our vCenter would have locked up at that point. 

But we will apply the update and see.  Then, reduce our memory reservation again, and see. 

-- PeterB

[FGShepherdP10]

We've been having a similar issue recently (painfully long (30+ minutes) startups/shutdowns of the vCenter server), high "cached" memory counts (In one case, vpxd.exe was using <1GB of RAM, but the server was using 20GB, and when we killed the vpxd process, the RAM dropped to almost 0.)...I was thinking that it was some sort of memory leak (and it may still be), since that had been a problem in the past (not this again, VMware).

In an attempt to troubleshoot the issue, I set all of the vCenter-related services to "Automatic-Delayed Start" and rebooted the server.  (In true break-fix style, I also doubled the RAM (16-->32), added an 8GB reservation, and reserved 6000MHz of CPU, according to the post, above...all in one reboot...So, yes, this will make it difficult to determine which fix actually helped.  You may want to try each on its own to narrow down the best change.)

At any rate, the server is performing extremely well at the moment...better than it has in months.  We're using a LOT of RAM, but our environment is decently large, so that's not surprising.  If things change, I'll re-post.

[FGShepherdP10]

Update: Nope.  That wasn't the fix.  Over the last 24 hours, at least twice, vpxd used up all available RAM (32GB, see above post), and then Windows paged all of the RAM to disk.  RAM usage slowly falls (vCenter is unavailable during this time, naturally), while the Cached Physical Memory on Windows grows at the same rate.  When it's done, vCenter restarts automatically.

Thinking of bringing the RAM back down in the meantime, while troubleshooting, since it will at least shorten the time for this process (down from 30+ minutes to...?).

Side note: I've also disabled vCenter Operations, a small-footprint vKernel deployment, and EMC's ProSphere that were all connected to our environment (lots of old/idle sessions), but that apparently wasn't utilimately the root of the problem.

[Britz]

We seemed to get it to stabilize. Reboot the os machine with vcenter service disabled. Wait about a half an hour then enable and start only that service and see it is stable. If it is you can start inventory and web services service.

Worth a try till they get update 2 released. From looking at different forums it looks to be an issues when your hosting in a virtual machine and your disk latency spikes. Which means VMware most likely wont come out and say exactly what the problem is as it is against their business model. We have had a case open for over 2 months.

[FGShepherdP10]

Update: Lowered RAM to 16 GB (to decrease flush time), lowered vCPU to 4, and rebooted.  For about 18 hours, we've been hovering steady around 11 GB of RAM.  Secondary services/appliances are still powered off.

[PeterBlatherwic]

Hi FGShepherdP10,

Ouch, sounds pretty painful, as our issues have been also.  16 GB memory reservation .... Yeeesh!

Are you running on vCenter 5.0 update 1a, just plain update 1, or something else? 

As above, we intend to move to 5.0u1a, but have not yet been able to do so for operational reasons. We currently have a mostly stable vCenter, only one serious memory event since changes described above (... of course, just before a scheduoled executive demo ... ;^)  I am personally a bit dubious that 5.0u1a will fix the issue, but would like to hear from anyone who has applied it, and can verify (or not) that it does address the issue. 

-- PeterB

[PeterBlatherwic]

Update:

We have now applied vCenter 5.0 update 1a, the recommended "fix" from VMware.  We are still having issues.

Since applying the update last week, we have had multiple incidents of the vCenter going so sloooooooow we cannot connect through vSphere Client, operations failing on timeouts, etc -- had to restart the VM.  Although the specific symptoms seem to have changed a bit -- seeing high CPU % and relatively lower memory % (higher that should be, but not complete runaway).  Alas...

Has anyone else out there applied 5.0u1a?  Did it fix your issue, or not?  Any work-arounds?

Another thought, we forgot to mention before.  Our vCenter VM was created by transforming a formerly physical machine to a VM.  The physical machine did see this issue, but very very rare.  Now on VM we see it a lot.  But, the high incidence rate did not correspond with importing the physical to the VM -- it started occuring several months later.  Are others also running vCenter that moved from physical to virtual, or having same issue on a VM vCenter that was never physical?

We have re-opened the support ticket with VMware on this.

-- PeterB

[FGShepherdP10]

Update 2: For over a week now, vCenter has been stable.  We're running 7-12 GB of RAM on the vpxd service itself, and roughly 4 GB of RAM, each, on Tomcat6 and Java.  The 16 GB allocation of RAM to the entire server is mostly used, but response times inside of the client are good and the services have all stayed up with only intermittent, superficial errors in Windows Events.

At this point, I'm willing to point to the auxiliary services that were plugged into vCenter, though I can't say with certainty which one was the real culprit.  Here's what we've turned off:

• VKernel
• vCenter Operations Manager
• vCenter Chargeback Manager (watch this one-- it will CRUSH a datastore)
• EMC ProSphere

We are not planning on turning any of these services back on yet, but will post results when we do.

BTW, we're still on 5.0.0 (Build 455964), so I'm not looking at this from an upgrade standpoint.

Hopefully, this helps someone...Good luck to those of you still having this problem and thanks to those who have contributed to finding a fix.

[SangNg]

I want to bring this topic back to discussion. I'm having same issue here. VPXD and Java processes consistently peak out CPU and memory. I'm using vCenter 5.0 and sounds like  5.0 u1a will not fix the issue at all. Is anyone able to determine what really causing this? Many thanks

[jdoz_bw]

I have just installed VC 5.0 Update1b (released last week).  I am seeing Java.exe processes use a ton of memory on my VC server.  It's not pegging out, but it is certainly using a large amount of resources.  Is there a fix for this yet, or a root cause, or is it just by design?

[illvilja]

Hej!

Jag är på tjänsteresa i USA och har begränsad möjlighet att svara i telefon.

Vänligen kontakta Anders Sevelin 021-805808 så slussar han brådskande ärenden vidare.

Hi!

I'm at VMworld in San Francisco, I have limited access to answering the phone.

Please contact Anders Sevelin at [phone] for urgent matters.

Mvh//Martin

[Britz]

Check to see if any of your VM's CD/DVD rom are connected to a device that doesn't exist.  I have noticed that when Live migrating VM's from one host to another, and it was using the previous Hosts physical CD/DVD rom it can cause a lot of problems.  I had several machines in this state, and since changing it back to Client Device, we have not had this issue again.

[PeterBlatherwic]

Hi Britz,

> ... see if any of your VM's CD/DVD rom are connected to a device that doesn't exist ...

Interesting suggestion, and something to watch for.  However, very impractical.  We have hundreds of VMs deployed, and we are not even that big a datacenter (we are an R&D team).

Why would CD/DVD on some VM(s) cause vCenter/vpxd to eat all memory and choke.?? 

-- PeterB

[SangNg]

I updated my vCenter today to 5.0 U1a. After the update, vpxd process creeped up and eventually consumed all memory (8GB). I rebooted it but it won't help at all (gradually consumed all memory until vCenter became really slow). I then stopped all vmware related services (update manager, orchestrator, etc) and restarted vpxd (vCenter) service manually. It came back up gradually consuming memory again (up to 3GB) but suddenly vpxd memory dropped and stayed around 600MB. Then I proceeded to start other vmware services. I also configured tomcat to start with 265MB and max at 512MB.

I know this doesn't really show what is the real culprit behind vpxd consuming all memory, but right now my vCenter is consuming only 6GB (I gave it 8GB, with vpxd process taken up only 600MB). I only have update manager and orchestrator running beside vCenter. Hope it helps others who are having same problem because I scratched my head around this problem for last 2 weeks. Good luck!!!!

[tommy269]

I have the exact smae problem too.  We were on a physcal VC 5.0 with 6 gig of ram and the DB on another SQL server too.  I p2v our VC over and gave it 16 gigs of ram and a week later it too up all the ram and cause major slowness.  Support has told me reinstall the VMware tools and if it happen again enable Verbos logging for further support??? This is what I found out and done so far that kind of help:

1. decrease VC logging to warning and errors only

2. disabled auto deployed that another admin was testing

3.create a new large page file drive for just virtual memory usage only

So far it is running better at 10 gig of ram for a week now but still would like to see this number drop lower if possible.

[tommy269]

http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-67C4D2A0-10F7-4158-A249-D1B7D7B3BC99.html

Two services that I found is the actual cause of the high Ram but it is noted on the linked above from vmware for running inventory services and storage driven serives.  FYI

[GalNeb]

Here is what cured our problem with VPXD growing until it crashed.  We found that somewhere on our network there was a vSphere Client that had been forgotten, perhaps the user disconnected a remote session and left the client window open and the session going.  This client was trying to reconnect after vCenter had rebooted/restarted etc.  Trying and Trying and Trying.  Once we killed this client, then the memory growth issue went away.

[CELLULESYSTEME]

Thank you, that fixed the problem !!!!!

[zomx]

We are running vCenter Server 4.1 Update 3 | 30 Aug 2012 | Build 799345. Our vpxd continually would grow and eventually consume all 16GB of RAM in the VM. SQL is colocated on the same VM.  Neither vCenter vpxd service restarts or reboots helped. We saw this post and decided to both disconnect all existing client sessions and restart our management workstations which solved the problem.

Thanks so much for posting the resolution.

Here is what cured our problem with VPXD growing until it crashed.  We  found that somewhere on our network there was a vSphere Client that had  been forgotten, perhaps the user disconnected a remote session and left  the client window open and the session going.  This client was trying to  reconnect after vCenter had rebooted/restarted etc.  Trying and Trying  and Trying.  Once we killed this client, then the memory growth issue  went away.

[illvilja]

Hej!

Jag är ledig och har begränsad möjlighet att svara i telefon och mail.

Vänligen kontakta Anders Sevelin 021-805808 så slussar han brådskande ärenden vidare.

Åter i tjänst 19e nov

Hi!

I'm on vacation and have limited access to answering the phone.

Please contact Anders Sevelin at [phone] for urgent matters.

Back in business the 19th of november

Regards//Martin

[bugblatterbeast]

I had exactly the same issue with vCenter 4.1 build 799345.  I had a VM inside a cluster that someone had left a session connected to the console.  It seems only terminating the session works.  Restarting vCenter had no effect at all.  Symptoms were ridiculous memory usage by vpxd and then subsequent crash of the vpxd service.  I also had lots of repeat events in the Cluster itself reporting this session connection attempts.

Thanks to all for pointing me in the right direction!

Danny

[GalNeb]

I have now seen the problem in vCenter 5.1 as well.  So they have not fixed it.  Try an "arp -a" from the vCenter server to see what IP addresses are talking to the vCenter server.  That will tell you where to look for bogus Client connections that are causing the memory leak.

[ranjitcool]

The real issue may be http://kb.vmware.com/kb/2007600

Once you killed the client, lesser cache was being consumed so you did not see the issue. VMware admits it did not increase the cache size enough.

RJ

[GalNeb]

This article is not relevant to the situations that I have seen.  First this article makes no mention of Memory growth.  It also is only an issue with very large implementations that have been upgraded from previous versions.  I have seen this problem with fresh installs on systems with as few as 2 hosts and 10 VMs running.

[ranjitcool]

Did you check if you have any evictions? Do not judge the relevance without checking for the symptoms in the article descirbed.

RJ

[3CV]

We had a similar issue, however I upped it to 10GB from 8GB, but the biggest factor by far was putting the database on a separate vdisk.  I had installed it on a seperate D partition because it wasn't going to be a full production system at the time, but as often happens, that changed.   So migrated it to a new disk and the speed increase was very dramatic.  Also, as someone has said, make sure there are no CDs or ISOs mounted.  They cause havoc if you try doing anything admin-ish!  there are scripts floating around t'interweb that will scan your system for VMs with anything mounted.  It's bad enough with 35 VMs let alone hundreds!

[rockymiho1]

spot on. killing the clients solved the problem.

Thanks heaps

[ranjitcool]

Hello

Can you tell me in detail how you killed the clients?

Thanks

RJ

[rockymiho1]

Hi,

Login to vsphere client, Administration >> administration >> Sesssions.

You will see the active sessions.  right click and terminate session. However, doing this what I noticed client was connected again automatically. So end up logging out the client from the windows.

Hope terminating the session will work for you so you dont have to force log off the user from windows.

[ranjitcool]

Thanks a lot, so once terminated did vpxd drop? Did tomcat also drop in ram?

Also how do you find who is using a session as in we have a huge dc and many admins so I want to be careful

Does killing idle sessions also resolve this?

Thanks again

[rockymiho1]

never had problem with tomcat, however while fixing the vpxd issue i did stop the vmware web service but that really dint help.

when you look at the sessions in vsphere client, it will display the username and you can check with your system administrator. I would say why not kill everyone to fix the problem?

[rockymiho1]

also you just having issue with slowness or is your vcenter server consuming all the memory? If the slow problem then this is not the correct thread, this thread is mainly for people experiencing with the mem leak issue.

[ranjitcool]

Thanks, memory leak and vcenter service dieing. VPXD maxing out. We fixed the buffer cache size issue and set it to unlimited as per my earlier posts where i have the kb article.

Thanks

RJ

[rockymiho1]

I would just terminate all the sessions to fix the issue.

not sure about the clients session usage off vpxd

[ranjitcool]

Thanks, I killed a few idle sessions - did not see any drop in vpxd. Did you see a drop right away when you killed some idle sessions?

[rockymiho1]

I had only 3 users logged in including myself, when I killed the other 2 sessions I dint see any drop as for some reason the sessions were reconnected automatically.  So I end up force log off from the windows where they were connecting from. In our environment users can only connect from 1 windows box therefore It made me easy. Not sure about your environment.

[ranjitcool]

Thanks for the reply, so you went into AD and did a force logoff? Or logged into their windows box and did a log off and once done u saw a drop right away?

[rockymiho1]

i logged into windows box from where they were connecting and did force log off. yes as soon as I kicked out of windows, memory usage dropped to 7 to 8% and its still same since yesterday :smileyhappy:

[ranjitcool]

Is there any measure of how much each vcenter client session would use off of vpxd? As in 100mb per session may be?

Would anyone know?

RJ

[ranjitcool]

Try this to fix ths issue -

http://kb.vmware.com/kb/2007600

If you found the evictions then this will certainly and hopefully fix the issue.

Thanks

RJ

[sbrown218]

by default mssql server will use all available ram on a server for sql query caching.  if you look at sqlservr.exe in task manager/processes it will typically say its using 128mb ram, but this is misleading. goto sql management studio and limit the amount of ram that it is allowed to use and you will see the ram use on your virtual center server immediately drop. from sql management studio right click on your sql server and goto properties - what you are looking for is the memory tab - on an 8gb server i would only allow it to use 2048mb max (not the 2tb that is the default). my virtual center is on a hard blade with 32gb ram so i configured sql server to use a maximum of 12gb.

--scott

[ThunderboltsRck]

I have pin pointed the cause of this memory issue in my 5.1 environment and stopped the issue by disabling the Single Sign on component. The steps that we performed are NOT supported by VMwrae so use at your own risk:

1. RDP to vcenter

2. Locate vpxd.cfg    C:\ProgramData\VMware\VMware VirtualCenter\

3. Edit this file with notpad++ (google it if you don't have it)

4. Modify line 54 or find code:

<sso>

          <admin>

                         <url>https://vcenter FQDN/sdk/uri>

          </admin>

          <enabled>true</enabled>

          <groupcheck>

          <lookupService>

                         <serviceId>{Large String here}:5</serviceId>

          <lookupService>

Change the Value "true" to "false"  (without quotation marks) and then re-boot your vcenter. In doing this our vcenter has gone from a woolly mammoth using 40GB of ram to using 7.5 - 8GB and it now stable and responding fast and not producing false alerts.

I hope this may be of some small help to someone but as I said at the start use at your own risk as this is not supported configuration by VMware.

         

[tonyhtsun]

We are using 5.0 and found out that the problem was caused by the performance monitor in vsphere client.

If there is a vsphere client keeps monitoring the graph from vsphere client, the vCenter memory will be eaten up by the vpxd,exe.

Once I shutdown the vsphere client machine and restart vcenter server everything becomes normal again.

So somehow the performance monitoring on vcenter has memory leak I guess.

Any idea?

[seycha]

We had the same issue for weeks and finally resolved it with help from VMware. We use a VMware Data Recovery appliance (v2.0.0) and have the corresponding plugin for the vSphere client installed.

It turns out this plugin was creating an awful lot of connections to the vCenter Server when it was restarted, causing the memory to go 100%. After uninstalling the plugin, upgrading the VDR appliance to the latest version (2.0.2), and re installing the plugin from its latest release, the problem went away. VMware also made us upgrade the vCenter SQL Express database to a full edition, but I'm not sure that made a difference.

So if you have the memory leak issue and are using VDR, you might want to do the same. Good luck !

[tonyhtsun]

Hi Seycha,

May I know which version of VDR plugin you are using?

Thanks

[seycha]

The latest version (2.0.2) is fine.

[DavidSilva77]

Hi my friend, check this KB , is possible that Tomcat es causing your trouble

VMware KB: VMware vCenter Management Webservices features do not function properly

[SteveEdi]

I had the same problem and its cause was that the database was full, so he couldnt write any data to the database and the cpu runs 100%

I solved the problem with this:

First check if the vcenter database is full.

If yes, try this:

VMware KB: Purging old data from the database used by VMware vCenter Server 4.x and 5.x

After that, restart the vcenter service and it should stay under 50%

Greetings EDI