vpxd consuming all memory -- vCenter slooooooow and cannot connect

[PeterBlatherwic]

As reported in http://communities.vmware.com/message/2020222 we are still experiencing the issue with vpxd consuming huge amounts of RAM and causing severe vCenter slowness.  This has gotten much worse recently.  We are hoping to renew the conversation on this, as it is becoming intollerable, and our efforts have so far failed. 

Configuratoin: VMware vCenter Server 5.0, update 1, version 5.0.0 build 623373.  Running on Windows Server 2008 R2 Standard SP1, in a virtual machine with 4 vCPUs and 8 GB of memory, SQL database on same machine.

When the issue opccurs, the main vCenter executable, vpxd.exe uses up all available memory, causing severe vCenter slowness, to the point where vSphere Client cannot connect, tasks fail due to timeouts, remote desktop to the machine are extremely slow and almost completely unusable.  After stop/start of vpxd.exe service, or restart the Windows VM, the issue initially clears, but then returns later. We are now seeing this more than once per day, sometime 3 - 4 times in a single day.

In more detail:

- Windows task manager reports 99% of  memory is consumed.  Typically, this is around 6.5 GB of memory consumed  by vpxd.exe.

- Before the issue occurs, vpxd is typically consuming around 330 MB memory (reasonable)

- Other large consumers are sqlservr.exe around 1.0 GB, java.exe around 820 MB + 380 MB (there are two processes), tomecat6.exe 690 MB.  We think these are probably normal, and they do not grow out of control like vpxd does.

- During the issue, CPU also becomes very high, near 100%, as seen through vSphere Client performance

- VM is running current VMware tools (8.6.5, build 652272)

Things we have tried:

- Upgraded to vCenter Server 5.0u1

- Rebalanced vCPUs (as suggested in other thread).  Initially was 4 CPUs on 1 socket, changed to 2 CPUs on 1 socket, now 4 CPUs split across 2 sockets.

- Moved to VM version 8

- Set service to autostart, delayed

- General cleanup -- removed a bunch of servies we are not using (Orchestrator, Update Manger...)

Any further suggestions, known issues, FIXES, whatever would be helpful! 

-- PeterB

[Britz]

same issue!!!

[jizaymes]

Also have the same issues. In our deployment, the vCenter server is on a standalone box so vmware tools things dont apply. Its using 6GB of RAM where it shouldn't really be.

[Britz]

We fixed it by installing java from java.com<http://java.com>

We installed it to ship logs to VMware though their website and while doing so it fixed it. Let me know of this was a fluke or an actual fix.

[Britz]

nevermind rebooted after a few weeks problem is back!!!

[Britz]

by chance does anyone have the vsphere web client installed???

disabled the service and seemed to stabalize

[illvilja]

Hi!

vCenter 5 Update1A was released today which resolves an issue with tomcat6.exe allocating alot of memory.

vCenter Server Web services might consume all the memory assigned to it
vCenter Server Webservices (tomcat6.exe) might consume all the memory assigned it. Increasing the Tomcat maximum memory pool does not resolve the issue.
This affects any vCenter web services related functionality.
This issue is resolved in this release.

http://www.vmware.com/support/vsphere5/doc/vsp_vc50_u1a_rel_notes.html

I'm not sure if this has anything to do with vpxd.exe, but it's worth a shot.

--- Martin

[PeterBlatherwic]

Based on responses, it appears multiple others are also having the same issue, or at least same symptoms. 

We worked through our issues with VMware support, and they also recommended applying 5.0u1a when available.  We will apply it and see.  Read on.

Things we did:

- Increased memory reservation to 16 GB / Unlimited  (YES 16 GB .. hideously large !!)

- Set CPU reservation to 8 Ghz / Unlimited (high, but not as insane as the memory one)

- Disabled web services (tomcat).  This was reenabled later on in the process, after the memory / CPU reservartions changed.

- Increased tomcat memory pool (had no effect)

- Rebalanced vCPUs, as above (no effect on the runaway memory issue)

- Cleaned up dB -- SQL was *very* fragmented (marginal effect on performance, no effect on run-away memory consumption)

- Cleaned up some services we are not actually using daily (Orchestrator, Update Manager)

Lookin at the fix in 5.0u1a release notes, I am personally not very convinced this is really the same thing.  For us, it was vpxd that was consuming all memory, as seen through Task Manager.  We still see memory spikes occasionally, up to about 70+% of total RAM with the new reservation.  This would have exceeded our previous reservation -- seems likely (to me) our vCenter would have locked up at that point. 

But we will apply the update and see.  Then, reduce our memory reservation again, and see. 

-- PeterB

[FGShepherdP10]

We've been having a similar issue recently (painfully long (30+ minutes) startups/shutdowns of the vCenter server), high "cached" memory counts (In one case, vpxd.exe was using <1GB of RAM, but the server was using 20GB, and when we killed the vpxd process, the RAM dropped to almost 0.)...I was thinking that it was some sort of memory leak (and it may still be), since that had been a problem in the past (not this again, VMware).

In an attempt to troubleshoot the issue, I set all of the vCenter-related services to "Automatic-Delayed Start" and rebooted the server.  (In true break-fix style, I also doubled the RAM (16-->32), added an 8GB reservation, and reserved 6000MHz of CPU, according to the post, above...all in one reboot...So, yes, this will make it difficult to determine which fix actually helped.  You may want to try each on its own to narrow down the best change.)

At any rate, the server is performing extremely well at the moment...better than it has in months.  We're using a LOT of RAM, but our environment is decently large, so that's not surprising.  If things change, I'll re-post.

[FGShepherdP10]

Update: Nope.  That wasn't the fix.  Over the last 24 hours, at least twice, vpxd used up all available RAM (32GB, see above post), and then Windows paged all of the RAM to disk.  RAM usage slowly falls (vCenter is unavailable during this time, naturally), while the Cached Physical Memory on Windows grows at the same rate.  When it's done, vCenter restarts automatically.

Thinking of bringing the RAM back down in the meantime, while troubleshooting, since it will at least shorten the time for this process (down from 30+ minutes to...?).

Side note: I've also disabled vCenter Operations, a small-footprint vKernel deployment, and EMC's ProSphere that were all connected to our environment (lots of old/idle sessions), but that apparently wasn't utilimately the root of the problem.

[Britz]

We seemed to get it to stabilize. Reboot the os machine with vcenter service disabled. Wait about a half an hour then enable and start only that service and see it is stable. If it is you can start inventory and web services service.

Worth a try till they get update 2 released. From looking at different forums it looks to be an issues when your hosting in a virtual machine and your disk latency spikes. Which means VMware most likely wont come out and say exactly what the problem is as it is against their business model. We have had a case open for over 2 months.