Memory and WPs

[Anon Anon]

Good Afternoon,

We've been noticing lately that our memory consumption on our AE servers are only growing and had an instance where it filled the pagefile consuming 98% of our disk space on the server (we're running on Windows).  Luckily, Automic stayed up and everything was running as expected until we rebooted.

Support has recommended that we update to the lastest 11.2 as it's possible we are experiencing the memory issues that are resolved as well as the Deadlock messages issue.  We are going to update, but need to seek all of the approvals and timing is always not available due to other work being conducted.  In the meantime, they've recommended that we cycle the WPs on a regular basis to help with the memory and deadlocks until we get approval to update the environment.

I am curious to know if anyone else has had to do this, or are currently doing this as a workaround to potential performance issues.  If so, are you always stopping the job queues before doing this or have the WPs been taking everything over as expected?  Are all WPs including the primary and JWP being cycled, or only the culprits consuming the most memory?

I've received feedback from support, but just wanting to know if anyone else has experienced this and what the results were.

Thanks in advance for the feedback!!

[Mihail Cheie]

We had the same issue until we updated to 11.2.6. We initially used to bounce the system once a week to bring the RAM consumption down (on our server it was raising with 10GB/day). It is better though to cycle through WPs based on the amount of ram they use - stop the ones that have the highest usage from ServiceManager Dialog one by one and try to keep the usage down for best results.

 

We had to stop a WP, wait for a couple of minutes for things to get stable, start it back up, wait another couple of minutes, move to the next one - to not put to much stress on the system.

 

Our WPs use now 500 MB/WP and never raise higher than this.

Hope this helps.

[Tina Hart]

Hello,

This was actually my original post  

We had since upgraded to 11.2.6 earlier this year, and we do still have the memory issues.  Though it doesn't go up as much per day, it's still there.  The latest update from support is that we have to update to the next released SP to resolve all of those issues.

We do still continue to cycle WPs, and that has helped as well.  Our next steps will be to focus on the next major release and moving to this instead.

[Pete Wirfs]

We have been on V11.2.1 for over 2 years and recycle our WPs regularly.   Initially we did full reboots on the weekends, but then we realized that usually only two of our five WPs leak at a time.  We now know that cautiously recycling just those two WPs is sufficient and not very disruptive (I use task manager to identify their PIDs, then recycle them via Service Manager.)  I've been recycling them in the middle of prime shift lately without any ill effects.  We are a relatively small shop though, so we have quiet times during prime shift that I can take advantage of to further reduce risk. 

 

We started an upgrade to V11.2.4 when it became available to repair this problem.  But others reported that it was still leaking too, so we stopped work on the upgrade.  Our current plan is to stay the course with our eyes on V12 next year.

 

Even minor upgrades have risks that demand you TEST ABSOLUTELY EVERYTHING.  When we upgraded from V11.2.0 to V11.2.1, it introduced a new bug that impacted us!  We were able to code a workaround for this new bug, but there is always the potential that the next new bug will not be so easy to deal with.