Have other CA Spectrum r10.1 customers experienced SpectroServer memory leaks?

[Brian Hunter]

Since upgrading from CA Spectrum r10.0 to r10.1, we've been experiencing what appears to be memory leaks or continually increasing memory usage of the SpectroServer processes.  We've had a support case open for over a month, so all the obvious things have been checked.  I'm just curious to know if we are the only customers with this issue?  In our experience, r10.0 was very stable in regards to SpectroServer process memory usage.

[YAN.XU]

Hi Brian,

 

We haven't seen much of memory leak issues with Spectrum 10.x. However, it depends on the environment as well. Since you have already raised a ticket with support, they will look into that and advise the best course of action.

 

Cheers.

Yan

[FRANK ELLIGER]

Hi,

we have the same issue. TicketID is 00427135.

Regards, Frank

[Jason Meader]

The only memory leak issue that we’ve seen at multiple sites is due to juniper chassis reconfiguration causing thousands of duplicated modules…check out this tech doc:

 

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1666174.aspx

 

If you do need one of the patches, please open a support ticket and request it.  Also take note the patch only fixes the problem.  It does not delete the duplicate modules.  So once you’ve installed the patch, you’ll need to go into chassis manager, find the models that have duplicate modules.  Select the main module that has all the duplicates, delete it.  Once all the modules are removed, reconfigure the model.  A quick way to find the duplicate modules is to just use CLI:

 

In the bash shell:

Export CLIMNAMEWIDTH=50

./show models > models.out

 

Open the models.out and scroll through.  You’ll quickly notice module duplicates…

 

Once it’s cleaned up you should then save the SSdb and you may need to stop/restart the SS to free up the memory.

 

Hope that helps

Jay

[FRANK ELLIGER]

We don't have Junipers and we have seen the issue as describred by Brian.

CA Support still analyzes our datas. Last update was 3 days ago.

Maybe, we have no leak, but we see "continually increasing memory usage of the SpectroServer processes".

[Anon Anon]

Hi Frank

 

I've commented in your second ticket with potential sources of memory growth, based on your performance data.  There are other plausible explanations for the growth, such as the number and size of host configs, chronic loss of contact, and trap storms.  Dencel should help you rule these out before we assume there is a leak.

 

Scott

[Justin_Kulikowski]

How much time does it take for the memory leak to manifest into a problem?

 

Our SpectroSERVERs are restarted every quarter (~3 months) for OS patching, and within that time I do see some memory growth, but only about ~2-3%.

 

Interestingly though, if I go back to the timeframe when we were running 10.0.0 I don't see that behavior - it remains very flat.  The growth (albeit very small) only appears to begin after our upgrade to 10.1.1.

[Brian Hunter]

We have no juniper devices in our environment, and we don't use Spectrum configuration manager. Under direction of CA support, we tried using the SpectroSERVER_umem binary to do memory debug, but it quickly went to 10GB of memory usage and terminated the SpectroServer after using all available memory.  So that doesn't seem to be an option.  With r10.1 our SpectroServers can't make it two weeks without a restart.

[thomas.w.burgett]

We have 15 locations with Spectrum distributed installation.  Each location that has Juniper devices crash anywhere from once / month, to once/week.

 

We have the same memory leak issue and were advised to upgrade to 10.2.   We Did that, and have the same problem if not a little worse.    1 week after upgrade, the SpeoctroSERVER crashed, and we had to restore from backup to startup again.

We have a P1 incident open now with not much help other than uploading information from logs, etc.

 

Also - wrote a script to watch the memory of that individual process, and once the memory use exceeds threshold for over 3 hours, it will stop, and start SS.   Which still is a problem, but so far prevents crashing the DB, and requiring a restore.

[Brian Hunter]

We still have the same issue, as when I first opened this discussion, with SpectroServers needing to be restarted to "reset" memory usage.  We created a monitor for spectroserver process memory and alert when it exceeds a certain threshold (6GB).  We are running 10.1.2 with various patches.  So far it's unknown as to what is causing the issue.

[karl.haworth]

We're running 10.1.2 and having memory leaks. Support is actively working with us to identify the issue. We do not have Juniper devices. We did not have issues with 10.1.1.

[FRANK ELLIGER]

I have successfully tested 10.1.1_PTF_10.149 and I have closed my ticket.

I think, this  is now part of 10.2.

[Brian Hunter]

Thanks for the info in regards to the patch.  I'll ask support if it was ported for 10.1.2.

[Veronique Lefebure]

We have the same issue since July. We tried several patches with no success. Lately we have been asked to test:

10.2 with PTF_10.2.003 (To avoid SS crash due to Alarm Snooze) + 10.02.00.D122a (snmp layer improvement). We had no time yet to do that. Was anyone successful with that version ? thanks, Veronique

[thomas.w.burgett]

I have 15 Spectrum instances feeding a single MLS for consolidated view of the enterprise network monitoring.

At Version 9.4.2 I was told by support that the problem would be fixed in release 10.2 so I waited.

The problem DID seem to be related to Juniper devices because only 3  of 15 locations were experiencing the memory leak, and those 3 locations had the most Juniper devices.

However, Now that we have 10.2 installed for over 2 months, we still continue to have the same symptoms, but support is saying the leak is with "hos / workstation" models, or Cisco ASA models.  

We still have 3 centers that crash about once per month ( more or less ).

We created a "self heal" script that does a stopSS.pl - wait 10 minutes - startSS.pl.   This allows Spectrum to fail over and fail back which is painful, but not nearly as painful as rebuilding the database from a hard crash.

 

If anyone figures this out, please let me know.  So far, Spectrum developers and engineers have not been able to pin-point the problem.

[Veronique Lefebure]

Thanks for your inputs thomas.w.burgett ! Do you have path 10.02.00.D122a installed on top of 10.2 ?

[Jason Meader]

There have been a couple of different leaks found and there have been patches released.  I believe most of those fixes are all in 10.2 with the exception of the leak due to the SNMP stack.  An updated patch for that was released last week  -- Spectrum_10.02.00.PTF_10.2.001 (which replaces 10.02.00.D122a).

 

Unfortunately memory leaks are one of the hardest issues to diagnose externally due to all the many “moving parts” of the SS ☹ It can take some time to nail down the exact cause of leaks in millions of lines of code.  If you are still having leaks please continue to work with us to resolve them.

 

Thank you

Jay