SpectroSERVER crashes with error: SpectroSERVER.exe is out of memory while allocating 64 bytes. Continuing shutdown. CA memory allocation error. Aborting.

[Legacy User]

We are running CA Spectrum 10.2. The primary SpectroServer in our FT environment has been crashing frequently for the past few days. This is our MLS as well.

We had an issue with the Archive Manager. The DDMBB\Event table had grown to a size of about 97 GB and Archive Manager failed to start. Multiple attempts to repair the table failed. As a last resort, the table was truncated, and we were able to get ArchMgr up and running.

 

Since then, SpectroServer crashes at least once a day. VNM.out is attached.

 

Any ideas what might be causing this?

Attachment(s)

VNM.OUT.zip   3 KB 1 version

[Jason Volpini]

Hi Basam

 

I have taken a look at the VNM.out you provided. I am seeing a few things here.

 

One is there are messages on missing intelligence modules. I ran some search on this and didn't find an exact match for Spectrum - so not exactly sure what the cause of this is. Are you starting the SpectroSERVER with the Spectrum Admin user?  Did you load a previously good Backup SSdb into the server after the crash?

 

Jul 19 04:46:51 : Validating and loading SPECTRUM Intelligence Modules.

Error while loading SPECTRUM Intelligence Modules:

D:/win32app/Spectrum/SS/Modules/lib3cm1009ss.dll not loaded : The specified module could not be found.

 

D:/win32app/Spectrum/SS/Modules/lib3cm1011ss.dll not loaded : The specified module could not be found.

 

D:/win32app/Spectrum/SS/Modules/lib3cm1012ss.dll not loaded : The specified module could not be found.

 

 

As for the crash, this looks like its caused by flood of inbound trap processing and Spectrum ran out of Memory.

 

First there are timeout delay messages:

 

Jul 19 15:08:51 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.

Jul 19 15:08:51 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.

Saved compact diagnostic file to 'D:\win32app\Spectrum\SS\support\SpectroSERVER_20180719_1508.dmp'

 

A stack on trap processing:

 

0x7fee728085alibalert.dll!TrapQueueNode::TrapQueueNode

0x7fee7281dd3libalert.dll!RemoteForwardingTrapQueue::queue_trap

0x7fee7281b69libalert.dll!CsAlertManagerMT::process_trap_MT

0x7fee7281f35libalert.dll!CsAlertManagerMT::trap_receiver

 

Then the server shuts down out of memory:

 

Jul 19 15:09:46 ERROR TRACE at VNM.cc(841): D:/win32app/Spectrum/SS/SpectroSERVER.exe is out of memory while allocating 64 bytes.  Continuing shutdown.

CA memory allocation error. Aborting.

 

 

First you can try disabling Unmanaged Trap Handling and see if that helps the server stay up. Setting is on VNM model > Trap Mgmt subview

 

 

 

Next you'll want to find what device(s) are spamming your server with traps. Spectrum alarms when a device hits Spectrum with a trap storm - event 0x10253 TRAP STORM DETECTED. You can review a summary of models asserting this alarm and see if that helps determine which devices to check for excessive or unwanted/unnecessary traps.

 

1. Nav to <SPECROOT> and run the following:
2. ./perfCollector9
3. It will output to <SPECROOT>/Performance
4. In /Performance directory look for a file for Summary for event 0x10253 - Trap Storm Detected - filename will be called <hostname>_0x10253_summary.txt
5. Review the model handles in this report and use Locator Search > by Model > by Model Handle to find the devices.
6. You can also use the Search as a List to find them in bulk

 

 

Hope that helps,

Jay V

 

 

 

 

[Legacy User]

Hi Jay,

 

Thanks for the response.

 

The missing intelligence modules errors have been there for quite some time, but never impacted the running of the SpectroServer.

 

 I am starting the SpectroSERVER with the Spectrum Admin account and I did load a good backup after the crash.

 

I have disabled Unmanaged Trap Handling.

We did have a Trap Storm alert from a managed device last week and the network team got it sorted.

I will run perfCollector9 to see what devices are spamming the server with traps.

 

One thing I noticed is that the Archive Manager on the secondary SpectroSERVER was also running all this while. I normally have ArchMgr running only on the primary server, I guess it was recommended by CA.

After I stopped ArchMgr on the secondary server, memory utilization has gone down and there are no timeout errors anymore in SCP. SpectroServer.exe memory utilization has come down from 3GB to less thann 2GB. At least the server available memory is not hovering around 0MB anymore. I have also requested for a server memory upgrade to be on the safe side.

 

Basam

[Legacy User]

The server has been up since making the above two changes. We have a scheduled task to backup device configurations from 1AM - 4AM daily. I am seeing timeout messages starting 1:06AM.

 

Jul 21 01:06:43 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 01:06:43 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 01:08:07 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.

 

Memory utilization has also gone up.

Even though both primary and secondary are running, the status in the OneClick console is red (down).

I cannot even open the Performance view. It says the SpectroSERVER doesn't seem to be running.

[Legacy User]

Things seem to be back to normal.

SS status on the console is back to green. Memory utilization is stable.

 

Apart from the 30 or so minutes where we had timeout errors, the SS has been behaving in a good manner.

The timeout errors might corresponded to high I/O wait time for the OS. 

 

The SS has been running for 20 hours now

[Legacy User]

The SS crashed again.

 

VNM.OUT output:

Jul 21 07:56:44 ERROR TRACE at VNM.cc(797): D:/win32app/Spectrum/SS/SpectroSERVER.exe is out of memory while allocating 48 bytes. Scheduling shutdown.
Saved compact diagnostic file to 'D:\win32app\Spectrum\SS\support\SpectroSERVER_20180721_0756.dmp'

0x7feec9484f6libGlobl.dll!CsSymbolInfo::get_current_context
0x7feec9485a9libGlobl.dll!CsSymbolInfo::print_current_stack
0x7fee9c25e15libsskrnl.dll!SearchManager::update_mh_searches
0x7fef56031f2libPort.dll!Cs_new_handler
0x7fef56085b6libPort.dll!malloc
0x7feeb6b5540libdcm.dll!CsDcmIfBlk::get_varbind
0x7fee9f1d88elibalert.dll!CsDcmAlert::CsDcmAlert
0x7fee9f1dd9blibalert.dll!CsSnmpAlert::CsSnmpAlert
0x7fee9f125a5libalert.dll!CsAlertManager::create_alert
0x7fee9f21360libalert.dll!CsAlertManagerMT::process_trap_MT
0x7fee9f21f35libalert.dll!CsAlertManagerMT::trap_receiver
0x7fee9f21dfdlibalert.dll!CsAlertManagerMT::receiver_func
0x7feec89769blibmoot.dll!IOEvent::setup_fd
0x7777866d kernel32.dll!CreateFiberEx

Jul 21 07:57:55 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 07:58:05 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 07:58:06 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 07:58:27 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 07:58:30 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.
Jul 21 07:58:30 WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 60000ms.

Jul 21 07:58:38 : SpectroSERVER has received shut down signal - scheduling shut down
Jul 21 07:58:38 : D:/win32app/Spectrum/SS/SpectroSERVER.exe is shutting down...
Jul 21 07:58:38 ERROR TRACE at CsIHPrtIPLS.cc(1090): Waited 60000 seconds for IPLS evaluate lock for mh: 0x1623e0, continuing without lock
Jul 21 07:58:40 ERROR TRACE at VNM.cc(841): D:/win32app/Spectrum/SS/SpectroSERVER.exe is out of memory while allocating 64 bytes. Continuing shutdown.
CA memory allocation error. Aborting.

 

I am running perfcollector9 to find the offending devices.

 

I also saw the below KB article published yesterday.

Why does the SpectroSERVER shutdown very soon afte - CA Knowledge 

 

What are the implications of disabling all trap processing on the SS?

[Jason Meader]

Good to hear  

[Legacy User]

I ran perfCollector9. I didnt get any <hostname>_0x10253_summary.txt file.

I do see 3 devices that are constantly having Trap Storm Detected alarms in the console.

I will disable trap-based events for these devices till the network team investigates why they are generating excessive traps.

[Jason Meader]

Hi Basam,

   The stack you posted on the 21st is noting trap processing again.  You may find that even though you disable trap based events, it may not relieve the memory issue because if the traps are sent to the SS, the SS will still need to "process" them.  The issue will need to be fixed at the device(s).  The only other temporary options are to change the SS trap port in the .vnmrc and cycle the SS (but keep in mind that will change it for all devices, so if you don't update your devices you won't process any traps in Spectrum) or use a temporary firewall configuration to block the traps into the SS.

After fixing the 3 offending devices if you still see this issue, it may be good to start a sniffer trace and see what traps are being sent in and from what devices.

Cheers

Jay

 

PS  The <hostname>_0x10253_summary.txt would be in the $SPECROOT/Performance directory.  You should have other summary files in there as well.  If there aren't any summary files then one cause could be the user that you ran the perfcollector command with is not a user in the SSdb.  

[Legacy User]

Thanks for the reply Jason.

 

The 3 devices are being investigated.

And you are right, disabling all traps is not a viable solution.

 

I have also requested for a memory upgrade for the server.

 

I ran the perfcollector command using another user. Will run it again using the correct user credentials.

[Shane Ryan]

Hi Basam,

 

the performance data contains sensitive device information.

please open an issue to securely upload it and we can help you to examine it in a more private environment.

 

best regards

Shane

[Eduardo Ignacio Serrano]

Hi, Legacy User

I'm having a similar memory issue. The only difference is that this SpectroServer isn't receiving traps, nor am I getting "trap storm" alarms. Memory usage spikes quickly; this started after I initialized the DDM because storage usage had climbed to 85%. It didn't crash, but I did restart it, which freed up storage-yet since then, memory is being consumed much faster. How did your issue get resolved? Could you recommend a fix?

Regards,

[Sanyasirao Chillarige]

Excessive events can cause high RAM and CPU utilization in the
SpectroSERVER, Archive Manager (ArchMgr), and MySQL processes. When an
event storm occurs, the Archive Manager process grows in RAM size; if it
exhausts available memory, the SpectroSERVER may gracefully shut down to
avoid database corruption.
Identifying the Event Storm

You can query the DDM database on the SpectroSERVER to identify which
events and models are flooding the system.

1. Log into the SpectroSERVER as the user that owns the installation.
2. Navigate to $SPECROOT/mysql/bin.
3. Log into the Archive Manager MySQL DB: ./mysql
--defaults-file=../my-spectrum.cnf -uroot -p MySqlR00t
4. Run the following queries to identify top offenders:

Top 10 events most commonly generated:
sql

SELECT hex(type), COUNT(*) as cnt FROM event GROUP BY type ORDER BY
cnt DESC LIMIT 10;

Top 10 models with the most events:
sql

SELECT hex(e.model_h), m.model_name, COUNT(*) as cnt FROM event e,
model m WHERE e.model_h=m.model_h GROUP BY e.model_h ORDER BY cnt DESC
LIMIT 10;

Recommendations

- Disable Unmanaged Trap Handling: If the storm comes from unmanaged
devices, turn off "Unmanaged Trap Handling" on the VNM model for a
significant performance improvement.
- Maintenance Mode: Once the offending model is identified, place
it in Maintenance
Mode to stop further event generation until the root cause is addressed.
- Event Configuration: If certain informative events (e.g.,
Authentication Failure) are not needed, disable "Store Event in Historical
Database" in the Event Configuration utility in OneClick.

Hope this helps

Sunny