CPU Utilization - total processor utilization across all cores

[MADANRAJ SIVAGNANAM]

CPU Utilization - total processor utilization across all cores is high for certain probes and because of that CPU utilization is very high in the UIM server . What is needed to be done to reduce this cpu utilization !!!



Regards
Amar

[Gene Howard]

high CPU can be caused by probes not having enough memory, slow disks subsystems or a whole host of other things.


To be able to help you you will need to probe more details as to which probe are causing high cpu there versions.
your os type and version etc.

Checking loglevel 3 logs or higher for problem probes is always a good place to start.

if this just started happening, check the environment for problems first.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[MADANRAJ SIVAGNANAM]

Hi ,

Windows server 2012 R 2 standard  -- server details 

EMS probe - 10.22 version 

EMS probe is taking high cpu utilization and which in turn causing hub for high cpu utilization of over 90 % . 

Regards
Amar

[MADANRAJ SIVAGNANAM]

Hi ,

I also want to know that what happens if I de-activate ems probe and run only nas probe ? Will that affect the alarms flow ? Will we get the alerts ?

Regards
AMar

[MADANRAJ SIVAGNANAM]

Hi ,

Please find the  emslog  details

yScheduler_Worker-3, ems] ComputerSystemDescriptionCache: Reloading cache
Sep 20 12:10:46:252 [main, ems] Performing EMS database migrations
Sep 20 12:10:48:489 [main, ems] Migrations complete
Sep 20 12:10:49:371 [main, ems] Performing EMS database migrations
Sep 20 12:10:50:076 [main, ems] Migrations complete
Sep 20 12:10:59:964 [MyScheduler_Worker-3, ems] Loaded 9408 items into 'ComputerSystemDescriptionCache'
Sep 20 12:14:16:075 [main, ems] Added 235842 non-existent metric IDs to cache.
Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.common.rules.RuleManagementServiceImpl to path: @javax.ws.rs.Path(value=api/ems/rules)
Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.SynchronizeAlarmsServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarm-sync)
Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.EmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarms)
Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.InternalEmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/internal/ems/alarms)
Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.TestShiftServiceImpl to path: @javax.ws.rs.Path(value=api/test)
Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.nas.NasAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/nas/alarms)
Sep 20 12:14:23:811 [main, ems] ****************[ Booting ]****************
Sep 20 12:14:23:811 [main, ems] version 10.2.2
Sep 20 12:14:23:814 [main, ems] ems 10.2.2
Sep 20 12:14:23:845 [main, ems] Map DB open subscribers_2 Thread ID 1
Sep 20 12:14:23:936 [main, ems] Probe exception: com.nimsoft.events.common.exception.EMSRestartException: Data Store Open Failed
Sep 20 12:14:23:936 [main, ems] =========== START: ShutdownAndRestartCleanup(true) ========
Sep 20 12:14:23:936 [main, ems] EMS - shutdownAndRestartCleanup begin
Sep 20 12:14:23:936 [main, ems] EMS - shutdownAndRestartCleanup end
Sep 20 12:14:25:876 [main, ems] Login to NimBUS is OK
Sep 20 12:14:28:742 [main, ems] Started quartz scheduler
Sep 20 12:14:28:825 [MyScheduler_Worker-1, ems] AccountUserEmailCache: Reloading cache
Sep 20 12:14:28:827 [MyScheduler_Worker-2, ems] MetricTypeDescriptionCache: Reloading cache
Sep 20 12:14:28:829 [MyScheduler_Worker-1, ems] Loaded 1 items into 'AccountUserEmailCache'
Sep 20 12:14:29:387 [MyScheduler_Worker-3, ems] ComputerSystemDescriptionCache: Reloading cache
Sep 20 12:14:29:396 [main, ems] Performing EMS database migrations
Sep 20 12:14:29:589 [MyScheduler_Worker-2, ems] Loaded 25845 items into 'MetricTypeDescriptionCache'
Sep 20 12:14:36:812 [main, ems] Migrations complete
Sep 20 12:14:37:622 [main, ems] Performing EMS database migrations
Sep 20 12:14:37:711 [main, ems] Migrations complete
Sep 20 12:14:42:257 [MyScheduler_Worker-3, ems] Loaded 9408 items into 'ComputerSystemDescriptionCache'
Sep 20 12:18:03:696 [main, ems] Added 235842 non-existent metric IDs to cache.
Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.common.rules.RuleManagementServiceImpl to path: @javax.ws.rs.Path(value=api/ems/rules)
Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.SynchronizeAlarmsServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarm-sync)
Sep 20 12:18:09:381 [main, ems] Registering

'ComputerSystemDescriptionCache'
Sep 20 12:18:03:696 [main, ems] Added 235842 non-existent metric IDs to cache.
Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.common.rules.RuleManagementServiceImpl to path: @javax.ws.rs.Path(value=api/ems/rules)
Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.SynchronizeAlarmsServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarm-sync)
Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.EmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarms)
Sep 20 12:18:09:382 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.InternalEmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/internal/ems/alarms)
Sep 20 12:18:09:382 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.TestShiftServiceImpl to path: @javax.ws.rs.Path(value=api/test)
Sep 20 12:18:09:382 [main, ems] Registering service name: com.nimsoft.events.nas.NasAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/nas/alarms)
Sep 20 12:18:09:464 [main, ems] ****************[ Booting ]****************
Sep 20 12:18:09:464 [main, ems] version 10.2.2
Sep 20 12:18:09:467 [main, ems] ems 10.2.2
Sep 20 12:18:09:504 [main, ems] Map DB open subscribers_2 Thread ID 1
Sep 20 12:18:09:595 [main, ems] Probe exception: com.nimsoft.events.common.exception.EMSRestartException: Data Store Open Failed
Sep 20 12:18:09:595 [main, ems] =========== START: ShutdownAndRestartCleanup(true) ========
Sep 20 12:18:09:595 [main, ems] EMS - shutdownAndRestartCleanup begin
Sep 20 12:18:09:595 [main, ems] EMS - shutdownAndRestartCleanup end
Sep 20 12:18:11:883 [main, ems] Login to NimBUS is OK
Sep 20 12:18:16:186 [main, ems] Started quartz scheduler
Sep 20 12:18:16:266 [MyScheduler_Worker-1, ems] AccountUserEmailCache: Reloading cache
Sep 20 12:18:16:269 [MyScheduler_Worker-2, ems] MetricTypeDescriptionCache: Reloading cache
Sep 20 12:18:16:270 [MyScheduler_Worker-1, ems] Loaded 1 items into 'AccountUserEmailCache'
Sep 20 12:18:16:513 [MyScheduler_Worker-2, ems] Loaded 25845 items into 'MetricTypeDescriptionCache'
Sep 20 12:18:16:873 [MyScheduler_Worker-3, ems] ComputerSystemDescriptionCache: Reloading cache
Sep 20 12:18:16:882 [main, ems] Performing EMS database migrations
Sep 20 12:18:17:099 [main, ems] Migrations complete
Sep 20 12:18:17:873 [main, ems] Performing EMS database migrations
Sep 20 12:18:17:965 [main, ems] Migrations complete
Sep 20 12:18:31:559 [MyScheduler_Worker-3, ems] Loaded 9408 items into 'ComputerSystemDescriptionCache'


Regards
AMar

[Franklin Ravi D'souza]

Hi

Can try below if it helps as errors look similar

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103137

Also check count of alarms in nas_transaction_summary and nas_tranaction_log and nas_alarms if count too high in database 

[MADANRAJ SIVAGNANAM]

Hi Franklin ,

I have tried the below link alreadybut I have observed that the ems probe is taking high cpu utilization  and is fluctuating  high and low . Please let me know what else could help us in resolving this issue to settle down ems probe .

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103137

Regards
AMar

[Franklin Ravi D'souza]

Hi 

Check if having high count of alarms in nas tables in UIM database 

nas_transaction_log / nas_transaction_summary/nas_alarms 

[MADANRAJ SIVAGNANAM]

Hi ,

Yes infact high amount of alarms are there in UIM database .

I am able to see 245111 alarms in IM console . 
 
Regards
Amar

[MADANRAJ SIVAGNANAM]

Hi ,


We have tried the below link as well . Please tell me if I do this below will that help me !  How to fix this data engine queue / CA UMP loading problem 

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=102315

Regards
AMar

[MADANRAJ SIVAGNANAM]

Hi ,

I also want to know that what happens if I de-activate ems probe and run only nas probe ? Will that affect the alarms flow ? Will we get the alerts ?

Regards
AMar

[MADANRAJ SIVAGNANAM]

Hi Franklin ,

I have observed that even CA UMP is loading on frequent basis . Will you please help us how to resolve that as well . I am suspecting  as the alarms count is seeping high in CA UIM we are facing these issues like  CA UMP loading , data engine in queue and also one more issue is  ems is piling up and  it is taking more cpu utilization .

Regards
Amar

[Franklin Ravi D'souza]

Hi

High Alarm count could be one symptom causing this issue .Suggest open support case for checking your logs/ enviornment to verify as could be other related reasons

Below links have some information to reduce the alarms via sql queries / nas settings etc 

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=113086https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=113086

https://community.broadcom.com/communities/community-home/librarydocuments/viewdocument?DocumentKey=6e1abefd-170e-4341-a5db-19ae0d292345

Reduce the retention settings in nas gui Setup->Transaction log ( this is local sqllite database in nas folder)
Setup ->Nis Bridge ( this is in the UIM backend database)


->Also new feature In UIM 9.20 to improve USM performance

• Improved UMP performance (EMS Offloading)The UMP performance has been improved in this release. A new configuration parameter ems_offload_enabled is now available that you can add to wasp.cfg. This parameter enables UMP to bypass the ems probe and directly connect to the CA UIM database to fetch the alarms. This ability improves the UMP performance because UMP no longer goes through ems while getting the alarms data. To enable this functionality, set the parameter value to true and restart the wasp probe; the default value is false. Note that all alarm actions still go through ems. In case of any issue after enabling ems_offload_enabled, review the wasp.log file.

https://docops.ca.com/ca-unified-infrastructure-management/9-0-2/en/release-notes/ca-unified-infrastructure-management-9-2-0#CAUnifiedInfrastructureManagement9.2.0-PerformanceImprovements

[MADANRAJ SIVAGNANAM]

Hi ,

Still the ems is piling up continuously after following the below link .Please let me know what needed to be done further .

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103137

Regadrs
AMar