DX Infrastructure Management

Expand all | Collapse all

CPU Utilization - total processor utilization across all cores

  • 1.  CPU Utilization - total processor utilization across all cores

    Posted 30 days ago
    CPU Utilization - total processor utilization across all cores is high for certain probes and because of that CPU utilization is very high in the UIM server . What is needed to be done to reduce this cpu utilization !!!



    Regards
    Amar









  • 2.  RE: CPU Utilization - total processor utilization across all cores

    Posted 30 days ago
    high CPU can be caused by probes not having enough memory, slow disks subsystems or a whole host of other things.


    To be able to help you you will need to probe more details as to which probe are causing high cpu there versions.
    your os type and version etc.

    Checking loglevel 3 logs or higher for problem probes is always a good place to start.

    if this just started happening, check the environment for problems first.

    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 3.  RE: CPU Utilization - total processor utilization across all cores

    Posted 26 days ago
    Hi ,

    Windows server 2012 R 2 standard  -- server details 

    EMS probe - 10.22 version 

    EMS probe is taking high cpu utilization and which in turn causing hub for high cpu utilization of over 90 % . 

    Regards
    Amar



  • 4.  RE: CPU Utilization - total processor utilization across all cores

    Posted 26 days ago
    Hi ,

    I also want to know that what happens if I de-activate ems probe and run only nas probe ? Will that affect the alarms flow ? Will we get the alerts ?

    Regards
    AMar


  • 5.  RE: CPU Utilization - total processor utilization across all cores

    Posted 26 days ago
    Hi ,

    Please find the  emslog  details

    yScheduler_Worker-3, ems] ComputerSystemDescriptionCache: Reloading cache
    Sep 20 12:10:46:252 [main, ems] Performing EMS database migrations
    Sep 20 12:10:48:489 [main, ems] Migrations complete
    Sep 20 12:10:49:371 [main, ems] Performing EMS database migrations
    Sep 20 12:10:50:076 [main, ems] Migrations complete
    Sep 20 12:10:59:964 [MyScheduler_Worker-3, ems] Loaded 9408 items into 'ComputerSystemDescriptionCache'
    Sep 20 12:14:16:075 [main, ems] Added 235842 non-existent metric IDs to cache.
    Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.common.rules.RuleManagementServiceImpl to path: @javax.ws.rs.Path(value=api/ems/rules)
    Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.SynchronizeAlarmsServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarm-sync)
    Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.EmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarms)
    Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.InternalEmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/internal/ems/alarms)
    Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.TestShiftServiceImpl to path: @javax.ws.rs.Path(value=api/test)
    Sep 20 12:14:23:731 [main, ems] Registering service name: com.nimsoft.events.nas.NasAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/nas/alarms)
    Sep 20 12:14:23:811 [main, ems] ****************[ Booting ]****************
    Sep 20 12:14:23:811 [main, ems] version 10.2.2
    Sep 20 12:14:23:814 [main, ems] ems 10.2.2
    Sep 20 12:14:23:845 [main, ems] Map DB open subscribers_2 Thread ID 1
    Sep 20 12:14:23:936 [main, ems] Probe exception: com.nimsoft.events.common.exception.EMSRestartException: Data Store Open Failed
    Sep 20 12:14:23:936 [main, ems] =========== START: ShutdownAndRestartCleanup(true) ========
    Sep 20 12:14:23:936 [main, ems] EMS - shutdownAndRestartCleanup begin
    Sep 20 12:14:23:936 [main, ems] EMS - shutdownAndRestartCleanup end
    Sep 20 12:14:25:876 [main, ems] Login to NimBUS is OK
    Sep 20 12:14:28:742 [main, ems] Started quartz scheduler
    Sep 20 12:14:28:825 [MyScheduler_Worker-1, ems] AccountUserEmailCache: Reloading cache
    Sep 20 12:14:28:827 [MyScheduler_Worker-2, ems] MetricTypeDescriptionCache: Reloading cache
    Sep 20 12:14:28:829 [MyScheduler_Worker-1, ems] Loaded 1 items into 'AccountUserEmailCache'
    Sep 20 12:14:29:387 [MyScheduler_Worker-3, ems] ComputerSystemDescriptionCache: Reloading cache
    Sep 20 12:14:29:396 [main, ems] Performing EMS database migrations
    Sep 20 12:14:29:589 [MyScheduler_Worker-2, ems] Loaded 25845 items into 'MetricTypeDescriptionCache'
    Sep 20 12:14:36:812 [main, ems] Migrations complete
    Sep 20 12:14:37:622 [main, ems] Performing EMS database migrations
    Sep 20 12:14:37:711 [main, ems] Migrations complete
    Sep 20 12:14:42:257 [MyScheduler_Worker-3, ems] Loaded 9408 items into 'ComputerSystemDescriptionCache'
    Sep 20 12:18:03:696 [main, ems] Added 235842 non-existent metric IDs to cache.
    Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.common.rules.RuleManagementServiceImpl to path: @javax.ws.rs.Path(value=api/ems/rules)
    Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.SynchronizeAlarmsServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarm-sync)
    Sep 20 12:18:09:381 [main, ems] Registering

    'ComputerSystemDescriptionCache'
    Sep 20 12:18:03:696 [main, ems] Added 235842 non-existent metric IDs to cache.
    Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.common.rules.RuleManagementServiceImpl to path: @javax.ws.rs.Path(value=api/ems/rules)
    Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.SynchronizeAlarmsServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarm-sync)
    Sep 20 12:18:09:381 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.EmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/ems/alarms)
    Sep 20 12:18:09:382 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.InternalEmsAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/internal/ems/alarms)
    Sep 20 12:18:09:382 [main, ems] Registering service name: com.nimsoft.events.alarm_mgr.services.TestShiftServiceImpl to path: @javax.ws.rs.Path(value=api/test)
    Sep 20 12:18:09:382 [main, ems] Registering service name: com.nimsoft.events.nas.NasAlarmServiceImpl to path: @javax.ws.rs.Path(value=api/nas/alarms)
    Sep 20 12:18:09:464 [main, ems] ****************[ Booting ]****************
    Sep 20 12:18:09:464 [main, ems] version 10.2.2
    Sep 20 12:18:09:467 [main, ems] ems 10.2.2
    Sep 20 12:18:09:504 [main, ems] Map DB open subscribers_2 Thread ID 1
    Sep 20 12:18:09:595 [main, ems] Probe exception: com.nimsoft.events.common.exception.EMSRestartException: Data Store Open Failed
    Sep 20 12:18:09:595 [main, ems] =========== START: ShutdownAndRestartCleanup(true) ========
    Sep 20 12:18:09:595 [main, ems] EMS - shutdownAndRestartCleanup begin
    Sep 20 12:18:09:595 [main, ems] EMS - shutdownAndRestartCleanup end
    Sep 20 12:18:11:883 [main, ems] Login to NimBUS is OK
    Sep 20 12:18:16:186 [main, ems] Started quartz scheduler
    Sep 20 12:18:16:266 [MyScheduler_Worker-1, ems] AccountUserEmailCache: Reloading cache
    Sep 20 12:18:16:269 [MyScheduler_Worker-2, ems] MetricTypeDescriptionCache: Reloading cache
    Sep 20 12:18:16:270 [MyScheduler_Worker-1, ems] Loaded 1 items into 'AccountUserEmailCache'
    Sep 20 12:18:16:513 [MyScheduler_Worker-2, ems] Loaded 25845 items into 'MetricTypeDescriptionCache'
    Sep 20 12:18:16:873 [MyScheduler_Worker-3, ems] ComputerSystemDescriptionCache: Reloading cache
    Sep 20 12:18:16:882 [main, ems] Performing EMS database migrations
    Sep 20 12:18:17:099 [main, ems] Migrations complete
    Sep 20 12:18:17:873 [main, ems] Performing EMS database migrations
    Sep 20 12:18:17:965 [main, ems] Migrations complete
    Sep 20 12:18:31:559 [MyScheduler_Worker-3, ems] Loaded 9408 items into 'ComputerSystemDescriptionCache'


    Regards
    AMar


  • 6.  RE: CPU Utilization - total processor utilization across all cores

    Posted 26 days ago
    Hi

    Can try below if it helps as errors look similar

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103137

    Also check count of alarms in nas_transaction_summary and nas_tranaction_log and nas_alarms if count too high in database 






  • 7.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi Franklin ,

    I have tried the below link alreadybut I have observed that the ems probe is taking high cpu utilization  and is fluctuating  high and low . Please let me know what else could help us in resolving this issue to settle down ems probe .

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103137

    Regards
    AMar



  • 8.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi 

    Check if having high count of alarms in nas tables in UIM database 

    nas_transaction_log / nas_transaction_summary/nas_alarms 






  • 9.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi ,

    Yes infact high amount of alarms are there in UIM database .

    I am able to see 245111 alarms in IM console . 
     
    Regards
    Amar


  • 10.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi ,


    We have tried the below link as well . Please tell me if I do this below will that help me !  How to fix this data engine queue / CA UMP loading problem 

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=102315

    Regards
    AMar



  • 11.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi ,

    I also want to know that what happens if I de-activate ems probe and run only nas probe ? Will that affect the alarms flow ? Will we get the alerts ?

    Regards
    AMar


  • 12.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi Franklin ,

    I have observed that even CA UMP is loading on frequent basis . Will you please help us how to resolve that as well . I am suspecting  as the alarms count is seeping high in CA UIM we are facing these issues like  CA UMP loading , data engine in queue and also one more issue is  ems is piling up and  it is taking more cpu utilization .

    Regards
    Amar


  • 13.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi

    High Alarm count could be one symptom causing this issue .Suggest open support case for checking your logs/ enviornment to verify as could be other related reasons

    Below links have some information to reduce the alarms via sql queries / nas settings etc 

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=113086https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=113086

    https://community.broadcom.com/communities/community-home/librarydocuments/viewdocument?DocumentKey=6e1abefd-170e-4341-a5db-19ae0d292345

    Reduce the retention settings in nas gui Setup->Transaction log ( this is local sqllite database in nas folder)
    Setup ->Nis Bridge ( this is in the UIM backend database)


    ->Also new feature In UIM 9.20 to improve USM performance

    • Improved UMP performance (EMS Offloading)The UMP performance has been improved in this release. A new configuration parameter ems_offload_enabled is now available that you can add to wasp.cfg. This parameter enables UMP to bypass the ems probe and directly connect to the CA UIM database to fetch the alarms. This ability improves the UMP performance because UMP no longer goes through ems while getting the alarms data. To enable this functionality, set the parameter value to true and restart the wasp probe; the default value is false. Note that all alarm actions still go through ems. In case of any issue after enabling ems_offload_enabled, review the wasp.log file.

    https://docops.ca.com/ca-unified-infrastructure-management/9-0-2/en/release-notes/ca-unified-infrastructure-management-9-2-0#CAUnifiedInfrastructureManagement9.2.0-PerformanceImprovements


  • 14.  RE: CPU Utilization - total processor utilization across all cores

    Posted 23 days ago
    Hi ,

    Still the ems is piling up continuously after following the below link .Please let me know what needed to be done further .

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103137

    Regadrs
    AMar