High CPU causes and mitigation

[CMCN1982]

Hi guys,

one of our application servers recently experienced a spike in CPU usage and this wasn't picked up on for a few days.

Going back through the logs now would be like looking for a needle in a hay stack.

What I would like to discuss is typical causes of high CPU and what can be done to proactively mitigate against them?

Any suggestions would be most welcome. Thank you.

[CMCN1982]

I should add the following:

It is quite erratic and not limited to 1 application server.
We use v13.1 sp6

We have 7 app servers:

Server 1 - nsa, bg and app
Server 2 - app, bg
Servers 3-7 - app servers only

OS: Win 2008 R2 SP1 64bit
Installed Memory: 8GB

[Suman Pramanik]

Hi Syed,

How frequent you see the CPU spike, what database you are using? Is the spike stay for sometimes and comes back normal, you need to check what else is consuming the CPU apart from clarity

Regards

Suman Pramanik

[Andrew M]

I'd start by finding out when this is happening and what applications it is stemming from, so install logging software on the servers.
as you don't know the current time etc you won't be able to easily cross check against the system events log.

In my experience the following could cause CPU spikes
anti virus scanning
multiple jobs running especially the timeslice job
software patching - i.e. patch Tuesday
heavy loads - (load balancer not balancing!)

I'm assuming that the CPU on the server is a modern high powered multi core CPU. Check the spec

[CMCN1982]

I just thought I'd give this thread a bump.

 

Last week, on one of our app servers we saw that CPU had spiked to 100% for a period of time.

When I logged on to the server and checked task manager, it was the java.exe which was consuming the majority of the memory.

 

In the end I had to take the server offline in our webfarm cluster and reboot it. It has been performing fine since.

I checked the app-ca.log file to see if I could spot any errors or indicators there but could not see anything which could explain the spike,

I also checked the event logs but there was nothing useful there either,

 

The CPU on the server is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz 2,40 GHz

 

I realise that it's difficult after the event to pinpoint a cause. But I just wondered if anyone else experienced similar in the past and what they did to determine the cause.

[Anon Anon]

I would advice you to enable heap dump on your system to identify any potential actions that might be causing this hike and eventually making the app crash.

 

TEC594208

 

Title:  How to generate a JAVA HEAP DUMP?

 

1. Add the following argument string in the APP JVM (Application Java VM Parameters) and/or BG JVM (Background Java VM Parameters)
   
   -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<PATH_FOLDER>\<file_name>.hprof
2. Restart APP and/or BG service, depending on which one you have edited.

 

Next time there is an Out of Memory error (java.lang.OutOfMemoryError: Java heap space), a dump will be generated under the path folder location indicated. You can use a 3rd party tool to analyse it or contact CA Support for further assistance.

 

Note: This is valid when Clarity is running on Tomcat or Weblogic as a Web Server.

[CMCN1982]

Thank you for your response.

We already have enabled heap dumps on our application servers and I have analysed them using Eclipse MAT in the past. Heap Dumps are usually generated when the GC Heap threshold is breached.

 

In the high CPU occurrence last week a heap dump was not generated however.

[Aurora_Gaimon]

mmm... it looks like it could be caused by a huge export to excel ....

Check the AWR if you can find what queries are running when CPU is high.

Check in APP logs if you have any reference to export to excel too.

High CPU will not generate the HPROF file (java heap dump) if there is not out of memory, but you can manually capture a heap dump (ie: by using jmap command). It may help to understand what's running in JVM and why it's causing high CPU.

[CMCN1982]

Yes I think the best thing for us to do is whenever there is another high CPU incident, we should check for what queries are occurring at that point in time. We have set up alerts to inform us when CPU goes over 90%, so hopefully next time we can pin down the cause. Thanks to everyone for their advice and suggestions.

[Jeanne Gaskill]

You can also use any tool that monitors threads and tells you which thread is taking up the most CPU.  I have used Top Threads (a plugin for Jconsole) successfully in the past.

 

Sincerely yours,

 

Jeanne Gaskill

Senior Support Engineer

CA Clarity

[Suman Pramanik]

Hi Cmcn,

 

You could probably take a look at the sizing to see if the application indeed need much higher CPU than allocated.

 

Regards

Suman Pramanik