EM Memory Issue

[Anon Anon]

We are seeing our EM is utilizing lot of memory and we understand we should increase the heap size. But we can see on our logs that a specific thread is causing memory to become full. We want to know what this thread does. The thread is Metadata Perf Metric Updater. Error log is below.

 

 

[ERROR] [Metadata Perf Metric Updater] [Manager] Uncaught Exception in Enterprise Manager:　 In thread Metadata Perf Metric Updater and the message is java.lang.OutOfMemoryError: Java heap space

[David Lewis]

Hi,

It's not necessarily the same correlation, this is usually what is busy when the out of memory condition is reached but it isn't necessarily the actual cause of out of memory.

Out of memory is just undersized EM for heap because there is too much load, and/or you could have a lot of historical Smartstor data and traces data.

 

Basically, the metadata is like an index of the Smartstor metric data. Generally speaking this should only be being updated if new metrics are being reported to the EM, if you get the same metrics reported over and over again, the metadata doesn't need to be updated, so if

it is constantly busy, you might have what we call a metric leak where you are constantly getting new metrics (could be related to Sockets, JMX or SQL metrics, anything that could change frequently and randomly)

I guess the way to think of it is if you didn't have the metadata, even if all the Smartstor data was still stored, you wouldn't be able to search for the data, so you would effectively have no (access to) historical data

 

You could have a look at the amount of Historical Metrics of the EM to see if that is constantly rising. You could also check the Historical Metric Count value for agents that have a high value of that, then work out what types of metrics are causing it to grow.

Even if the heap is raised, if the historical data keeps on rising, you might be coming back to the same problem a few months down the line.

 

Thanks,

David