Duplicate or tardy metric value reported

[pivst01]

hi All,

 

do you have please any idea about the root cause of this warning message that's repeating every 5mins? and solution how to fix it?

 

7/30/15 04:32:25.017 PM CEST [WARN] [Harvest Engine Pooled Worker] [Manager.Agent]  Duplicate or tardy metric value reported: SuperDomain|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)|Enterprise Manager|Analysis Server:Metric Format Time (ms) (time=95884449/last closed timeslice=95884449); ignoring

 

thank you,

stefan

[Hallett German]

Hi Stefan:

    Please review and see if this KB applies. It lists MQ Metrics as a contributor.

 

Thanks

Hal German

APM Support

 

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec595225.aspx

[Hallett German]

"Duplicate or tardy metric value reported" message

could generally indicated the following:

 

1. That duplicate metrics were generated by duplicate instrumentation due

to duplicate pbd files or some java scripts.

2. That there were some connectivity problem between the EMs.

3. EM being overloaded

 

Please consider opening a case to research these further.

[Billy Cole]

Stefan,

 

I found the "MOM_Infra_Monitoring_MM" and the collector.jar management modules very helpful with the duplicate harvest warning messages.

 

1.  Try to determine when the messages started and what changed in your implementation.  Introduction to very broad JavaScript calculators, specific agents producing more metrics, your application generating more traces.  See if the collectors are having issues with meta data writes, smart queries.  If you are having collector issues, work with CA Support and your system admins to find out what resources are the bottle neck.

 

2.  Pull the perf.log for your collectors and MOM and send them to CA Support.  There is quite a bit of useful information within this log.

 

3.  If you find that the tardy/dup metrics are coming only from one host/agent, then check that host to see if it has network bandwidth, cpu ad resources for the agent.  While you are in there, check the system clock on the agent host against your collectors.

 

4.  Depending on your agent/application/trace load, your collectors may not be able to handle and the processing queue gets backed up.  There is a few settings on queue depth/size in the Enterprise manager configuration that might help.

 

5.  Somewhere there is a performance check list, CA support will more than likely send you this when you open a ticket.

 

6.  If you had any JavaScript Calculators deployed and the tardy metric is from the java script, try to narrow the agent/metric expressions and or break up the agent/metrics to smaller chuncks so the java script might have a chance to pull the live metrics, do the math and report back within the 8 second window.

 

The main point here is to find out what agent and hopefully metric is tardy/duplication and then trace how that metric is reported.  If it is from a Java Agent, then is highly advised to check your pbd files like Hal mentioned.

 

Connectivity problems, the infrastructure management modules are very helpful in tracing that down.

 

EM load, get the performance check list from CA Support.  Last I saw the check list there was like eleven bullet points.

 

Hope this helps,

Billy

[Hallett German]

Hi Stefan:

   Thanks for your question, I am having a KB being created for a future reference.

 

Thanks

Hal German

[Anon Anon]

We monitor MQ and this article solved our issue:

Duplicate or tardy metric value reported

Document ID:  TEC595225
Last Modified Date:  8/2/2013
ShowHide Technical Document Details            

• Products                    
  • CA Application Performance Management
• Components                    
  • APPLICATION PERFORMANCE MANAGEMENT

 

Description:

The EMService.log contain many messages "Duplicate or tardy metric value reported", for example:-

SuperDomain|*******|WebSphere|**********/**********Cluster_*******Node01|WebSphereMQ|Connector|mqseries_tst|mqsnttst|Queues|SERVICES.RESPONSE|Get:Concurrent Invocations (time=87239764/last closed timeslice=87239764); ignoring.

SuperDomain|*******|WebSphere|**********/**********Cluster_*******Node01|WebSphereMQ|Connector|mqseries_tst|*CA|Operations|Close:Concurrent Invocations (time=87298547/last closed timeslice=87298547); ignoring

The messages are reported against only those agents enabling the MQ metrics, and all such MQ metrics are aggregated ones (calculated inside of the MQAgent).

Solution:

Please add this additional parameter to the EM properties file:

log4j.logger.Manager.Agent=ERROR

(the ERROR setting will prevent WARNING messages from appearing)

[Hallett German]

Hi Patrick:

This is good to hear. We are actively updating and creating new knowledge documents every day

Thanks

Hal German