CA PC / Karaf.log Error Message - Failed to Push 1200 Events

[LM04]

I see the following recurring error in the /opt/IMDataAggregator/apache-karaf-2.4.3/data/log/karaf.log on our CA PC Data Aggregator:

ERROR | EventPush | 2020-11-18 10:40:19,490 | EventProducerWS | or.eventproducer.EventProducerWS 945 | a.im.NPCConnector.bundle | | Failed to push 1200 events: org.apache.cxf.interceptor.Fault: Marshalling Error: No route to host (Host unreachable)

Our DR recently crashed.  It is now back up and running.  The DR was started then the DA then the DC's.  Everything looks fine in CAPC and the all system statuses show as "good".  I'm not sure what the above error message means or how to resolve it.  Has anybody else run across this error?

[Jeffrey Pinard]

Please check http://DA:8581/rest/emregistration

What is the URL it's using.  Does the DA have access to get to that EM IP and port?

We need port 8281 open on PC server for the DA to connect to, so it can send events.

Does the PC have multiple IPs, is the emregistration using the correct one to connect to?

[LM04]

Sorry, forgot to mention we are using Linux servers.

[LM04]

When I go to http://DA:8581/rest/emregistration the IP listed for "eventConsumerURL" is incorrect.  How do I change it?
Also the Event Management data source  in CA PC is correct and showing as Available.

[Michael Poller]

Instructions in the following KB article will help fix that wrong value.

https://knowledge.broadcom.com/external/article/106090

------------------------------
Technical Support Engineer IV
Broadcom
------------------------------

[LM04]

I've tried both suggestions, and unfortunately, neither worked for us.  Based on the KB articles everything looks correct except for http://DA:8581/rest/emregistration. Instructions in the KB articles have been followed and services have been stopped/restarted on the DC's, DA + PC.  For now, we have decided to just turn off events since we are not using them anyway.  I was hoping if I disabled the Event Manager data source and stopped the event manager service on Performance Center that the error would go way, but I still see it in the karaf .log.

Is there anything else I can do or turn off to stop the error from showing up in the karaf.log?  The error shows up multiple times per second and my concern is that eventually the log will consume too much disk space.

[Jeffrey Pinard]

So...

1) Confirm hostname/IP in Event Manager data source is correct.
2) Confirm in mysql using "select * from em.general;" that WebSite.Address is same as the EM data source hostname/IP value. 
    If this isn't right, the Event Manager URL sent to DA will be wrong. 
    If it doesn't match, you can try updating the EM data source again, or do a full EM sync. 
    When the sync completes, check WebSite.Address attribute again in em.general.
3) Then run this again:  update em.data_sources set LastEvent=0,ConsumerID=0;
4) Then kick off an incremental sync for EM.
5) When EM completes PUSH, it will kick off Inventory sync which will beginInventory/pullInventory/endInventory first.  Then it will print to EMService.log "Polling Event Producers".  At this point, it re-registers with the data sources (like DA) since we set LastEvent/ConsumerID to 0.   See if we see any errors or never see this section in EMService.log.

Then check emregistration URL again.

You can send us the EM wrapper-<DATE>.log for when you ran these steps, if it still doesn't work.