Interruption of DataAggregator and Performance Center in DX PM

[José Carlos Montes de Oca Luna]

Good afternoon, Team.
I have some questions and issues regarding DX PM.
I have a distributed environment with Fault Tolerance; the service is designed to automatically restore when the primary DA1 goes down-a process that takes about 45 minutes for the secondary DA2 to start up. (While this failover used to happen, it was a rare occurrence in the past.) However, over the last two weeks, I have frequently experienced a loss of connectivity with the DAs, causing the Performance Center service ("caperfcenter_console") to shut down; I then have to manually restart it to bring the graphical interface back online. Once restored, I notice the platform is often sluggish when querying reports or dashboards, and subsequently, the DA, Repository, and Collector servers trigger "Degraded" or "Failed" alarms within the Performance Center interface. I checked the PCservice.log files and found the following message from the time the system outage occurred:

PCservice.log:

ERROR | Status Poller-thread-1   | 2026-07-21 22:00:23,022 | com.ca.im.portal.api.xml.parser.XmlParser                        
      | Failed to parse XML 

En el log Wrapper:

INFO   | jvm 9    | 2026/07/21 21:59:17 | WARN  | RIB/SupportedMFCacheScanner | 2026-07-21 21:59:17,130 | com.ca.im.portal.plugins.polaris.cache.utils.DAWsCacheUtils      
INFO   | jvm 9    | 2026/07/21 21:59:17 |       | Encountered status code 503 while connecting to http://PERFORMANCEQROPROXY:8581/genericWS/metricFamilies
INFO   | jvm 9    | 2026/07/21 21:59:17 | WARN  | RIB/SupportedMFCacheScanner | 2026-07-21 21:59:17,131 | com.ca.im.portal.plugins.polaris.cache.metricfamily.SupportedMetricFamiliesWsDispatcher
INFO   | jvm 9    | 2026/07/21 21:59:17 |       | Unable to find any metric families at http://PERFORMANCEQROPROXY:8581/genericWS/metricFamilies
INFO   | jvm 9    | 2026/07/21 21:59:23 | ERROR | Status Poller-thread-1   | 2026-07-21 21:59:23,012 | com.ca.im.portal.services.status.impl.DcStatusServiceProvider    
INFO   | jvm 9    | 2026/07/21 21:59:23 |       | Failed to make rest call: 503
INFO   | jvm 9    | 2026/07/21 21:59:23 | ERROR | Status Poller-thread-1   | 2026-07-21 21:59:23,013 | com.ca.im.portal.api.xml.parser.XmlParser                        
INFO   | jvm 9    | 2026/07/21 21:59:23 |       | Failed to parse XML
INFO   | jvm 9    | 2026/07/21 21:59:23 | org.xml.sax.SAXParseException: Content is not allowed in prolog.

Log DMService: 

INFO  | pool-3-thread-1          | 2026-07-21 22:00:05,399 | com.ca.im.portal.dm.productsync.RuleSync                         
      | End item relation validation with result = true
ERROR | pool-4-thread-3          | 2026-07-21 22:00:20,455 | com.ca.im.portal.api.services.datasource.DataSourcePoll          
      | Received WebServiceException from version check for data source Data Aggregator@PERFORMANCEQROPROXY.  CAUSE=org.apache.cxf.transport.http.HTTPException: HTTP response '503: Service Unavailable' when communicating with http://PERFORMANCEQROPROXY:8581/DataSourceWS. MESSAGE=Could not send Message..  Returning DS_COMM_FAILURE result.
ERROR | pool-4-thread-3          | 2026-07-21 22:00:20,455 | com.ca.im.portal.api.services.datasource.DataSourcePoll          
      | javax.xml.ws.WebServiceException: Could not send Message.
ERROR | pool-4-thread-3          | 2026-07-21 22:00:20,455 | com.ca.im.portal.dm.productsync.DataSourcePoller                 
      | Unable to contact Data source DS(3) Data Aggregator@PERFORMANCEQROPROXY
ERROR | pool-4-thread-3          | 2026-07-21 22:00:20,481 | com.ca.im.portal.dm.productsync.SyncPhaseExecutor                
      | Aborting sync phase for DataSource Data Aggregator@PERFORMANCEQROPROXY because an exception occurred: com.ca.im.portal.api.services.interfaces.datasource.DataSourceOp$Exception: enum.datasourceerror.DS_COMM_FAILURE
INFO  | pool-4-thread-2          | 2026-07-21 22:00:20,552 | com.ca.im.portal.dm.productsync.DataSourcePull                   
      | BEGIN PULL: DS(2): http://PERFORMANCEQROAPP:8281/EventManager/ProductSync2WS.asmx
INFO  | pool-4-thread-2          | 2026-07-21 22:00:20,552 | com.ca.im.portal.dm.productsync.DataSourcePull                   
      | DS(2) SyncState[ClientId 00603f55-922f-42de-b0b5-b3f2189be900, Base 1756147160, Increment 1784692389]
WARN  | pool-4-thread-1          | 2026-07-21 22:00:20,555 | com.ca.im.portal.dm.datasourcedb.DBDataSourceDAO                 
      | persistDataSource - delete SRM RIB IRIBSourceWS interface for dsId: 4

I would greatly appreciate your help in explaining what has happened, based on your knowledge, and knowing how I can proceed with this.

Thank you for your time.

Best regards!

[Jeffrey Pinard]

So when a DA goes down fully, aka can't even reach vertica, in a FT env, it takes 5 mins for the other DA to be triggered to start the AMQ/DA processes.

Then it allows the DA 45 mins to come up before giving up and stopping the DA.  It shouldn't take 45 mins for the DA to start unless huge environment and slow vertica DB.

Check the DA trying to start under IMDataAggregator/consul-ext/data/logs/consul-ext.log  See if it's able to get to DA rest port.

If it's not able to see DA rest in 45 mins it will shutdown.  If DA is taking a long time to start, check the DA data/logs for any errors.

Ignore the DM logs until you can confirm DA is up and DA rest is working.  Then if it is, check DM logs are able to sync with DA.

I might suggest opening service ticket if things don't fully start on DA.  They can get you up and running faster.