Hello Jeffrey,
I saw in some logs these errors below:
WARN | FacetBatcher | 2021-01-13 22:57:18,771 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[2638240, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 22:57:57,850 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[127447, {http://im.ca.com/normalizer}NormalizedAddressInfo]
INFO | nstance_Worker-2 | 2021-01-13 22:58:10,982 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: GroupsETLJob completed in 0:00:10.982( 10982 ms )
WARN | FacetBatcher | 2021-01-13 22:58:23,804 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[127742, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 22:58:23,818 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[126684, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 22:58:40,496 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[125104, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 22:58:57,396 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[2296077, {http://im.ca.com/normalizer}NormalizedAddressInfo]
INFO | nstance_Worker-1 | 2021-01-13 22:59:10,720 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: GroupsETLJob completed in 0:00:10.719( 10719 ms )
INFO | hPoller-thread-5 | 2021-01-13 22:59:20,001 | DAHealthPollTaskImpl | itor.poller.DAHealthPollTaskImpl 253 | .ca.im.aggregator.loader | | Failed to find expression for pollable metric family attr {http://im.ca.com/normalizer}NormalizedRollUpInfo.RunCount on vendor cert {http://im.ca.com/selfmonitor}DAHealthRollUpCalculationTimes
INFO | nstance_Worker-1 | 2021-01-13 23:00:11,592 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: PollItemsETLJob completed in 0:00:11.591( 11591 ms )
INFO | nstance_Worker-2 | 2021-01-13 23:00:13,911 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: GroupsETLJob completed in 0:00:13.910( 13910 ms )
WARN | Task-thread-4184 | 2021-01-13 23:00:35,840 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[3046591, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 23:01:03,658 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[125578, {http://im.ca.com/normalizer}NormalizedAddressInfo]
INFO | nstance_Worker-1 | 2021-01-13 23:01:10,820 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: GroupsETLJob completed in 0:00:10.819( 10819 ms )
INFO | uler-thread-1466 | 2021-01-13 23:01:54,150 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 587 | .im.aggregator.discovery | | Discovery of MF {http://im.ca.com/normalizer}NormalizedPortInfo on device 2325176 is queued because related MF {http://im.ca.com/normalizer}NormalizedVirtualInterfaceInfo is running
WARN | uler-thread-1466 | 2021-01-13 23:01:54,156 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedSIPPassiveInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,156 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraDrmaPeerStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,156 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessCDRMEASInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,156 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedPolicyCamInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,158 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageLunInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,159 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessNDSPGWInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,160 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraBberfStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,160 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageSystemInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,160 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageAggregateCapacityStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,165 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageVolumeInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,168 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraBberfPeerStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,168 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessSystemAppKPIInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,169 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessHSSInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,169 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageDiskInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,171 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessSgwKPIInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,172 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraPcefStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,173 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedNuageQoS
WARN | uler-thread-1466 | 2021-01-13 23:01:54,173 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraDraStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,174 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraDrmaStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,175 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessCSCFInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,176 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageLunCapacityStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,177 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessAPNInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,177 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedSDNCountInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,177 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWireslessDiameterMraAfPeerStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,177 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessPGWInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,178 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelesMPEStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,178 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedSDNTunnelInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,179 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraAfStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,180 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessAAA2Info
WARN | uler-thread-1466 | 2021-01-13 23:01:54,180 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessbgppeerInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,181 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessSgwPMEpsCtrlInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,181 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessPGWDSInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,183 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedSDNSLAPathInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,183 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessDiameterMraPcefPeerStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,183 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageVolumeCapacityStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,183 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessTITRAFFICInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,184 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedDCMHealthInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,184 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessmmekpisInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,185 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageDiskCapacityStatsInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,186 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedStorageVfilerInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,187 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedHealthScoreInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,187 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}MobileWirelessPMEpsTrafInfo
WARN | uler-thread-1466 | 2021-01-13 23:01:54,191 | NormalizedFacetSource | lizer.impl.NormalizedFacetSource 282 | .im.aggregator.discovery | | Unable to find MetricFamilyVendorPriority item for MF {http://im.ca.com/normalizer}NormalizedMobileWirelessSIPActiveTestInfo
INFO | nstance_Worker-2 | 2021-01-13 23:02:09,749 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: GroupsETLJob completed in 0:00:09.747( 9747 ms )
INFO | FacetBatcher | 2021-01-13 23:02:31,824 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 1369 | .im.aggregator.discovery | | MF disc of {http://im.ca.com/normalizer}NormalizedVirtualInterfaceInfo on device 2325176 finshed. Invoking a queued MF discovery: NormalizedPortInfo
WARN | FacetBatcher | 2021-01-13 23:02:42,028 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[2325176, {http://im.ca.com/normalizer}NormalizedAddressInfo]
INFO | nstance_Worker-1 | 2021-01-13 23:03:11,095 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener 242 | .ca.im.aggregator.loader | | Batch process job: GroupsETLJob completed in 0:00:11.094( 11094 ms )
WARN | FacetBatcher | 2021-01-13 23:03:11,124 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[127447, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 23:03:11,134 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[124673, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 23:03:11,145 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[124713, {http://im.ca.com/normalizer}NormalizedAddressInfo]
WARN | FacetBatcher | 2021-01-13 23:03:54,254 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl 771 | .im.aggregator.discovery | | Unable to retrieve mfInfo for key=MultiKey[125104, {http://im.ca.com/normalizer}NormalizedAddressInfo]
It happens always in the same time and it takes a long time to finish the poll cycle and all the warn that you saw could be impact in the performance of the server?Is there anything to improve or clean the vertica database,because it worked very well and in these days has started to take a long time to finish de poll cycle and all the graphs are full of the gaps.
thank you
Original Message:
Sent: 01-12-2021 08:18 AM
From: JEFFREY PINARD
Subject: CAPC - Poll Cycle Percent
I believe this falls under the Threshold/Eventing self-monitoring.
It means it's taking a long time to process threshold profiles.
Either, you have:
1. Too many profiles that have the same MF (the more profiles, the longer it takes to process them each poll cycle - we process each profile for a MF at a time per data load)
2. There are many data loads per poll cycle (due to number of items being polled). We run profiles per data load.
3. Your windows are too large (looking at tons of data to determine whether meets criteria)
4. Your DR are too slow, either cpu or disk, or not enough memory to allocate to the query to make it run at peak performance.
You may want to open a support ticket so they can delve further into the self-monitoring metrics to find where the time is being taken.
Original Message:
Sent: 01-12-2021 07:52 AM
From: Valeria Cunha
Subject: CAPC - Poll Cycle Percent
Hello CA Team,
I would to know how can I decrease the Poll Cycle Percent in CAPC because all the graphs has been full of gaps because of it.I realized that happens at dawn,so I attached here the image you can see the problem happening better.
thank you,