DX Infrastructure Management

Just for note: with nas 8.42 the problem occurs.

Aug 02 09:24:06:737 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count done waiting for alarms to be flushed time: 20000
Aug 02 09:24:08:755 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count waiting for alarms to be flushed
Aug 02 09:24:28:755 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count done waiting for alarms to be flushed time: 20000
Aug 02 09:24:30:767 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count waiting for alarms to be flushed
Aug 02 09:24:50:769 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count done waiting for alarms to be flushed time: 20002
Aug 02 09:24:52:775 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count waiting for alarms to be flushed
Aug 02 09:25:12:782 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count done waiting for alarms to be flushed time: 20007
Aug 02 09:25:14:790 [attach_clientsession, alarm_enrichment] AlarmQueueReader alarm count waiting for alarms to be flushed

..............

Hi Dan,

That key post_max_age = 60 sets a timeout for message from the message bus. By default, messages older than 2 seconds are dropped, and this increases the timeout to 60 sec. So rather than causing messages to be lost (such as alarms), it should do the opposite.

If you are slightly unsure about its impact you could increase it more slowly (maybe start with 10 or 15)

cheers

Rowan

Hi Daniel,

Yes, the post_max_age setting should also be added into the hub probe via Raw Configure as well.  I was mainly pointing out in my last comment that the extra three lines have broken several customer environments.  The following are the correct settings that should be added via Raw Configure in the probe setup section.

*  nas

lower_memory_usage_threshold_percentage = 0.90
upper_memory_usage_threshold_percentage = 0.90
memory_usage_exceeded_threshold = 3

*  hub

post_max_age = 60 

Thank you also for your time in pointing this out.

Regards,

Ryan Currey

Hi Yu, I know that.. I'm asking Ryan if we need that additional line added to the hub's CFG or do we just need the 3 lines added to the nas's cfg?

Hello.

post_max_age is a key in hub.cfg (in under <hub> section)

Regards,

Yu Ishitani

Hi Ryan,

what about that extra hub.cfg line? Is that needed or just the 3 lines in the nas.cfg <setup> section?

Is this needed:

Hub:
<hub> overwrite
post_max_age = 60
</hub>

Thanks,

Dan

Thanks Ryan, I have edited the original post to reflect your comments.

In order to correct this issue, ONLY the following keys need to be added:

lower_memory_usage_threshold_percentage = 0.90
upper_memory_usage_threshold_percentage = 0.90
memory_usage_exceeded_threshold = 3

Please refer to the steps in the below KB article:

*  alarm_enrichment probe become unresponsive 

I have had several customers say that when adding the following extra keys, it broke alarm_enrichment:

restart_memory_usage_threshold_percentage = 0.85
force_flush_active_alarms_timeout = 300
force_flush_active_alarms_before_shutdown = yes

Regards,

Ryan Currey

UIM Support

Hi Daniel,

I follow-up the ticket you opened and it seems CA support provided you a fix (not sure if you installed it or not) and the issue no longer happens. Can you confirm so we can put some light on this thread?

Thanks!

Nestor

This did not help my issue. I have the exact problem and tried running nas versions 4.94 and 8.43 and with these options set still hitting the alarm_enrichment queue backup stoppage. 

Have a case open.

Second Daniel's statement. Experienced similar case interaction several months back when last requested CA assistance on the topic.

Honestly this is another blatant example how CA needs an announcement board for discovered defects w/in this product.

I opened the same exact case a few weeks ago and was being handled by another engineer. I got a sub-set of these suggestions only for the nas probe and it seemed to have fixed this exact issue. Overall since upgrading to 8.4 Sp2 our nas has had many issues. I shared this topic with him and he had no idea. 

CA how about communicating w/in your own support teams issues that pop up so its not a run around from scratch each time. 

CA, is this fixed in 8.5? We are running 8.4 SP2 and hit this exact issue many times. I added the hot fixes and the options mentioned above but just want to know, in 8.5 is this fixed and will it work w/o the additional options added to the nas and hub probe? 

Can someone explain what exactly does the   "post_max_age=60" mean? I like to know what this does in case we start seeing odd behavior.

Hello.

Please vote.

Include some important keys for alarm_enrichment probe as default. 

Regards,

Yu Ishitani

Not really, i would use the settings during implemenations to avoid this problem

Hi Luc,

If you do encounter the repetitive message as above, then yes, go ahead and make the above changes.

Thanks.

Regards

-Sayeed

Can we add these settings as a default?