Two MOMs in wily 9.1.1 cluster

[Anon Anon]

So here is the situation:

We have a Wily 9.1.1 cluster with 2 collectors and 1 MOM. That's working fine. But now we want to add another EM to the cluster as a MOM.
So the requirement is to connect two MOMs to the same cluster at same time (no active/passive thing as in case of hot-failover).

This was working in our 8.2.3.8 environment but after upgrading to 9.1.1 we are getting the following exception in MOM log file.
It seems that in 9.1.1 only one MOM can communicate to collectors because of some clamped value assigned to them. Is there a way to resolve this issue?

Other thing is even if we would be able to connect these 2 MOMs to the cluster,
will it be possible to report data(CEM and Triage Map info) from the 2 collectors in our cluster to two different Postgre instances?


**********************************************************************************************************************************************************************************************************************************************************************************************************************


00/00/12 00:00:00.683 AM EDT [DEBUG] [Collector ***.***.***.***@5001] [Manager.Cluster] Failed to connect to the Introscope Enterprise Manager at ***.***.***.***@5001 (21) because: The Collector has rejected the connection as the clamp value for MOM is reached[size=7][color=#ff4f4f][color][size].
00/00/12 00:00:00.683 AM EDT [DEBUG] [Collector ***.***.***.***@5001] [Manager.Cluster]

com.wily.isengard.messageprimitives.MaxMomsExceededException: MOM connection from Node=Workstation_26, Address=***.***.***.***/***.***.***.***:14880, Type=socket was rejected since the clamp value of 1 has been reached
at com.wily.introscope.server.beans.session.SessionBean.authenticateMOM(SessionBean.java:398)
at sun.reflect.GeneratedMethodAccessor511.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.wily.isengard.messageprimitives.MethodCallUtilities.callInterface(MethodCallUtilities.java:75)
at com.wily.isengard.messageprimitives.MethodCallUtilities.callInterface(MethodCallUtilities.java:29)
at com.wily.isengard.messageprimitives.service.MessageService.attemptMethodCall(MessageService.java:183)
at com.wily.isengard.messageprimitives.service.MessageService.handleMethodCallMessage(MessageService.java:135)
at com.wily.isengard.messageprimitives.service.MessageService.receiveMessage(MessageService.java:161)
at com.wily.isengard.postoffice.Mailbox.handleMessage(Mailbox.java:252)
at com.wily.isengard.postoffice.PostOffice.deliverInternal(PostOffice.java:501)
at com.wily.isengard.postoffice.PostOffice.access$1(PostOffice.java:447)
at com.wily.isengard.postoffice.PostOffice$DeliveryItem.run(PostOffice.java:786)
at com.wily.EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:728)
at java.lang.Thread.run(Thread.java:662)

[Legacy User]

CA deliberately prevents connecting two MOMs to a collector since 9.1. We personaly would like as well a hot failover mechanism with 2 MOMs or partialy even more e.g. 2 just for alerting and the other for users.

MOM HA without relying on any OS/HW feature is something i just recently requested with the MyCAs Idea tab: https://caideation.secure.force.com/ideation/ideaView?id=08730000000i7ay

the dual active/active MOM would clearly be one way to do it, and we did this in our 8.2 environment and got advised by CA to NOT do it as it might causes issues, which we didnt fully understand but as we have a stability issue just followed the advise (didnt help with the stability issue but untill its solved we leave the active / active MOM topic by side).

[Guenter DI Grossberger]

Hi,,

In an EM cluster you can only have one MOM but since 9.1 you can have several Cross-cluster Data Viewers (CDV) in addition.

BR,
Guenter

[Anon Anon]

@aaronritter
Are there any known issues or is it an 'Unpredictable Behavior' thing?

@Guenter
We have a special requirement where we are using the second MOM to access the cluster in case the first one goes down.
We cann't use CDV in this case as CDV is not an equal substitution for a MOM.

[Anon Anon]

Hi
We would like to know as well if we enable Active Active MOM scenario what impact will it have on the cluster ??

Thanks

[Legacy User]

manvinderboparai4204892 wrote:

@aaronritter
Are there any known issues or is it an 'Unpredictable Behavior' thing?

We did not get answerd why we are not allowed to run two MOMs and what the known issues would be. The only thing we got told is that it likely has a unwanted performance impact.

GuenterGrossberger wrote:

Hi,,

In an EM cluster you can only have one MOM but since 9.1 you can have several Cross-cluster Data Viewers (CDV) in addition.

BR,
Guenter

CDV does not solve the fact that the MOM needs to be high available including its different functions e.g. alerting / dashboards / workstation / etc.

manvinderboparai4204892 wrote:

@Guenter
We have a special requirement where we are using the second MOM to access the cluster in case the first one goes down.
We cann't use CDV in this case as CDV is not an equal substitution for a MOM.

this is our issue too, you could help by vote for the idea i posted. i have commented the active/active HA MOM solution as one potential idea in how to solve this

Srikant.Noorani wrote:

Hi
We would like to know as well if we enable Active Active MOM scenario what impact will it have on the cluster ??

Thanks

in 9.1 you can't activate it because the second mom gets blocked, and for 8.2 we did not got answered the question. Maybe someone from CA could comment on this?

[Anon Anon]

So we had a very critical need on this since many of our 8.2.x clusters were setup in an active active mode. Now since we are upgrading the prod and realized this was not possible we got into a major predicament. We raised a ticket with CA on this and did not make much headway and so we did our own little "research" and identified the property that would allow MOM in active active mode :). We have run some extensive load and performance tests and so far we haven't encountered anything worse than single active mom.

[Legacy User]

hi Srikant

can you maybe let us know the setting :) we are starting as well looking in to 9.1 and this is one thing we realy would like to have too.

btw vote for the idea mom failover, i have added this thread as a potential solution in how to solve the OS unrelated failover :). (maybe leave a comment that you are as well looking for the solution with a active active mom.)

[Anon Anon]

Here you go add it to apm-events-thresholds-config.xml in both collectors and mom.


<clamp id="introscope.enterprisemanager.collector.mom.max">
<description>
Limits the number of MOM connections
</description>
<threshold value="3"/>
</clamp>


the reason the value is 3 is - once it reaches 2 active mom it puts a warning event that we have reached a max mom connection and might freak out ops.

[Legacy User]

thanks :) now CA should only tell us if its supported

[Anon Anon]

Though support guy did mention it is supported after I mentioned that we figured out the ppty but highly doubt he knew what he was saying based on my interaction with him

[Anon Anon]

aaronritter wrote:

thanks :) now CA should only tell us if its supported

Hi ,

Could you please help us to configure 2 MOM in 8.x version ... What are the requirements ? ...


Thanks
HK

[Anon Anon]

it should work out of box. just point them to the same set of collectors

[Legacy User]

yes, in theory you can just copy the mom you have to a second server and then only change the dns name to the new one e.g. from my-mom01.my.dns to my-mom02.my.dns and just start it. Anything else stays the same, just where you listen on will change.

which will create different challenges:

- how do you stay in sync with:
1) managment modules which include the alerting etc.
2) configs like domain.xml realms.xml etc.

- runtime issues:
1) if you have configured alerting e.g. sending events to mail reciver or for example a alert system like netcool or similar. you might need to disable it on one mom to avoid duplicated events.
2) to many requests on the collectors because two moms querry permanently stuff from the collectors (the lest active stuff like calculators etc you have setup the less load you generate) - the load of users likely stays the same unless you had already load issue on the first mom.

[Anon Anon]

yup, good points.

Yup sync is a huge issue with active active config unlike active/passive where everything is deployed in a shared folder - active active you have to consciously keep them sync'ed. You guys need to be aware of load on the collectors ( though user load was not an issue since dns was point to one active mom ) everything else will remain same. If you have too many such instances it will become a maintenance nightmare

[ANURANJAN KUMAR]

Hi All,

I was facing MOM-Collector disconnection issue, got helped from this thread.

added same to my collector & mom
<clamp id="introscope.enterprisemanager.collector.mom.max">
<description>
Limits the number of MOM connections
</description>
<threshold value="3"/>
</clamp>

Issue is fixed, collectors got connected with MOM.

We have 10.3 in our environment.


Regards,
AnuRanjan Kumar

------------------------------