USM error for Firehunter

[Manish Arora]

Hi

Getting the below error while trying to open UMP after upgrade, firehunter dataexception

[David Michel]

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=100785
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=103902

------------------------------
Support Engineer
Broadcom
------------------------------

[Manish Arora]

Did not worked this is after the fail over to DC 2, complete error

An unknown error has occurred.
Refreshing your browser may resolve the issue.

Details:
com.firehunter.ump.exceptions.DataFactoryException : Received status (120) on response (for sendRcv) for cmd = 'get_next_fire_time_for_schedules'
Please check the log for more information.
Stack Trace:
(120) Callback error, Received status (120) on response (for sendRcv) for cmd = 'get_next_fire_time_for_schedules'
at com.nimsoft.nimbus.NimSessionBase.sendRcv(NimSessionBase.java:615)
at com.nimsoft.nimbus.NimSessionBase.sendRcv(NimSessionBase.java:561)
at com.nimsoft.nimbus.NimClientSession.send(NimClientSession.java:170)
at com.nimsoft.nimbus.NimRequest.sendImpersonate(NimRequest.java:263)
at com.nimsoft.nimbus.pool.NimRequestPool.sendImpersonate(NimRequestPool.java:92)
at com.nimsoft.nimbus.pool.NimRequestPool.sendImpersonate(NimRequestPool.java:74)
at com.nimsoft.nimbus.pool.NimRequestPool.send(NimRequestPool.java:66)
at com.nimsoft.nimbus.pool.NimRequestPoolInstance.send(NimRequestPoolInstance.java:191)
at com.nimsoft.nimbus.pool.NimRequestPoolInstance.send(NimRequestPoolInstance.java:150)
at com.firehunter.umpportlet.PDSUtils.send(PDSUtils.java:55)
at com.firehunter.usm.Maintenance.getNextFireTimeFromPDS(Maintenance.java:93)
at com.firehunter.usm.Maintenance.getNextFireTimes(Maintenance.java:76)
at com.firehunter.usm.Maintenance.getSchedules(Maintenance.java:158)
at com.firehunter.usm.Maintenance.getMaintenanceSchedules(Maintenance.java:183)
at com.firehunter.usm.DataFactory.getMaintenanceSchedules(DataFactory.java:8250)
at com.firehunter.usm.DataFactory.getMaintenanceSchedules(DataFactory.java:8241)
at sun.reflect.GeneratedMethodAccessor705.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at flex.messaging.services.remoting.adapters.JavaAdapter.invoke(JavaAdapter.java:418)
at flex.messaging.services.RemotingService.serviceMessage(RemotingService.java:183)
at flex.messaging.MessageBroker.routeMessageToService(MessageBroker.java:1400)
at flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:1005)
at flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:103)
at flex.messaging.endpoints.amf.LegacyFilter.invoke(LegacyFilter.java:158)
at flex.messaging.endpoints.amf.SessionFilter.invoke(SessionFilter.java:44)
at flex.messaging.endpoints.amf.BatchProcessFilter.invoke(BatchProcessFilter.java:67)
at flex.messaging.endpoints.amf.SerializationFilter.invoke(SerializationFilter.java:166)
at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:291)
at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:353)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.firehunter.ump.auth.InvalidHttpSessionFilter.doFilter(InvalidHttpSessionFilter.java:29)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:73)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:117)
at sun.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy1231.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:73)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:168)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

[Alex Yasuda]

you have this message in the log:
get_next_fire_time_for_schedules
So please check this document:
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=141807



------------------------------
Technical Support Engineer
Broadcom
------------------------------

[David Michel]

Take note the error string is now different. 

General troubleshooting info:
'cmd = ...' is the important part. 
cmd = 'dispatcher' is the most common, and in that case the next line is typically the root of the problem. 
for example:
cmd = 'dispatcher': (120) Callback error, Received status (120) on response (for sendRcv) for cmd = 'dispatcher'
at com.nimsoft.events.client.EmsClient.handleServiceException(EmsClient.java:778)

This indicates a failure to communicate with ems and a restart of ems typically resolves that. 

In your second case it changed to cmd = 'get_next_fire_time_for_schedules'
This provides a unique string get_next_fire_time_for_schedules to search on, and get the hit that Alex provided. 
Unique strings are most always more helpful to find what's needed. 
get_next_fire_time_for_schedules 'schedules' in that provides the hint this is related to maintenance schedules which the KB confirms. 

For some reason basic items were left out from your description, such as UIM revision and that becomes important since:
http://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/ca-unified-infrastructure-management-probes/GA/alphabetical-probe-articles/maintenance-mode-maintenance-mode/maintenance-mode-release-notes.html
(8.53 HF2) Added the MAINTENANCE_WINDOW purge task, where the MAINTENANCE_WINDOW entries are purged when the corresponding MAINTENANCE_SCHEDULE is deleted.
so prior to maintenance_mode 8.53_hf2 there was no auto cleanup. If users just kept creating schedules without deleting their old ones those tables just built up. 

So based on the KB your problem is now related to an old maintenance schedule, and since you can't get into USM to check and delete the old schedules it will probably be necessary to go direct to the db. 
These are the tables:
MAINTENANCE_SCHEDULE
MAINTENANCE_SCHEDULE_MEMBERS
MAINTENANCE_WINDOW
Recommendation is to work with your db admin to have him clean out all the old unneeded schedules. 
 




------------------------------
Support Engineer
Broadcom
------------------------------