But not able to see anything on sdm logs. I am checking on stdlog* and not able to find trace related to UIM event
Original Message:
Sent: 12-06-2019 09:28 AM
From: Gene HOWARD
Subject: Incident not created for UIM alarms
the http error 500 is being returned by the service desk side.
Please check with your service desk manager to review the logs on that side to understand why the request is failing when sent.
------------------------------
Gene Howard
Principal Support Engineer
Broadcom
Original Message:
Sent: 12-06-2019 05:58 AM
From: MADANRAJ SIVAGNANAM
Subject: Incident not created for UIM alarms
Hi,
It was working fine still 04-Dec-19 and all of a sudden it stops working.. We haven't any changes in system as well..
"For Alarm Id: YC05097491-12723 incident creation failed in CASDM with error:{"Error":{"ErrorCode":154,"ErrorMessage":"Exception occurred during operation createIncident","DetailErrorMessage":"java.net.SocketTimeoutException: Read timed out","HTTPResponseCode":"INTERNAL_SERVER_ERROR","HTTPResponseCodeValue":500}}" this is the error we are getting
Original Message:
Sent: 12-05-2019 11:10 AM
From: Kathryn Maguire
Subject: Incident not created for UIM alarms
Madanraj -
From the Nim log file snippet posted, it looks like the sdgtw probe is making the request the create the incident in CASDM, but the request is timing out:
exception message 'java.net.SocketTimeoutException: Read timed out'
The probe made the API call to the CASDM service desk application and it has stopped responding so the probe could not create the indicent.
You need to check with your CASDM Service Desk administrato to review their logs around the times that the sdgtw probe is attempting to created an incident to see why they could not be created.
------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
Original Message:
Sent: 12-05-2019 12:32 AM
From: MADANRAJ SIVAGNANAM
Subject: Incident not created for UIM alarms
NIM API logs
exception message 'java.net.SocketTimeoutException: Read timed out'
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: Problem with 'createIncident'; Printing stack trace if available: null
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.ExceptionFactory.newNormalizationEx(ExceptionFactory.java:37)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.ExceptionFactory.mdrException(ExceptionFactory.java:300)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.GenericSOAPDispatchService.invokeWithRetry(GenericSOAPDispatchService.java:226)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.GenericSOAPDispatchService.doInvoke(GenericSOAPDispatchService.java:200)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.GenericSOAPDispatchService.invoke(GenericSOAPDispatchService.java:194)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMWebService.invoke(CASDMWebService.java:384)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMWebService.createRequest(CASDMWebService.java:1235)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMWebService.createIncident(CASDMWebService.java:1124)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMProvider.createIncident(CASDMProvider.java:870)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMProvider.createCI(CASDMProvider.java:804)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.resources.im.ServiceManagementFactory.createCI(ServiceManagementFactory.java:116)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.resources.im.ServiceManagementFactory.createCI(ServiceManagementFactory.java:185)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: com.ca.integration.normalization.resources.IncidentResource.createCI(IncidentResource.java:107)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: java.lang.reflect.Method.invoke(Method.java:498)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.Server.handle(Server.java:497)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[05/12/19 10:55:45:045 IST] [qtp1461149300-31] ERROR sdm.CASDMProvider: java.lang.Thread.run(Thread.java:745)
[05/12/19 10:56:08:008 IST] [qtp1461149300-23] ERROR common.GenericSOAPDispatchService: Type of exception thrown in Invoke Try: 'javax.xml.ws.WebServiceException': Unknown problem occurred with message: 'java.net.SocketTimeoutException: Read timed out'
[05/12/19 10:56:08:008 IST] [qtp1461149300-23] ERROR common.GenericSOAPDispatchService: javax.xml.ws.WebServiceException thrown in Invoke With Retry: Exception Message: 'java.net.SocketTimeoutException: Read timed out': attempting call #2
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR common.GenericSOAPDispatchService: Type of exception thrown in Invoke Try: 'javax.xml.ws.WebServiceException': Unknown problem occurred with message: 'java.net.SocketTimeoutException: Read timed out'
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR common.GenericSOAPDispatchService: WebserviceException thrown in Invoke With Retry: Exhausted all retries with exception message 'java.net.SocketTimeoutException: Read timed out'
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: Problem with 'createIncident'; Printing stack trace if available: null
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.ExceptionFactory.newNormalizationEx(ExceptionFactory.java:37)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.ExceptionFactory.mdrException(ExceptionFactory.java:300)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.GenericSOAPDispatchService.invokeWithRetry(GenericSOAPDispatchService.java:226)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.GenericSOAPDispatchService.doInvoke(GenericSOAPDispatchService.java:200)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.common.GenericSOAPDispatchService.invoke(GenericSOAPDispatchService.java:194)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMWebService.invoke(CASDMWebService.java:384)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMWebService.createRequest(CASDMWebService.java:1235)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMWebService.createIncident(CASDMWebService.java:1124)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMProvider.createIncident(CASDMProvider.java:870)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normalization.mdr.ca.sdm.CASDMProvider.createCI(CASDMProvider.java:804)
[05/12/19 10:56:32:032 IST] [qtp1461149300-23] ERROR sdm.CASDMProvider: com.ca.integration.normaliz
Original Message:
Sent: 12-04-2019 11:28 AM
From: Kathryn Maguire
Subject: Incident not created for UIM alarms
Madanraj -
The new code snippet you posted is not from the ALARM ASSIGN processing of the UIM alarm with nimid = SZ95876021-08248. Search your sdgtw.log file for the string "ALARM ASSIGN" then look for the nimid SZ95876021-08248 in the lines after it. You should then see the request to the NIM API to create the incident and the return status from this API. I suspect that the return status is not 200 which means that an error occurred.
The Nim.log files are found in the following directory:
<installPath>\Nimsoft\probes\gateway\sdgtw\NIMSM_WORKSPACE\webapp\WEB-INF\logs
However, if you have not modified the <installPath>\Nimsoft\probes\gateway\sdgtw\NIMSM_WORKSPACE\webapp\WEB-INF\config\log4j2.xml file and replaced all instances of the following 2 line:
level="ERROR"
level="INFO"
with the following:
level="ALL
then the Nim.log* files will not contain any details to determine the reason that the incident has not been created.
Your other option is to look at this from the CASDM side. The following Knowledge Document provides information on the UIM/CASDM integration through the sdgtw probe and contains some troubleshooting tips from the CASDM side as well:
KB122961 : Configure UIM for SDM integration
------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
Original Message:
Sent: 12-04-2019 10:44 AM
From: MADANRAJ SIVAGNANAM
Subject: Incident not created for UIM alarms
From where I can get these NIM API logs ?
Please provide the path to get these logs .
Original Message:
Sent: 12-04-2019 10:29 AM
From: Kathryn Maguire
Subject: Incident not created for UIM alarms
Madanraj -
We would need to see the sdgtw.log file contents which shows the ALARM ASSIGN processing for this alarm to determine why the incident was not created in the configured CASDM service desk application. It also may be necessary to see the enhanced NIM API logs captured at the same time for the exact reason for the issue. The sdgtw probe puts the IN PROGRESS$CASDM string in the custom_4 field of the alarm when it first receives the ALARM ASSIGN message for the alarm, then puts a request into the NIM API to create the incident in CASDM. When the ALARM UPDATE message is received, it looks at the contents of the custom_4 field to identify a corresponding incident in CASDM in case the incident needs to be updated. Since the incident was not created, the ALARM UPDATE message is not processed because there is no information in the custom_4 alarm field to identify an incident in CASDM.
Look back through the sdgtw logs to find the ALARM ASSIGN processing for the UIM alarm with nimid of SZ95876021-08248 to determine why the incident was not created in CASDM.
------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
Original Message:
Sent: 12-04-2019 10:16 AM
From: MADANRAJ SIVAGNANAM
Subject: Incident not created for UIM alarms
Processing an event from the queue 'tickets'
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] ALARM UPDATE
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] currentTicketDetails : IN PROGRESS$CASDM
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] old Severity: alarmId : incidentId ::::critical : SZ95876021-08248 : CASDM
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] new Severity: alarmId : incidentId ::::critical : SZ95876021-08248 : CASDM
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] old Message: alarmId : incidentId ::::BootAlarm: Computer has been rebooted at 2019.12.4 19:30:16 : SZ95876021-08248 : CASDM
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] new message: alarmId : incidentId ::::BootAlarm: Computer has been rebooted at 2019.12.4 19:30:16 : SZ95876021-08248 : CASDM
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] The Update Alarm Event will not be processed for alarm :SZ95876021-08248with the Ticket Details :IN PROGRESS$CASDM
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] subject..alarm_update
Dec 04 19:31:13:928 [attach_clientsession, sdgtw] data..PDS [ht={i18n_token=Element [buf=null, pds=null, tpds=null, type=7, value=as#system.cdm.comp_reboot], assigned_accept=Element [buf=null, pds=null, tpds=null, type=1, value=0], arrival=Element [buf=null, pds=null, tpds=null, type=1, value=1575468059], origin=Element [buf=null, pds=null, tpds=null, type=7, value=CHECAUIMHUB01_Pri_hub], custom_4=Element [buf=null, pds=null, tpds=null, type=7, value=IN PROGRESS$CASDM],