10.1 Workstation disconnect

[David Richards]

Hi, I have upgradeded to 10.1 but seem to be having trouble with the workstation login to MOM.

Login to collectors is fine, but MOM connection drops almost instantly.

Webview is working OK.

 

MOM is AIX.

 

I saw this KB article and I have chcked the osgi file version, it looks OK to me.

 

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec612907.aspx

 

EM log

 

20151204 191933 GMT [VERBOSE] [Manager.PostOfficeHub] Connected To: Node=Workstation_6, Address=50.11.148.169/50.11.148.169:60901, Type=socket

20151204 191934 GMT [INFO] [Manager.SessionBean] Workstation User "Admin" connected successfully from host "Node=Workstation_6, Address=50.11.148.169/50.11.148.169:60901, Type=socket"

20151204 191936 GMT [VERBOSE] [Manager.EventHandler] Error reading incoming bytes in Server Hub Receive 12 event handler for Socket Transport connected with Socket[addr=50.11.148.169/50.11.148.169,port=60901,localport=25318] because java.io.IOException: An existing connection was forcibly closed by the remote host

20151204 191936 GMT [VERBOSE] [Manager.PostOfficeHub] Disconnected From: Node=Workstation_6, Address=50.11.148.169/50.11.148.169:60901, Type=socket

20151204 191936 GMT [INFO] [Manager.SessionBean] Workstation User "Admin" disconnected successfully from host "Node=Workstation_6, Address=50.11.148.169/50.11.148.169:60901, Type=socket"

 

 

Workstation log

 

12/04/15 07:13:25 PM GMT [ERROR] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [WARN] [Workstation.IsengardObjectInputStream] Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.IncomingMessageDeliveryTask] Caught exception while deserializing object: java.lang.ClassNotFoundException: Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

java.lang.ClassNotFoundException: Object of type "com.wily.rave.reports.ChangeTableElement" cannot be deserialized from loader:

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readClass(IsengardObjectInputStream.java:599)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:703)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:347)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readArrayList(IsengardObjectInputStream.java:872)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readSerializable(IsengardObjectInputStream.java:997)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:368)

    at com.wily.reporting.common.SmartReportsTemplate.readExternal(SmartReportsTemplate.java:225)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:825)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:347)

    at com.wily.reporting.descriptor.ESESmartReportTemplateDescriptor.readExternal(ESESmartReportTemplateDescriptor.java:256)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:825)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:347)

    at com.wily.introscope.spec.server.beans.WireConstruct.readExternal(WireConstruct.java:78)

    at com.wily.introscope.spec.server.beans.DomainAwareTrackedConstruct.readExternal(DomainAwareTrackedConstruct.java:40)

    at com.wily.introscope.spec.server.beans.BundledTrackedConstruct.readExternal(BundledTrackedConstruct.java:43)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:825)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:347)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readArrayList(IsengardObjectInputStream.java:872)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readSerializable(IsengardObjectInputStream.java:997)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:368)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:389)

    at com.wily.isengard.messageprimitives.MethodCallMessage.readExternal(MethodCallMessage.java:133)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:825)

    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:347)

    at com.wily.isengard.postofficehub.link.v1.IncomingMessageDeliveryTask.deliverNextMessage(IncomingMessageDeliveryTask.java:80)

    at com.wily.isengard.postofficehub.link.v1.IncomingRouteConnector.receiveIncomingMessages(IncomingRouteConnector.java:170)

    at com.wily.isengard.postofficehub.link.v1.IncomingRouteConnector.doTask(IncomingRouteConnector.java:89)

    at com.wily.isengard.util.thread.AThreadedExecutable.run(AThreadedExecutable.java:192)

    at java.lang.Thread.run(Thread.java:722)

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.IncomingMessageDeliveryTask] Unrecoverable exception while reading from Socket Transport connected with Socket[addr=/22.114.0.27,port=25318,localport=59365]

java.lang.ClassCastException: com.wily.isengard.messageprimitives.MethodCallMessage cannot be cast to com.wily.isengard.postoffice.Address

    at com.wily.isengard.postofficehub.link.v1.IncomingMessageDeliveryTask.deliverNextMessage(IncomingMessageDeliveryTask.java:76)

    at com.wily.isengard.postofficehub.link.v1.IncomingRouteConnector.receiveIncomingMessages(IncomingRouteConnector.java:170)

    at com.wily.isengard.postofficehub.link.v1.IncomingRouteConnector.doTask(IncomingRouteConnector.java:89)

    at com.wily.isengard.util.thread.AThreadedExecutable.run(AThreadedExecutable.java:192)

    at java.lang.Thread.run(Thread.java:722)

12/04/15 07:13:25 PM GMT [VERBOSE] [Workstation.PostOfficeHub] Disconnected From: Node=Server, Address=/22.114.0.27:25318, Type=socket

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOfficeHub] Notifying transport Loopback PO: Workstation_5.client_main that route Post Office: Server.WatchedAgentPO, 1 hops via Socket Transport connected with Socket[addr=/22.114.0.27,port=25318,localport=59365] is down

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.IsengardClient] Lost connection to server tier

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOffice] Closing PostOffice.

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOffice] Closing PostOffice.

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOffice] PostOffice successfully closed.

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOfficeHub] Notifying transport Loopback PO: Workstation_5.client_main that route Post Office: Workstation_0.client_main, 2 hops via Socket Transport connected with Socket[addr=/22.114.0.27,port=25318,localport=59365] is down

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOfficeHub] Notifying transport Loopback PO: Workstation_5.client_main that route Post Office: Server.AppMapAgentPO, 1 hops via Socket Transport connected with Socket[addr=/22.114.0.27,port=25318,localport=59365] is down

12/04/15 07:13:25 PM GMT [DEBUG] [Workstation.PostOfficeHub] Notifying transport Loopback PO: Workstation_5.client_main that route Post Office: Server.main, 1 hops via Socket Transport connected with Socket[addr=/22.114.0.27,port=25318,localport=59365] is down

12/04/15 07:13:33 PM GMT [DEBUG] [Workstation] Checking IntervalHeartbeats.

[Lynn Williams]

Hi Dave,

As connectivity to the Collectors is successful it sounds like you have correct osgi version & also a workstation reinstall likely won't help.

I had a look at your full workstation.log on the case you raised (00262128) and it seems the workstation is processing data before the problem occurs.

Have you tried to restart the MOM to see if the problem persists?

Thanks

Lynn

[David Richards]

Hi Lynn - I was away yesterday so its a delayed response

 

I have just restarted the MOM and the Webview server, but the workstation issue still persists

 

20151208 091950 GMT [VERBOSE] [Manager.PostOfficeHub] Connected To: Node=Workstation_1, Address=22.188.26.197/22.188.26.197:49352, Type=socket

20151208 091952 GMT [INFO] [Manager.SessionBean] Workstation User "Admin" connected successfully from host "Node=Workstation_1, Address=22.188.26.197/22.188.26.197:49352, Type=socket"

20151208 091957 GMT [VERBOSE] [Manager.EventHandler] Error reading incoming bytes in Server Hub Receive 7 event handler for Socket Transport connected with Socket[addr=22.188.26.197/22.188.26.197,port=49352,localport=25318] because java.io.IOException: Connection reset by peer

20151208 091957 GMT [VERBOSE] [Manager.PostOfficeHub] Disconnected From: Node=Workstation_1, Address=22.188.26.197/22.188.26.197:49352, Type=socket

20151208 091957 GMT [INFO] [Manager.SessionBean] Workstation User "Admin" disconnected successfully from host "Node=Workstation_1, Address=22.188.26.197/22.188.26.197:49352, Type=socket"

[Anon Anon]

Hi Dave,

Can you check if on the MOM/ws-plugins folders you have change detector jars present (com.wily.cd.*.jar) and if they are correct version i.e 10.1 ?

If not then that likely might be causing the issue, also check for SOA dependency map extension sometimes if SOA map is corrupted then workstation won't launch as well.

Regards,

Kulbir.

[David Richards]

Kulbir

 

On the MOM, the ws-plugins directory is completely empty. No files are present

In my backed up 9.5.2 installation there are files there.

 

Also, in the 10.1 collector there are files present

 

/ws-plugins > ls -ltr

total 6968

-rwxrwxr-x    1 sxepadm  sxepadm       87537 03 Oct 06:08 com.wily.introscope.ui.tomsawyer_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm        6513 03 Oct 06:08 com.wily.introscope.soa.tracefilters.workstation.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       47842 03 Oct 06:08 com.wily.introscope.soa.tracefilters.workstation_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm        6914 03 Oct 06:08 com.wily.introscope.soa.tracefilters.common_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm        8189 03 Oct 06:08 com.wily.introscope.soa.dependencymetrics.typeviewer.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       39028 03 Oct 06:08 com.wily.introscope.soa.dependencymetrics.typeviewer_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       21482 03 Oct 06:08 com.wily.introscope.soa.dependencymap.ui.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm      251820 03 Oct 06:08 com.wily.introscope.soa.dependencymap.ui_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm        3593 03 Oct 06:08 com.wily.introscope.soa.dependencymap.common.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       37966 03 Oct 06:08 com.wily.introscope.soa.dependencymap.common_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm        4317 03 Oct 06:08 com.wily.introscope.soa.crossprocessviewer.workstation.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       70806 03 Oct 06:08 com.wily.introscope.soa.crossprocessviewer.workstation_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       32458 03 Oct 06:09 com.wily.cd.common.workstation_server_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm      102459 03 Oct 06:09 com.wily.cd.common.all_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       53264 03 Oct 06:09 com.wily.ui.guiffy.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm     1348639 03 Oct 06:09 com.wily.ui.guiffy_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm       19716 03 Oct 06:09 com.wily.rave.workstation.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm        3108 03 Oct 06:09 com.wily.rave.common.workstation_server.nl1_10.1.0.jar

-rwxrwxr-x    1 sxepadm  sxepadm     1383802 03 Oct 06:09 com.wily.cd.workstation_10.1.0.jar

[David Richards]

Hi Kulbir - you were pointing me in the right direction, thanks for that.

 

We decided to restore the 9.5.2 build and re-install the 10.1.0.15 upgrade, as the initial attempt had hit f/s space issues and I think the installation was compromised somehow.

 

This time round the upgrade has gone in with no issues, and the workstation is now available.

 

Thanks for your help, it definitely helped us resolve this today.