AppWorx, Dollar Universe and Sysload Community

Expand all | Collapse all

AIX OS update broke Applications Manager 9.0.1

  • 1.  AIX OS update broke Applications Manager 9.0.1

    Posted 09-19-2020 08:11 PM
    Hi, we've got an old Appworx installation we use in conjunction with Banner for customer billing.  Recently (last Saturday) our UNIX team upgraded the AIX OS on our non-prod system and since then Appworx will not start.  The awcomm, RMI Server, and Agent processes launch but the Agent and the RMI master refuse to communicate.  With debug enabled the Agent logs the following: 
    18:15:26.604 AgentService: .AwEnvOptions: using default: socketconnecttimeout = 30
    18:15:26.604 AgentService: .AwEnvOptions: socketconnecttimeout = 30
    18:15:26.604 AgentService: .AwEnvOptions: using default: responsetimeout = 60
    18:15:26.604 AgentService: .AwEnvOptions: responsetimeout = 60
    18:15:26.604 AgentService: .AgentService: getting awcomm rmi info
    18:15:26.606 AgentService: .AwComm: lookup: C R:QUALSITE AgentService on our_qual_non_prod_host:2137
    18:15:26.607 AgentService: .AwComm: connect to awcomm from: 0.0.0.0/0.0.0.0:0: -> our_qual_non_prod_host:2137
    18:15:26.666 AgentService: .AwComm: awcomm connect true 60
    18:15:26.666 AgentService: .AwComm: Get connect info for: C R:QUALSITE AgentService
    18:15:26.666 AgentService: .AwComm: sending message to awcomm 23C R:QUALSITE AgentService*
    18:15:26.672 AgentService: .AwComm: lookup 60010 Y 10.25.158.22
    18:15:26.674 AgentService: .AgentService: createSocket: our_qual_non_prod_host:60010 From our_qual_non_prod_host/10.25.158.22:0 encrypt true
    18:15:26.674 AgentService: .AgentService: using ibm ssl
    18:15:27.466 AgentService: .AxRmiServer: SSL protocol: TLSv1
    18:15:27.467 AgentService: .AgentSocketManager: addRmiSocket 7ee27ee2[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=our_qual_non_prod_host/10.25.158.22,port=60010,localport=51125]]
    18:15:27.470 AgentService: .B: 7ee27ee2[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=our_qual_non_prod_host/10.25.158.22,port=60010,localport=51125]]
    18:15:27.470 AgentService: .AgentSocketManager: sendHandshake 7ee27ee2[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=our_qual_non_prod_host/10.25.158.22,port=60010,localport=51125]]
    18:15:27.493 AgentService: AwE-5103
    18:15:27.495 AgentService: .AxOptions: NoErrorMsgProperties=false
    ErrorMsg: AwE-5103 network socket error (9/18/20 6:15 PM)
    Details: Network socket error
    java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2670)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3145)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:878)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at com.appworx.shared.code.server.B.C(RequestSocket.java:114)
    at com.appworx.agent.AgentSocketManager.A(AgentSocketManager.java:167)
    at com.appworx.agent.AgentSocketManager.A(AgentSocketManager.java:108)
    at com.appworx.agent.AgentSocketManager.B(AgentSocketManager.java:79)
    at com.appworx.agent.AgentService.A(AgentService.java:677)
    at com.appworx.agent.AgentService.<init>(AgentService.java:362)
    at com.appworx.agent.AgentService.main(AgentService.java:998)

    The master logs a connection attempt but nothing that appears to be an error:
    Applications Manager RMI Server
    Copyright (C) 1999-2014 Automic Software Gmbh. All Rights Reserved.
    Release: v9.0.1_28314_28331 Build: 28331 Tue Jun 30 08:30:55 PDT 2015
    RA Framework 6.4.8+build.1 Build: 6735001 Thu Jun 25 09:59:31 PDT 2015
    Java VM: IBM Corporation 1.6.0
    Fri Sep 18 18:15:24 CDT 2020
    18:15:24.730 main: .AwEnvOptions: using default: logdirectorypurgesize = 500
    18:15:24.730 main: .AwEnvOptions: logdirectorypurgesize = 500
    18:15:24.730 main: .AxRmiServer: purgeLogFiles: RmiServer.log 0 0.04462909698486328
    18:15:24.733 main: .AxRmiServer: purgeLogFiles: RmiServer2009181815.log 0 0.04513263702392578
    18:15:27.23 SM:accept: .AxRmiServer$3: accept connection 7ac07ac[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51125,localport=60010]]
    Fri Sep 18 18:15:27 CDT 2020 accept connection 7ac07ac[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51125,localport=60010]]
    18:15:27.273 SM:accept: .AxRmiServer: SSL protocol: TLSv1
    18:15:27.275 SM:validate: .AxRmiServer$3$1: reading handshake 7ac07ac[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51125,localport=60010]]
    18:15:27.277 SM:validate: .B: 7ac07ac[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51125,localport=60010]]
    18:15:27.491 SM:validate: .AxRmiServer$3$1: handShake SeqNo null Agent QUALSITE Master null service handshake Method null [Applications Manager Agent Service
    Copyright (C) 1999-2014 Automic Software Gmbh]
    18:15:27.491 SM:validate: .AxRmiServer$3$1: no SocketManager null
    18:15:37.531 SM:accept: .AxRmiServer$3: accept connection 23102310[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51130,localport=60010]]
    Fri Sep 18 18:15:37 CDT 2020 accept connection 23102310[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51130,localport=60010]]
    18:15:37.669 SM:accept: .AxRmiServer: SSL protocol: TLSv1
    18:15:37.670 SM:validate: .AxRmiServer$3$1: reading handshake 23102310[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51130,localport=60010]]
    18:15:37.670 SM:validate: .B: 23102310[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51130,localport=60010]]
    18:15:37.873 SM:validate: .AxRmiServer$3$1: handShake SeqNo null Agent QUALSITE Master null service handshake Method null [Applications Manager Agent Service
    Copyright (C) 1999-2014 Automic Software Gmbh]
    18:15:37.874 SM:validate: .AxRmiServer$3$1: no SocketManager null

    This line in particular may be the issue but I'm not sure if it's abnormal as there doesn't appear to be any actual errors:
    18:15:37.874 SM:validate: .AxRmiServer$3$1: no SocketManager null

    But the fact that the Agent shows various SSL cipher details and etc where the master has a bunch of NULL NULL etc. has me concerned.  It looks like something is breaking with SSL negotiation.  Nothing within the Appworx configuration, databases, etc. changed pre or post OS update.  The only thing that changed was whatever the OS update touched and it shouldn't have gone into the Appworx application directory in any way.  Our UNIX guys and IBM are looking into what they might have broke with this update and I'm coming at the problem from the Appworx side of the equation.  Any ideas/suggestions?  Thanks.




  • 2.  RE: AIX OS update broke Applications Manager 9.0.1

    Posted 09-28-2020 12:30 PM
    Not sure if bumping this is acceptable within this community but I'm still struggling with this issue. Anyone have any ideas?


  • 3.  RE: AIX OS update broke Applications Manager 9.0.1

    Posted 09-28-2020 12:31 PM
    I should add as best I can tell SSL appears to be negotiating at least partially but appears to fail handshake eventually even though both sides, agent and RMI master, do appear to negotiate corresponding ciphers.