AIX OS update broke Applications Manager 9.0.1

[Justin_Brown_7156]

Hi, we've got an old Appworx installation we use in conjunction with Banner for customer billing.  Recently (last Saturday) our UNIX team upgraded the AIX OS on our non-prod system and since then Appworx will not start.  The awcomm, RMI Server, and Agent processes launch but the Agent and the RMI master refuse to communicate.  With debug enabled the Agent logs the following: 

18:15:26.604 AgentService: .AwEnvOptions: using default: socketconnecttimeout = 30

18:15:26.604 AgentService: .AwEnvOptions: socketconnecttimeout = 30

18:15:26.604 AgentService: .AwEnvOptions: using default: responsetimeout = 60

18:15:26.604 AgentService: .AwEnvOptions: responsetimeout = 60

18:15:26.604 AgentService: .AgentService: getting awcomm rmi info

18:15:26.606 AgentService: .AwComm: lookup: C R:QUALSITE AgentService on our_qual_non_prod_host:2137

18:15:26.607 AgentService: .AwComm: connect to awcomm from: 0.0.0.0/0.0.0.0:0: -> our_qual_non_prod_host:2137

18:15:26.666 AgentService: .AwComm: awcomm connect true 60

18:15:26.666 AgentService: .AwComm: Get connect info for: C R:QUALSITE AgentService

18:15:26.666 AgentService: .AwComm: sending message to awcomm 23C R:QUALSITE AgentService*

18:15:26.672 AgentService: .AwComm: lookup 60010 Y 10.25.158.22

18:15:26.674 AgentService: .AgentService: createSocket: our_qual_non_prod_host:60010 From our_qual_non_prod_host/10.25.158.22:0 encrypt true

18:15:26.674 AgentService: .AgentService: using ibm ssl

18:15:27.466 AgentService: .AxRmiServer: SSL protocol: TLSv1

18:15:27.467 AgentService: .AgentSocketManager: addRmiSocket 7ee27ee2[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=our_qual_non_prod_host/10.25.158.22,port=60010,localport=51125]]

18:15:27.470 AgentService: .B: 7ee27ee2[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=our_qual_non_prod_host/10.25.158.22,port=60010,localport=51125]]

18:15:27.470 AgentService: .AgentSocketManager: sendHandshake 7ee27ee2[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=our_qual_non_prod_host/10.25.158.22,port=60010,localport=51125]]

18:15:27.493 AgentService: AwE-5103

18:15:27.495 AgentService: .AxOptions: NoErrorMsgProperties=false

ErrorMsg: AwE-5103 network socket error (9/18/20 6:15 PM)

Details: Network socket error

java.io.EOFException

at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2670)

at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3145)

at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:878)

at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)

at com.appworx.shared.code.server.B.C(RequestSocket.java:114)

at com.appworx.agent.AgentSocketManager.A(AgentSocketManager.java:167)

at com.appworx.agent.AgentSocketManager.A(AgentSocketManager.java:108)

at com.appworx.agent.AgentSocketManager.B(AgentSocketManager.java:79)

at com.appworx.agent.AgentService.A(AgentService.java:677)

at com.appworx.agent.AgentService.<init>(AgentService.java:362)

at com.appworx.agent.AgentService.main(AgentService.java:998)

The master logs a connection attempt but nothing that appears to be an error:

Applications Manager RMI Server

Copyright (C) 1999-2014 Automic Software Gmbh. All Rights Reserved.

Release: v9.0.1_28314_28331 Build: 28331 Tue Jun 30 08:30:55 PDT 2015

RA Framework 6.4.8+build.1 Build: 6735001 Thu Jun 25 09:59:31 PDT 2015

Java VM: IBM Corporation 1.6.0

Fri Sep 18 18:15:24 CDT 2020

18:15:24.730 main: .AwEnvOptions: using default: logdirectorypurgesize = 500

18:15:24.730 main: .AwEnvOptions: logdirectorypurgesize = 500

18:15:24.730 main: .AxRmiServer: purgeLogFiles: RmiServer.log 0 0.04462909698486328

18:15:24.733 main: .AxRmiServer: purgeLogFiles: RmiServer2009181815.log 0 0.04513263702392578

18:15:27.23 SM:accept: .AxRmiServer$3: accept connection 7ac07ac[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51125,localport=60010]]

Fri Sep 18 18:15:27 CDT 2020 accept connection 7ac07ac[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51125,localport=60010]]

18:15:27.273 SM:accept: .AxRmiServer: SSL protocol: TLSv1

18:15:27.275 SM:validate: .AxRmiServer$3$1: reading handshake 7ac07ac[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51125,localport=60010]]

18:15:27.277 SM:validate: .B: 7ac07ac[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51125,localport=60010]]

18:15:27.491 SM:validate: .AxRmiServer$3$1: handShake SeqNo null Agent QUALSITE Master null service handshake Method null [Applications Manager Agent Service

Copyright (C) 1999-2014 Automic Software Gmbh]

18:15:27.491 SM:validate: .AxRmiServer$3$1: no SocketManager null

18:15:37.531 SM:accept: .AxRmiServer$3: accept connection 23102310[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51130,localport=60010]]

Fri Sep 18 18:15:37 CDT 2020 accept connection 23102310[SSL_NULL_WITH_NULL_NULL: Socket[addr=/10.25.158.22,port=51130,localport=60010]]

18:15:37.669 SM:accept: .AxRmiServer: SSL protocol: TLSv1

18:15:37.670 SM:validate: .AxRmiServer$3$1: reading handshake 23102310[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51130,localport=60010]]

18:15:37.670 SM:validate: .B: 23102310[SSL_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=/10.25.158.22,port=51130,localport=60010]]

18:15:37.873 SM:validate: .AxRmiServer$3$1: handShake SeqNo null Agent QUALSITE Master null service handshake Method null [Applications Manager Agent Service

Copyright (C) 1999-2014 Automic Software Gmbh]
18:15:37.874 SM:validate: .AxRmiServer$3$1: no SocketManager null

This line in particular may be the issue but I'm not sure if it's abnormal as there doesn't appear to be any actual errors:
18:15:37.874 SM:validate: .AxRmiServer$3$1: no SocketManager null

But the fact that the Agent shows various SSL cipher details and etc where the master has a bunch of NULL NULL etc. has me concerned.  It looks like something is breaking with SSL negotiation.  Nothing within the Appworx configuration, databases, etc. changed pre or post OS update.  The only thing that changed was whatever the OS update touched and it shouldn't have gone into the Appworx application directory in any way.  Our UNIX guys and IBM are looking into what they might have broke with this update and I'm coming at the problem from the Appworx side of the equation.  Any ideas/suggestions?  Thanks.

[Justin_Brown_7156]

Not sure if bumping this is acceptable within this community but I'm still struggling with this issue. Anyone have any ideas?

[Justin_Brown_7156]

I should add as best I can tell SSL appears to be negotiating at least partially but appears to fail handshake eventually even though both sides, agent and RMI master, do appear to negotiate corresponding ciphers.

[Maday Avila]

Hi,

We also have the same issue. Were you able to fix this issue?

Please advise,

Maday Avila

[Richard_Blumlein_8490]

Hello, 

I see that you re using AM 9.0.1 it looks like. I was wondering what level of Java are you using?
If you are using Oracle Java 8 201 or higher you would hit the issue with the java.security file have anon, and NULL in the exclusions. 
You either have to remove those from the statement on the host, and client, or what we did. We had upgrade to V9.3.1 and started using the user_keystore files.  

https://www.oracle.com/java/technologies/javase/8u201-relnotes.html

I don't see you getting a Java awapi timeout. We had to change the java.security file for the /dev/random to /dev/urandom. 
This fixed that. 

Best of luck.  Be sure to check the Java compatibility levels to the Appman release. 

Rich

[Justin_Brown_7156]

Sorry, regarding this issue, it's still broken.