AM V9.3.2 installation - java awapi no start

[Peter Wen]

Hi,

We are currently try to install AM V9.3.2 on REHL7 to replace old AM 9.2.2 server. We installed OpenJDK 11.0.7 according to compatibility matrix. We ran the install script successfully. However, when we try to start the server. It gave an error of Java awapi timed out. Then I checked log. I found the following error. Does anyone know how to solve it? Please help.

ErrorMsg: AwE-5102 Agent error (6/18/20, 9:57 PM)
Details: Socket[addr=/10.10.64.168,port=33435,localport=60010]
javax.net.ssl.SSLHandshakeException: No available authentication scheme
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:260)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(CertificateMessage.java:955)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(CertificateMessage.java:944)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1234)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1170)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:852)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:813)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:709)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:792)
at java.base/java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2802)
at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2818)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3315)
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:907)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350)
at com.appworx.shared.code.server.B.C(RequestSocket.java:124)
at com.appworx.server.data.AxRmiServer$3$1.run(AxRmiServer.java:711)

[Peter Wen]

I got answer from AppWorx Support.

[Don Weber]

Peter, we had a similar issue. What was the solution from AppWorx Support?

[Peter Wen]

Hi Don,

The solution for us is the keystore. it requires keystore files in $AW_HOME/data directory. Please check installation guide advance topic - 

Using Custom SSL Certificates for Connection Authentication for detail.

Regards,

Peter Wen

[Shashank Singh]

Hi Peter,

Is SSL certificate mandatory with v9.3 and above. We are planning an upgrade soon.

Regards,
Shashank Singh

[Peter Wen]

Hi Shashank,

Yes. It is mandatory for all servers and every users' computer that connect to server.


Regards,

Peter Wen

[Tania Ortega]

Hi Peter,

I took having the same issue below and did create the user_keystore and user_keystore_config files.
Any suggestions? I created user_keystore as this:
keytool -genkey -keyalg RSA -alias xxxxx -keypass xxxxx -storepass xxxxx -keystore user_keystore
then it was created in /opt/appworx/data.
Then I encrypted the password as this while in web/classes:
java -DAW_HOME=${AW_HOME} -cp AppWorx.jar:uc4-ra.jar com.appworx.util.EncryptKeystoreFile xxxx

My #1 issue is with the startup of appworx.
I'm having the "ErrorMsg: AwE-5103 network socket error " error in the AgentService file, yet also when I attempt start Appworx/AM it also has a java error: "Error: java awapi timed out."

I just upgraded from v8 to v9.3 of Appworx/AM in OCI Linux.

I have created "keytool -genkeypair -keyalg RSA" user_keystore and java web/classes created the EncryptKeystoreFile user_keystore_config in the /opt/appworx/data, then copied it over to my desktop into users\<username>\.Appowrx renaming user with my username.
I have installed jre1.8-1.8.0_251-fcs.x86_64.
I have removed the "anon, NULL" in jdk.tls.disabledAlgorithms from the java.security, as well as change the securerandom.source from /dev/random to /dev/urandom.
I have installed the rng-tools and add to the /etc/sysconfig/rngd file "'EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048" and chkconfig rngd on.
I have checked port 2136 and is in LISTEN as well as 6001 where appworx is LISTEN for java.
I have restarted both app and host.

But still I'm getting those errors above.
Any suggestions?

My #2 issue is when I attempt to connect remotely it fails with Connection refused, which I believe has to do with #1 above.​​​

[Peter Wen]

Hi Tania,

Can you please try the following? using apachectl start and startso all to start the master server. wait for 5 mins. Then issue awexe node command to see if all the agents is up or not (except remote agent). If not, turn the debug on and restart the services to see any error in the log.

ErrorMsg: AwE-5103 network socket error is the category error. Real reason is on the next few lines.

Regards,

Peter Wen

 

[Tania Ortega]

Hi Peter.

My sincere thanks for your help and your thoroughness on response as well.

http/apache is run as root, therefore, I verified that its running.

Then I did restart appworx with 'startso all' and that were we see that it hangs on this line:

Fri Jul 10 13:06:57 EDT 2020
agentservice.lock file is /opt/appworx/status/agentservice.lock
AgentService pid is 11459
waiting for java awapi to start
Error: java awapi timed out.
startso done

I waited the 5 minutes and ran "awexe node"

[appworx awappprod]: /home/appworx/scripts >awexe node
231 Error number from open pipe 2. /opt/appworx/pipe/AWAPI_APWXPRD_PIPE
334 Check that the API server is running.
Could not open server pipe.

Yet awstat is good:
[appworx awappprod]: /opt/appworx/site >awstat
Agent Program Status Sleep Last Act Pid
APWXPRD awcomm Running 0 7 minutes 11416
APWXPRD watchworx Offline 0 16 minutes 0

APWXPRD rmiserver Running 0 7 minutes 11422
APWXPRD agentservice Running 0 7 minutes 11459

Five+ minutes later the same error above from 'awexe node'.

This is just in the process of stating up and not yet attempting to communicte.

The error in the AgentService log file is this:
13:15:17.463 AgentService: .AgentService: using sun ssl
13:15:17.464 AgentService: AwE-5103
ErrorMsg: AwE-5103 network socket error (7/10/20 1:15 PM)
Details: 10.40.40.22:6001
network socket error : 10.40.40.22:6001 : java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

Yet the only changes done in java.security (did a diff from original) are:
1.  changed securerandom.source=file:/dev/random to securerandom.source=file:/dev/urandom
2.  removed "anon, NULL" for jdk.tls.disabledAlgorithms

and the java version is:
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)

[Peter Wen]

Hi Tania,

After agent service is up, the startso will use command awexe awapi_test to get the output as below

awapi_test: begin awapi_test
awapi_test: running "awexe awapi_test"
awexe_run: begin awexe_run
awexe_run: $AW_HOME/c/awexe awapi_test 1>/dev/null 2>&1 2>&1
awexe output=
:
awexe_run: awexe_ret no shift = 1024
awexe_run: awexe_ret with shift = 4
awapi_test: awapi failed to respond <- the startso will try 30 time(maybe) if not get positive output it will give Error: java awapi timed out.

Also, if you try awexe node and get the error, that means the pipe is not up. Therefore, the best thing to try is to debug the major services- awcomm, RMI, agent

Go to the $AW_HOME/site and adding Debug=ture in default section of awenv.ini

Then restart server by "stopso all; stopso awcomm" then "startso all"

Go to the $AW_HOME/log Wait for 5-10mins (using ls -ltr to check the errors write into log or not)

Then check all error log for awcomm, RmiServer, AgentService

especially for the connection between the port of RMI to Agent service

If no other error message beside you have right now, you probably have to do the port openness check or ssl poke.

Regards,

Peter Wen 

[Tania Ortega]

I Peter,

I just discovered that the local Oracle client is v11.0.2.4 when the remote primary DB is v19, therefore, it could be the ojdbc6.jar file it copied over to /opt/appworx/ojdbc folder during the AMv9.3 upgrade.

I'm waiting for the DB to remove the local v11x version and install the v19x client.

Regards,

Tania

[Tania Ortega]

Hi Peter,

Sincere thanks for all your feedback.

The primary issue was the local Oracle client version was v11 and the DBA had upgraded the database to v19.

All the errors pointed to a network error because the objdb6 couldn't connect.

The final straw to my challenge is that today we discovered a new wrench.

I now we need to start from scratch as we discovered that RA Oracle Retail v1.x+ is not supported in Applications Manager v9.x and we need the RA for our new OCI Oracle Retail v16. According to B Reseller it was back in February. 

I'm deep diving into the online docs and taking a fresh start.

Moving on to Automation Engine v12.3.3 w/RA OR v2.x on OL7.x. 

The Broadcom Community came through for me and I sincerely appreciated it!

Ciao.

[Richard_Blumlein_8490]

Hi everyone, 

That is a good point about the Oracle client. At one time we ran Appman from a 11.2.0.4 database, with an 11g client. 
Then later we upgraded the database to 12c, but still have the 11g client. It was working, and we could use the ojdbc6 jar.
Later I worked with the DBA to upgrade the client to 12c to match the database, and I started to use the ojdbc8.jar which showed 
compatible with 12.2 which is what we are at now. 

I am not logged on to the Oracle site but it makes me wonder what ojdbc driver you should use for 18 or 19 levels of the database.
The driver during installations asks you for the location and it copies it to the master ojdbc directory. I would have to look at my notes I don't think it copies it to the web/classes I might of done that manually. 

In the Banner agent manual. It mentions if the agent does database calls you need it in the remote agents web/classes. It does not use it 
out of the new ojdbc directory.   

Very good information everyone, and helping each other. 

Rich

[Mark Van Ditta]

That is not exactly true.  The problem is due to the "anon" and "NULL" cipher suites being disabled in the java.security file in newer Java releases. If one changes the entry:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
 EC keySize < 224, 3DES_EDE_CBC, anon, NULL

To:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 768, \
EC keySize < 224, 3DES_EDE_CBC

AppWorx will work with SSL turned on without having to install certificates.  This change has to be made on the servers and as well as the client work stations and it has to be made every time Java is upgraded.

[Richard_Blumlein_8490]

Hi everyone. 

That is very true about the anon, and NULL. We found this out the hard way. With Oracle Java 1.8 201 I believe and higher it added anon, and NULL. When we were at V9.1.1 of AM we had to remove those on the Linux, and PC clients. When we went to V9.3.1 we had the keystore files we could use. 

Very good information..

Thank you, 

Rich