Agent Validation Error

[Henson Sturgill]

Can someone help me figure out why I keep getting an error on my Master that says "Agent Validation Error" whenever I try to run startso on my agent?

Here's the error I'm getting on my agent:

[apwxwcu@paws31 appworx]$ startso all

starting agentservice

.

startup complete for agentservice

scanning log file

Mon Sep 07 14:41:38 EDT 2020 AgentService starting

Debug enabled at 9/7/20 2:41 PM

main: .AwEnvOptions: using default: serviceworkercount = 7

main: .AwEnvOptions: serviceworkercount = 7

main: .AwEnvOptions: using default: masterserverportstart = 60010

main: .AwEnvOptions: masterserverportstart = 60010

main: .AxOptions: sslDebug=null

main: .AwEnvOptions: using default: ssldebug = null

main: .AwEnvOptions: ssldebug = null

main: .SocketManager: jdk.tls.ephemeralDHKeySize: 2048

main: .AwEnvOptions: using default: password_encryption = AES

main: .AwEnvOptions: password_encryption = AES

main: .AwEnvOptions: using default: encryption_bits = 256

main: .AwEnvOptions: encryption_bits = 256

Initializing JCE: AES 256

main: encrypt 8

main: decrypt 0

Data decrypted Initialized JCE

JCE Initialized successfully 166

main: .AwEnvOptions: using default: serverlogmode = null

main: .AwEnvOptions: serverlogmode = null

main: .AwEnvOptions: using default: logfilezipsize = 50

main: .AwEnvOptions: logfilezipsize = 50

main: .DBG: Debug mode: null

main: .AwEnvOptions: using default: logfilezipsize = 50

main: .AwEnvOptions: logfilezipsize = 50

Applications Manager Agent Service

Copyright (C) 1999-2014 Automic Software Gmbh. All Rights Reserved.

Release: v9.3.0_28919_29103 Build: 29103 Tue Mar 12 14:25:12 PDT 2019

6.5.0+build.1 2018-08-31 18:09

Java VM: Java(TM) SE Runtime Environment 1.8.0_172

Mon Sep 07 14:41:38 EDT 2020

agentservice.lock file is /wcu/output/appworx/status/agentservice.lock

AgentService pid is 13687

waiting for java awapi to start

Error: java awapi timed out.

Here's my Agent's awenv.ini:

[default]

Debug=true

radebuglevel=99

AWCOMM_PORT=2136

Master=UC4PROD

Master_Ip_Address=152.4.164.42

Agent=UC4BANPRD

OsType=UNIX

old_agent=UC4BANPRD

[Install]

FireWall=No

AWWEB_HOME=/wcu/output/appworx

remove_links=No

And here's what the Master's awenv.ini looks like:

[default]

aw_ojdbc=C:\oracle\product\12.2.0\client_1\jdbc\lib\ojdbc8.jar

agent=UC4PROD

masterserverport=6103

db_service=WCUAPWP_PRMY.wcu.edu

DirectDBConnection=Y

awcomm_port=2136

tz=America/New_York

db_ip=wcuapwp-mcnc.uncecs.edu

agntmgr=N

masteragentencryption=No

OsType=WINDOWS

master=UC4PROD

rmi=Y

AW_HOME=C:\UC4

old_agent=UC4PROD

master_ip_address=APPWORXPROD.AD.UNCECS.EDU

null=

agentclientport=6104

db_port=15215

[install]

virtual_day=00:00

interactive=Y

lib_home=

smtp_host=smtp.wcu.edu

net_connect=@wcuapwp-mcnc.uncecs.edu:15215/WCUAPWP_PRMY.wcu.edu

update_repository=Y

apache_port=5050

PassWord=

apache_installed=Y

db_id=uc4

service_auto=N

rmi_only=N

saveinstall=Y

service_user=wcu\Appwrx

install_master=Y

apache_home=C:\Apache\Apache2

awweb_home=C:\UC4

autoanswer=N

smtp_email=[e-mail]

keep_password=N

smtp_port=25

firewall=Y

extract_files=N

db_sid=wcuapwp-mcnc.uncecs.edu:15215/WCUAPWP_PRMY.wcu.edu

[options]

RMIRegistryPortNumber=54321

RMIHostID=appworxprod.ad.uncecs.edu

Debug=Y

RMIServerHostname=

eventlog=N

ClientRMIPortNumber=6102

SSL=N

RMIDataPortNumber=6101

[parms]

sqlload=

sqlplus=

[Richard_Blumlein_8490]

Hi Henson, The only suggestion I can make is about the Java awapi timeout. Verify the Java being used to start the agent, and remote agents if any. Then drill down to the java.security file.  Change /dev/random  to /dev/urandom    This fixed our timeout issues.  I did this to my master, and remote agents. When you upgrade Java don't forget to change it again. We used Ansible to change it as part of a Java upgrade. 
Also depending on the Appman release you need to have the user_keystore files in place. When we upgraded from V9.1.1 to V9.3.1 this needed to be in place or the agents would not start. 

Good luck,

Rich

[CARLOS JUPIA]

Hi All

In parallel I also opened a call at Broadcom and the answer is that it is not possible to upgrading directly from 10 to 12.3,
and the strategy is upgrade first to 11.2, the best would be to take directly the latest package issued in this release, it was the package 11.2.10 and in a next step you can update to 12.3.x of your choice in a single step.

Thx

[Richard_Blumlein_8490]

Hi everyone, 

I have a question on this. When you mentioned 10 -  12.3 are you trying to upgrade from Appworx/Applications Manager 10 to the newer AE/Automation Engine/new names I forgot release?

I thought if you are on a AM release you cannot migrate from AM to AE>  I heard you need to engage the global services for this which I  don't get.  Seems hard to keep the product if you can't migrate.. 


Thank you, 

Rich

[Richard_Blumlein_8490]

Hi Henson, 
I have also had luck by removing the <master>/site files when it is down.  By clearing the files, it starts fresh. I also like to shutdown, and clear the log files with debug on. Then start, gather the logs, and open a ticket with support. 

Rich

[Richard_Blumlein_8490]

CORRECTION not site, status...    Geez, my brain is going..

Don't remove fles from site. 

Rich

[Henson Sturgill]

So, you were right about the JAVA. Something changed on our hosting provider to where $JAVA_HOME was pointing at Oracle 1.7 instead of OpenJDK 1.8.

I also found in our ~/.bash_profile where the sourcing of sosite had been commented out:

# . site/sosite

Still trying to find out who made the changes, but of course no one's owning up ;-)

Thanks for all your help
​ ​