CA Workload Automation Agent fails to start

[Anon Anon]

Hi, I reached this community seeking information to troubleshoot an issue that's hitting several AIX servers in our customer's environment. The installed product is "CA Workload Automation Agent for AIX R11.3, Build 353", and the issue is the agent fails to start after a reboot. It gets started from inittab via the following entry:

wa:2:once:/wa/wa/cybAgent &

running under root user.

We were told the agent fails to start due to it being unable to find the environment variable JAVA_HOME, but the application logs show it does...

I'm a newbie to this software, so if more information is needed to troubleshoot the issue, please let me know and I will gladly post it here.

Thanks in advance!

Luis.

[Anon Anon]

what's inside /wa/wa/cybAgent ?

[Anon Anon]

You mean what's inside /wa/wa, right? cybAgent is the script which starts CyberAgent:

[font=Courier New]root@myserver:/wa/wa> cat cybAgent
#!/bin/sh
#
# cybAgent for aix-ppc-32
#
LIBPATH="/wa/wa:/usr/java6/jre/lib/ppc/classic:/usr/java6/jre/lib/ppc:$LIBPATH"
export LIBPATH

#export IBM_JAVA_MMAP_JAVA_HEAP=true
#export LDR_CNTRL=MAXDATA=0x80000000

cd '/wa/wa'
exec ./cybAgent.bin "$@"

root@myserver:/wa/wa> ls -la
total 9904
drwxrwxr-x 8 root system 4096 Dec 11 10:05 .
drwxr-xr-x 4 root system 256 Nov 4 12:23 ..
-rw-rw-r-- 1 root system 16626 Nov 4 12:55 CA_Workload_Automation_Agent_R11.3_InstallLog.log
-rw-rw-r-- 1 root system 15063 Nov 4 12:55 CA_Workload_Automation_System_Agent_R11.3_Install_IA.log
-rwxr-xr-x 1 root system 46519 Nov 4 12:55 ESPlmi
-rwxrwxr-x 1 root system 3256 Nov 4 12:55 ESPlmi.lax
-rwxr-xr-x 1 root system 110 Nov 4 12:55 ESPmgr
-rwxrwxr-x 1 root system 362268 Jul 15 2010 ESPmgr.bin
lrwxrwxrwx 1 root system 13 Nov 4 12:55 Esplmi -> /wa/wa/ESPlmi
drwxr-xr-x 2 root system 256 Nov 4 12:55 Legal
-rwxr-xr-x 1 root system 46519 Nov 4 12:55 PluginInstaller
-rwxrwxr-x 1 root system 3216 Nov 4 12:55 PluginInstaller.lax
drwxr-xr-x 2 root system 4096 Nov 4 12:55 UninstallData
-rw-r--r-- 1 root system 2284 Dec 9 10:10 agentparm.txt
-rw-r--r-- 1 root system 2284 Nov 13 09:43 agentparm.txt.manager.Sun_Dec_09_08_45_14_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 08:45 agentparm.txt.manager.Sun_Dec_09_08_49_36_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 08:49 agentparm.txt.manager.Sun_Dec_09_09_04_52_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 09:04 agentparm.txt.manager.Sun_Dec_09_09_05_46_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 09:05 agentparm.txt.manager.Sun_Dec_09_09_45_36_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 09:45 agentparm.txt.manager.Sun_Dec_09_09_46_43_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 09:46 agentparm.txt.manager.Sun_Dec_09_10_05_45_CST_2012.txt
-rw-r--r-- 1 root system 2284 Dec 9 10:05 agentparm.txt.manager.Sun_Dec_09_10_10_28_CST_2012.txt
-rw-rw-r-- 1 root system 2225 Nov 4 12:55 agentparm.txt.manager.Tue_Nov_13_09_43_00_CST_2012.txt
-rwxrwxr-x 1 root system 59460 Nov 4 12:55 cacerts
-rwxrwxr-x 1 root system 85963 Jul 15 2010 chkusr
-rw-rw-r-- 1 root system 944 Nov 4 12:55 cryptkey.txt
-rwxr-xr-x 1 root system 248 Nov 4 12:55 cybAgent
-rwxrwxr-x 1 root system 1680229 Nov 13 09:17 cybAgent.bin
-rwxrwxr-x 1 root system 1565809 Jul 15 2010 cybAgent.bin.20121113
-rw-rw-r-- 1 root system 5140 Nov 4 12:55 cybagent-axappp01_agent.init
-rwxrwxr-x 1 root system 10207 Jul 15 2010 cybermation.mib
-rwxr-xr-x 1 root system 115 Nov 4 12:55 cybspawn
-rwxrwxr-x 1 root system 735156 Jul 15 2010 cybspawn.bin
drwxr-xr-x 2 root system 4096 Dec 12 10:00 database
-rwxr-xr-x 1 root system 205 Nov 4 12:55 ftpusrcfg
drwxr-xr-x 3 root system 4096 Nov 4 12:55 jars
lrwxrwxrwx 1 root system 15 Nov 4 12:55 jre -> /usr/java6/jre/
-rwxr-xr-x 1 root system 184 Nov 4 12:55 keygen
-rw-rw-r-- 1 root system 53795 Nov 4 12:55 lax.jar
-rwxrwxr-x 1 root system 14635 Jul 15 2010 libfilefilter.so
-rwxrwxr-x 1 root system 11188 Jul 15 2010 libftpsecurity.so
-rwxrwxr-x 1 root system 20124 Jul 15 2010 libobjmon.so
-rwxrwxr-x 1 root system 190867 Jul 15 2010 libplugininvoker.so
drwxrwxr-x 2 root system 4096 Dec 12 10:15 log
-rw------- 1 root system 0 Dec 11 10:05 nohup.out
-rw-r--r-- 1 root system 0 Dec 11 10:05 nohup.stderr
-rw-r--r-- 1 root system 0 Dec 11 10:05 nohup.stdout
-rwxr-xr-x 1 root system 187 Nov 4 12:55 password
-rw-r--r-- 1 root system 55 Dec 11 10:05 security.turbo
-rw-rw-r-- 1 root system 30 Nov 4 12:55 security.txt
-rw-rw-r-- 1 root system 1249 Nov 4 12:55 serverkeystore
drwxrwxrwt 3 root system 256 Nov 14 10:44 spool
-rw-r--r-- 1 root system 116 Dec 11 10:05 status
root@myserver:/wa/wa>[font]

[Anon Anon]

i'd change the inittab entry like this:

wa:2:once:/wa/wa/cybAgent > /tmp/cybAgent.log 2>&1

then analyze the log file.

[Anon Anon]

/wa/wa/cybAgent is just a wrapper which sets libpath and other stuff, so it logs nothing when called that way from inittab. The best I got was from running it through a shell:

[font=Courier New]root@myserver:/> sh -x /wa/wa/cybAgent
+ LIBPATH=/wa/wa:/usr/java6/jre/lib/ppc/classic:/usr/java6/jre/lib/ppc:
+ export LIBPATH
+ cd /opt/cyb
+ exec ./cybAgent.bin[font]

[Anon Anon]

well, having an empty log file would prove the the wrapper was executed. I think it is a valuable piece of information.

slimshady wrote:

/wa/wa/cybAgent is just a wrapper which sets libpath and other stuff, so it logs nothing when called that way from inittab. The best I got was from running it through a shell:

[font=Courier New]root@myserver:/> sh -x /wa/wa/cybAgent
+ LIBPATH=/wa/wa:/usr/java6/jre/lib/ppc/classic:/usr/java6/jre/lib/ppc:
+ export LIBPATH
+ cd /opt/cyb
+ exec ./cybAgent.bin[font]

[Ravi Kiran Kunduri]

Hi,

Can you please confirm if you give the "command : cybAgent & " , the agent do not start... Do you see any error message . If so , what is it..Also can you share the log files under <agent>\logs folder during the problem time.


Thanks and regards,
Ravi Kiran

[Anon Anon]

Hi Ravi, thanks for replying. The thing is this only happens during boot, and leaves no trace of an error in the application or system logs. If we manually start the Agent once the machines finish booting, everything goes fine. But if we issue a reboot, the application won't start as part of the normal boot sequence.

How do you invoke the application to auto-start under AIX or any other UNIX OS? The application team gave us an installer and told us to add it to /etc/inittab, via the following command:

[font=Courier New]mkitab "wa:2:once:/opt/cyb/cybAgent &"[font]

[Jonathan Wilson]

I don't have anything to offer on the agent aspect of this, however for another product I'm having a very similar problem. The root of the problem appears to be that during the AIX bootup sequence the shell environment isn't what we think it is - ie. JAVA_HOME isn't being set until sometime late in the boot(after our product tries to start) or perhaps it's not set by the normal .profile or .profile.CA at bootup. The result is very similar to what your seeing - at bootup our product complains of a missing JAVA_HOME value, but if I just login as root and run the startup task (ie /etc/init.d/xx) JAVA_HOME exists and our product starts up properly. Most people(me included) say "root" is the startup user but I'm thinking the normal sourcing of the root users profiles is different during bootup. I'm still working the problem and when I get it figured out I'll come back and update this thread.

[Anon Anon]

That was exactly my first tought when we got notified by the customer about this issue, wiljo. I've noticed the Autosys version of CA Workload Automation Agent asks to be installed under an accout other than root (autosys), and that account's profile must have JAVA_HOME declared. Maybe that's why the application fails to start under root: as you said, the environment isn't totally prepared when the application tries to start...