Monitoring Experian Powercurve with APM

[Legacy User]

We have a requirement to monitor an application called Experian Powercurve which runs in a windows Env and a service is created with a batch code where we have entered our agent config, we have been told that --javaagent is unsupported so we are using the xbootclasspath method. however the agent is not starting .. to configure the agent we create a agentconnector.jar using the steps described here : Process SAML Authentication and then edit the following bat file :

 

SET APM_HOME_DIR=E:\apm

 

%SERVICE_NAME%.exe //IS//%SERVICE_NAME% --Startup=auto --JvmOptions=-Xrs;-XX:PermSize=128M;-XX:MaxPermSize=256M;-Xbootclasspath/p:%APM_HOME_DIR%\connectors\AutoProbeConnector.jar';'%APM_HOME_DIR%\Agent.jar';'-

Dcom.wily.introscope.agentProfile=%APM_HOME_DIR%\core\config\IntroscopeAgent.profile;-Xms3072M;-Xmx3072M;-Dlogback.configurationFile="%CLIENT_SOLUTION%\conf\system\logback.xml";-

Dclient.solution.home="%CLIENT_SOLUTION%";-Dib.home="%IB_HOME%";-Duser.dir="%CLIENT_SOLUTION%";-Dgroovy.source.encoding=UTF-8;-Dderby.stream.error.file="%DERBY_LOG%";%BOOTSTRAP_DEBUG_OPT%;%SECURITY_POLICY_OPT%;

--Jvm=auto --Classpath="%IB_HOME%\lib\core\enterprise-bootstrap.jar";"%IB_HOME%\bin";"%CLIENT_SOLUTION%\conf\system" --StartPath="%IB_HOME%\lib\core" --StartClass=com.experian.eda.enterprise.startup.Bootstrap --

StartParams=-fa;"%CLIENT_SOLUTION%\conf\system\camel-context.xml"; --StartMode=jvm --StopPath="%IB_HOME%\lib\core" --StopClass=com.experian.eda.enterprise.startup.Bootstrap --StopMethod=stopApplication --

StopMode=jvm --StopTimeout=0 --LogPrefix=service --StdOutput=auto --StdError=auto --LogPath="%CLIENT_SOLUTION%\logs\service" --LogLevel=Info --JavaHome="C:\Program Files\Java\jre7"

@echo Successfully installed Connectivity service.

 

When we run the bat file this writes to the registry the java startup config like such :

 

-Xrs

-XX:PermSize=128M

-XX:MaxPermSize=256M

-Xbootclasspath/p:E:\apm\connectors\AutoProbeConnector.jar;E:\apm\Agent.jar;-Dcom.wily.introscope.agentProfile=E:\apm\core\config\IntroscopeAgent.profile

-Xms3072M

-Xmx3072M

-Dlogback.configurationFile=E:\Experian\PCE_CON_Solution\conf\system\logback.xml

-Dclient.solution.home=E:\Experian\PCE_CON_Solution

-Dib.home=E:\Experian\PowerCurve\Connectivity\Connectivity v1.2SP6

-Duser.dir=E:\Experian\PCE_CON_Solution

-Dgroovy.source.encoding=UTF-8

-Dderby.stream.error.file=E:\Experian\PCE_CON_Solution\logs\derby.log

 

When we then start the Application from services the Application starts successfully however the Agent does not.

Looking in the Application logs we see the following :

 

2015-05-11 17:04:59 Commons Daemon procrun stderr initialized

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Agent Release 9.6.1.0 (Build 961036)

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Using Java VM version "Java HotSpot(TM) 64-Bit Server VM 1.7.0_45" from Oracle Corporation

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Trying to load agent profile based on system property "com.wily.introscope.agentProfile"

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Unable to load agent profile because the system property could not be found

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Trying to load agent profile from resource from the class path based on system property "com.wily.introscope.agentResource"

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Unable to load agent profile because the system property could not be found

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Trying to load agent profile based on the working directory at: E:\Experian\PCE_CON_Solution\wily\IntroscopeAgent.profile

5/11/15 05:05:01 PM EST [INFO] [IntroscopeAgent.Agent] Unable to load agent profile because of: java.io.FileNotFoundException E:\Experian\PCE_CON_Solution\wily\IntroscopeAgent.profile (The system cannot find the path specified)

 

Any idea why the agent is looking in this location where we havent installed any APM files as they are in E:\APM ?

Something to do with using the AutoProbeConnector.jar file ?

 

 

Thanks.

[Guenter DI Grossberger]

Hi Andy,

 

as all the java options are written on a separate line and

-Xbootclasspath/p:E:\apm\connectors\AutoProbeConnector.jar;E:\apm\Agent.jar;-Dcom.wily.introscope.agentProfile=E:\apm\core\config\IntroscopeAgent.profile

is written on the same line I think there is an error with the quotes (you are using single quotes before and after the ';' ). They should be on separate lines like all the other options:

 

-Xbootclasspath/p:E:\apm\connectors\AutoProbeConnector.jar;E:\apm\Agent.jar

-Dcom.wily.introscope.agentProfile=E:\apm\core\config\IntroscopeAgent.profile

 

Therefore the property is not set in the JVM and the agent cannot load the profile.

 

Try:

-Xbootclasspath/p:"%APM_HOME_DIR%\connectors\AutoProbeConnector.jar;%APM_HOME_DIR%\Agent.jar";-

Dcom.wily.introscope.agentProfile="%APM_HOME_DIR%\core\config\IntroscopeAgent.profile";

 

Ciao,

Guenter

[Legacy User]

I have tried the change you recommended :

my config now looks like this :

 

%SERVICE_NAME%.exe //IS//%SERVICE_NAME% --Startup=auto --JvmOptions=-Xrs;-XX:PermSize=128M;-XX:MaxPermSize=256M;-Xbootclasspath/p:"%APM_HOME_DIR%\connectors\AutoProbeConnector.jar;%APM_HOME_DIR%\Agent.jar";-

Dcom.wily.introscope.agentProfile="%APM_HOME_DIR%\core\config\IntroscopeAgent.profile";-Xms3072M;-Xmx3072M;-Dlogback.configurationFile="%CLIENT_SOLUTION%\conf\system\logback.xml";-

Dclient.solution.home="%CLIENT_SOLUTION%";-Dib.home="%IB_HOME%";-Duser.dir="%CLIENT_SOLUTION%";-Dgroovy.source.encoding=UTF-8;-Dderby.stream.error.file="%DERBY_LOG%";%BOOTSTRAP_DEBUG_OPT%;%SECURITY_POLICY_OPT%;

--Jvm=auto --Classpath="%IB_HOME%\lib\core\enterprise-bootstrap.jar";"%IB_HOME%\bin";"%CLIENT_SOLUTION%\conf\system" --StartPath="%IB_HOME%\lib\core" --StartClass=com.experian.eda.enterprise.startup.Bootstrap --

StartParams=-fa;"%CLIENT_SOLUTION%\conf\system\camel-context.xml"; --StartMode=jvm --StopPath="%IB_HOME%\lib\core" --StopClass=com.experian.eda.enterprise.startup.Bootstrap --StopMethod=stopApplication --

StopMode=jvm --StopTimeout=0 --LogPrefix=service --StdOutput=auto --StdError=auto --LogPath="%CLIENT_SOLUTION%\logs\service" --LogLevel=Info --JavaHome="C:\Program Files\Java\jre7"

@echo Successfully installed Connectivity service.

 

The batch file runs successfully and we can now see the entries on separate lines in the registry but now the "E:\apm\Agent.jar is on a separate line.

 

-Xrs

-XX:PermSize=128M

-XX:MaxPermSize=256M

-Xbootclasspath/p:E:\apm\connectors\AutoProbeConnector.jar

E:\apm\Agent.jar

-Dcom.wily.introscope.agentProfile=E:\apm\core\config\IntroscopeAgent.profile

-Xms3072M

-Xmx3072M

-Dlogback.configurationFile=E:\Experian\PCE_CON_Solution\conf\system\logback.xml

-Dclient.solution.home=E:\Experian\PCE_CON_Solution

-Dib.home=E:\Experian\PowerCurve\Connectivity\Connectivity v1.2SP6

-Duser.dir=E:\Experian\PCE_CON_Solution

-Dgroovy.source.encoding=UTF-8

-Dderby.stream.error.file=E:\Experian\PCE_CON_Solution\logs\derby.log

 

The Service fails to start - no logs

 

If i then add single quotations between the following : -Xbootclasspath/p:"%APM_HOME_DIR%\connectors\AutoProbeConnector.jar';'%APM_HOME_DIR%\Agent.jar";

 

I then get the following output in the registry :

 

-Xrs

-XX:PermSize=128M

-XX:MaxPermSize=256M

-Xbootclasspath/p:E:\apm\connectors\AutoProbeConnector.jar;E:\apm\Agent.jar

-Dcom.wily.introscope.agentProfile=E:\apm\core\config\IntroscopeAgent.profile

-Xms3072M

-Xmx3072M

-Dlogback.configurationFile=E:\Experian\PCE_CON_Solution\conf\system\logback.xml

-Dclient.solution.home=E:\Experian\PCE_CON_Solution

-Dib.home=E:\Experian\PowerCurve\Connectivity\Connectivity v1.2SP6

-Duser.dir=E:\Experian\PCE_CON_Solution

-Dgroovy.source.encoding=UTF-8

-Dderby.stream.error.file=E:\Experian\PCE_CON_Solution\logs\derby.log

 

The bat is sucessfully run and the service starts however i now see this entry in the logs :

 

2015-05-12 08:38:10 Commons Daemon procrun stderr initialized

Unrecognized option: E:\apm\Agent.jar

 

2015-05-12 08:44:55 Commons Daemon procrun stderr initialized

May 12, 2015 8:45:43 AM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass

[Haruhiko Davis]

What's the reason for using '-Xbootclasspath' with HotSpot JRE instead of '-javaagent'? Why prepend instead of append when using '-Xbootclasspath'?

 

Did you also create your own AutoProbeConnector? What was the resulting output from creating it?

[Legacy User]

Hi Hiko,

 

1 and 2 advised by support ..

 

E:\apm\connectors>"c:\Program Files\Java\jre7\bin\java.exe" -jar CreateAutoProbeConnector.jar  -current

5/12/15 09:25:06 AM EST [INFO] [Creator] AutoProbe Connector Creation Utility Release 9.6.1.0 (Build 961036)

5/12/15 09:25:06 AM EST [INFO] [Creator] Using Java VM version "Java HotSpot(TM) 64-Bit Server VM 1.7.0_45" from Oracle

Corporation

5/12/15 09:25:06 AM EST [INFO] [Creator] CA Wily Introscope(R) Version 9.6.1.0

5/12/15 09:25:06 AM EST [INFO] [Creator] Copyright (c) 2014 CA. All Rights Reserved.

5/12/15 09:25:06 AM EST [INFO] [Creator] Introscope(R) is a registered trademark of CA.

 

The AutoProbe Connector was successfully created at E:\apm\connectors\AutoProbeConnector.jar.

 

E:\apm\connectors>

[Haruhiko Davis]

According to the documentation, you will need to specify '--JvmOptions' multiple times for every value being passed as a parameter.

To append, use '++' instead of '--'.

 

http://commons.apache.org/proper/commons-daemon/procrun.html

 

I believe this is why procrun doesn't recognize 'Agent.jar' as a parameter.

[Lynn Williams]

Hi Hiko,

Andy has a support case running for this which I have been helping my colleague Ying Zhou with.

The PowerCurve Java application uses the Apache Commons daemon procrun to be able to run as a Windows Service

Only "-D" & "-X" parameters are allowed for the procrun JvmOptions: http://commons.apache.org/proper/commons-daemon/procrun.html

So that is why we advised using Xbootclasspath

We also advised adding the single quotes for the ; separating the AutoprobeConnector.jar and Agent.jar within the Xbootclasspath expression to prevent procrun from processing that ; as it's own separator

 

Hi Andy,

From your latest update on the case it looks like the agent profile did not load because system property "-Dcom.wily.introscope.agentProfile" is not being found from the JVM initialisation. I think that is because you also added single quotes around the ; after the Agent.jar and so procrun did not process the "-Dcom.wily.introscope.agentProfile".

You should only need the single quotes around the ; separating the AutoprobeConnector.jar and Agent.jar within the Xbootclasspath expression.

 

Hope this helps

 

Lynn

[Haruhiko Davis]

Makes sense to me.

[Legacy User]

Hi Lynn - i made that change (using only the sigle quote separator earlier after seeing guenters suggestion - but still did not resolve the issue.

I shall look into using the ++ parameter.

[Lynn Williams]

Hi Andy,

Yes, using ++JvmOptions for each separate JVM parameter should also work.

However with what you had the top of this page the Agent.jar was being loaded correctly and the remaining problem was just the agent profile not being found.

So if you revert to that setup and just remove the single quotes after the Agent.jar "...'%APM_HOME_DIR%\Agent.jar';'-Dcom.wily.introscope.agentProfile ..." I would expect the profile file to be found and be loaded.

Regards,

Lynn

[Guenter DI Grossberger]

Hi Andy,

 

although not documented -javaagent works with procrun, too. Just tested it

 

So no need for -Xbootclasspath and the extra ';'!

 

I suspect specifying each single parameter with ++JvmOptions still won't work as -Xbootclasspath/p:"%APM_HOME_DIR%\connectors\AutoProbeConnector.jar';'%APM_HOME_DIR%\Agent.jar" is a single parameter and procrun will probably still interpret the semicolon as a parameter separator.

 

I think the correct syntax would be -Xbootclasspath/p:"%APM_HOME_DIR%\connectors\AutoProbeConnector.jar"';'"%APM_HOME_DIR%\Agent.jar"

In slow spelling: double quote after AutoProbeConnector.jar, hen single quoted semicolon, then double quoted Agent.jar!

 

Ciao,

Guenter

[Lynn Williams]

Hi Guenter,

When Andy first logged the case we saw this in the powercurve-connectivity-stderr*.log file:

2015-05-07 15:00:46 Commons Daemon procrun stderr initialized

Unrecognized option: -javaagent=E:\apm\Agent.jar

 

So after we checked the procrun web page we assumed -javaagent could not be used.

However I believe the Tomcat windows service also uses procrun and I know I can use -javaagent on the Java tab/Java Options for that, so I am not 100% convinced

Maybe it is the way the -javaagent parameter was originally being included?

 

Per the procrun web page using single quotes is the advised way to "escape" the ; i.e.

if you need to embed either # or ; character put them inside single quotes.

 

Regards,

Lynn

[Guenter DI Grossberger]

Hi Lynn,

 

After scrolling back up the whole page I think the error in the first post was that the ; after Agent.jar was also escaped with single quotes.

 

So for separating java options use the unquoted semicolon, when separating entries in a path use ';'.

 

As to -javaagent: the syntax is -javaagent:<path> for Windows, too. So the error was coming from the JVM rather than procrun/prunsrv. Try running java -javaagent=Agent.jar Agent.jar. You'll get the same error message!

 

So try with : instead of = and -javaagent should work!

 

Ciao,

Guenter

[Lynn Williams]

Thanks Guenter - I missed the -javaagent syntax problem

 

Andy,

So it looks like you have a couple of choices i.e.

Either continue with Xbootclasspath and remove the unwanted ';' escape

Or revert to using -javaagent via "-javaagent:E:\apm\Agent.jar "

 

Sorry we missed that!

 

Regards,

 

Lynn

[Legacy User]

Thanks Guenter and everyone for their input I am now up and running with the syntax Guenter provided.

[Legacy User]

FYI i have also tried using the following :

 

-javaagent:"%APM_HOME_DIR%\Agent.jar";-Dcom.wily.introscope.agentProfile="%APM_HOME_DIR%\core\config\IntroscopeAgent.profile"

 

Which also works - thanks again.

[Haruhiko Davis]

Do not use semicolon between 'javaagent' and the agent profile. They are two distinct Java options.

Instead use '++JvmOptions' and append the agent profile as another JVM argument.