EPAgeng script for Zlinux

[Anon Anon]

Is there anyone can share the scirpts for monitoring CPU and Memory on Zlinux please? Thanks

Zhang

[Haruhiko Davis]

I don't have a VM to test one, but if you could attach a file with the output from the commands you use to normally monitor them from the commandline (such as "vmstat", "netstat", "iostat"), I can write an updated plug-in for you.

I will write them in Perl so ensure you have at least Perl 5.6 or higher installed.

-Hiko

[Anon Anon]

Hi Hiko,
Thanks for your quick response. Attached is the output. Meanwhile I'd like to monitor CPU usage per each of process. I was using Remote Agent, but that agent has "negative value" issue. When the JVM's recycled the remote agent is not able to kill the old process instead it creates multiple nodes for one JVMs with the CPU usage value -1... So Can you share what command was usded for remote agent to monitor CPU usage or please share the plug-in for monitor CPU usage.. Basically what I want to monitor by EPAgent is CPU and Memory usage. Thanks.

hdavis wrote:

I don't have a VM to test one, but if you could attach a file with the output from the commands you use to normally monitor them from the commandline (such as "vmstat", "netstat", "iostat"), I can write an updated plug-in for you.

I will write them in Perl so ensure you have at least Perl 5.6 or higher installed.

-Hiko

Attachment(s)

output.txt   9 KB 1 version

[Haruhiko Davis]

Please show me in the text file the exact command you ran for each group of metrics, such as "vmstat 2 1".

I also need to know which bits of data you want, or all of it.

[Haruhiko Davis]

When you get a chance, please respond back to my question about the commands you want to have executed.

[Anon Anon]

hdavis wrote:

When you get a chance, please respond back to my question about the commands you want to have executed.

Hi Davis,
Sorry for my late response. The command I was using is ps -auxw, the output is as below:
[root@e41l902m ~]# ps -auxw
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2392 704 ? Ss May05 1:11 init [3]
root 2 0.0 0.0 0 0 ? S< May05 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN May05 0:13 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S< May05 0:00 [migration/1]

The metrics we are interested in are %CPU and %MEM for a specific process. Thanks. Please share your script if possible if possible. Appreciate it. THank you.
Zhang

[Haruhiko Davis]

Please run the following and post the output:

ps -eaf

Post only the lines with the JVM metrics.

Thanks,
Hiko

[Haruhiko Davis]

We should be able to adapt my existing WebSphere process monitoring script.

I just need to adjust the "ps" command to grab your processes and then we're ready for you to test.

[Anon Anon]

[tivadm@e48l002v ~]$ ps -eaf
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May05 ? 00:00:59 init [3]
pecaqv51 3260 1 1 Aug07 ? 00:16:38 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -server -Dwas.status.socket=40531 -Declipse.security -Dosgi.install.area=/opt/IBM/WebSphere/EntAppS
pecaqv51 3752 1 1 Aug07 ? 00:16:14 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -server -Dwas.status.socket=49699 -Declipse.security -Dosgi.install.area=/opt/IBM/WebSphere/EntAppS
pecaqv51 4132 1 1 Aug07 ? 00:17:36 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -server -Dwas.status.socket=44241 -Declipse.security -Dosgi.install.area=/opt/IBM/WebSphere/EntAppS

As the snapshot above, it didn't show full path, but these are the JVM instances...

[Haruhiko Davis]

I'm sorry it took so long to get to this point.

Please use this script I've already written for WebSphere JVM processes: https://communities.ca.com/documents/2301951/7b71afcc-170c-4f00-813f-6b1db008f839

[Anon Anon]

Hi Davis,
Thank you. I saw the name of this script is AIX-JVM-Stats.. This is also compatible with zlinux? And it could provide CPU and Memory info per each of JVM instances?
Zhang

[Haruhiko Davis]

The Perl code is certainly compatible. Any issues will be related to the actual "ps" command that is executed.

Please run the script as detailed in the script or let me know if you have any difficulties.

We could also do a G+ Hangout to do a screen sharing session.

-Hiko

[Haruhiko Davis]

I would just run it manually first without EPA and see if you get results or errors.

If you get errors, please post them here.

[Anon Anon]

Hi Davis,
Here is the output:

[root@e41l902m epaplugins]# perl psweb.pl
<metric type="StringEvent" name="JVM Stats|--noclear:CPU (%)" value="console" />
<metric type="IntCounter" name="JVM Stats|--noclear:RSS" value="1916" />
<metric type="StringEvent" name="JVM Stats|-:CPU (%)" value="su" />
<metric type="IntCounter" name="JVM Stats|-:RSS" value="67168" />
<metric type="StringEvent" name="JVM Stats|-:CPU (%)" value="-" />
<metric type="IntCounter" name="JVM Stats|-:RSS" value="62055" />
printMetric called with incorrect arguments ('type', 'name', and 'value' are mandatory) at psweb.pl line 52


Seems have to do some modification of this script.

[Haruhiko Davis]

Run the "ps" command as it appears in the script.

Look at the resulting output and match it to where the program fails. There appears to be some sort of mismatch in the metric type.

[Haruhiko Davis]

You'll want to edit this command at line 19:

my $psCommand='ps -eaf|awk \'/(websphere[0-9]\/java)/&&!/grep/{print $2}\'|xargs -n1 ps vww|awk \'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$7,$11,$NF;t+=$7;c+=$11}\'';

make sure your output looks like the output I've shown in the program. If it doesn't we'll need to adjust the command.

[Anon Anon]

make sure your output looks like the output I've shown in the program. If it doesn't we'll need to adjust the command.



Where is your output? I didn't see it.

[Haruhiko Davis]

run the entire "ps" command from line 19 in the program or use the code block in my previous post.

[Haruhiko Davis]

Try this:

ps -eaf|awk '/([wW]eb[sS]phere\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$7,$11,$NF;t+=$7;c+=$11}'

[Anon Anon]

Tried, got this


-rwxr-xr-x 1 root root 2380 Aug 8 14:44 psweb.pl
[root@e41l902m epaplugins]# perl psweb.pl
Bareword found where operator expected at psweb.pl line 20, near "]ebsphere"
(Missing operator before ebsphere?)
Backslash found where operator expected at psweb.pl line 20, near "ebsphere\"
syntax error at psweb.pl line 20, near "]ebsphere"
Execution of psweb.pl aborted due to compilation errors.

[Haruhiko Davis]

Thanks that was helpful.
Let me work on this to get this cleaned up for your manual test.

[Haruhiko Davis]

Run the "ps" command from the shell, not the Perl program:

ps -eaf|awk \'/(WebSphere\/java)/&&!/grep/{print $2}\'|xargs -n1 ps vww|awk \'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$7,$11,$NF;t+=$7;c+=$11}\'

Please update with the results.

[Anon Anon]

[root@e41l902m epaplugins]# ps -eaf|awk \'/(WebSphere\/java)/&&!/grep/{print $2}\'|xargs -n1 ps vww|awk \'NF>5&&!/PID/{printf "%s\t%s\t%s\n", $7,$11,$NF,t+=$7;c+=$11}\'
-bash: !/grep/{print: event not found

[Anon Anon]

Hi Davis,
Any update on this please?

[Haruhiko Davis]

I'm still working on the details of the "ps" output so we can get the parsing correct.
It's a bit difficult as I don't have a zSeries system to work with but I'm am testing on RHEL.

[Anon Anon]

Thank you, I can run the script and show you the output from my side for your testing if you need my help. Let me know.

[Haruhiko Davis]

Please send me your output from this command:

ps -eaf|awk '/WebSphere/&&!/grep/{print $2}'|xargs -n1 ps vww

[Anon Anon]

There is a long output.. I just take the part of it.


PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
18455 ? Sl 314:22 5942 50 3182233 1633156 3.3 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -Declipse.security -Dwas.status.socket=41866 -Dosgi.install.area=/opt/IBM/WebSphere/EntAppServer7-64 -Dosgi.configuration.area=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties:/opt/IBM/WebSphere/EntAppServer7-64/properties:/opt/IBM/WebSphere/EntAppServer7-64/lib/startup.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/bootstrap.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/jsf-nls.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/lmproxy.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/urlprotocols.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms1024m -Xmx2048m -Xcompressedrefs -Dws.ext.dirs=/opt/IBM/WebSphere/EntAppServer7-64/java/lib:/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/classes:/opt/IBM/WebSphere/EntAppServer7-64/classes:/opt/IBM/WebSphere/EntAppServer7-64/lib:/opt/IBM/WebSphere/EntAppServer7-64/installedChannels:/opt/IBM/WebSphere/EntAppServer7-64/lib/ext:/opt/IBM/WebSphere/EntAppServer7-64/web/help:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/EntAppServer7-64/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/EntAppServer7-64/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/EntAppServer7-64 -Dpython.cachedir=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -DPVM_LOAD_BALANCER_URL=http://pvm-v0.cmssvc.local:80 -DMSTR_URL=https://portalz7.cms.cmsval/wps/myportal/cms_portal/PV-Reports_V0 -DPVP_LANDING_PAGE_URL=https://portalz7.cms.cmsval/wps/myportal/cmsportal/PV-PQRS/PV_V0/overview1-pv-v0 -DPVA_LANDING_PAGE_URL=https://portalz7.cms.cmsval/wps/myportal/cmsportal/PV-PQRS/PV_V0/overview2-pv-v0 -DmdmCheck=true -javaagent:/opt/CA/PVMV0/wily/Agent.jar -Dcom.wily.introscope.agentProfile=/opt/CA/PVMV0/wily/IntroscopeAgent.profile -Djava.security.auth.login.config=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties/wsjaas.conf -Djava.security.policy=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/config ENT7-CELL node-01 PVMV0-002v-S1
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
21162 ? Sl 514:54 13672 50 1283713 492784 0.9 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -Declipse.security -Dwas.status.socket=33676 -Dosgi.install.area=/opt/IBM/WebSphere/EntAppServer7-64 -Dosgi.configuration.area=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties:/opt/IBM/WebSphere/EntAppServer7-64/properties:/opt/IBM/WebSphere/EntAppServer7-64/lib/startup.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/bootstrap.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/jsf-nls.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/lmproxy.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/urlprotocols.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms256m -Xmx512m -Xcompressedrefs -Dws.ext.dirs=/opt/IBM/WebSphere/EntAppServer7-64/java/lib:/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/classes:/opt/IBM/WebSphere/EntAppServer7-64/classes:/opt/IBM/WebSphere/EntAppServer7-64/lib:/opt/IBM/WebSphere/EntAppServer7-64/installedChannels:/opt/IBM/WebSphere/EntAppServer7-64/lib/ext:/opt/IBM/WebSphere/EntAppServer7-64/web/help:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/EntAppServer7-64/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/EntAppServer7-64/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/EntAppServer7-64 -Dpython.cachedir=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -Djava.security.auth.login.config=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties/wsjaas.conf -Djava.security.policy=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/config ENT7-CELL node-01 RACS-s1

[Haruhiko Davis]

That's exactly what I was looking for; I needed to validate that we're getting the same columns of data.

I'm almost there now.

[Haruhiko Davis]

Run this command:

ps -eaf|awk '/(WebSphere\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'

[Anon Anon]

[root@e48l002v ~]# ps -eaf|awk '/(WebSphere\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'
6568 0.0 -
3420 0.0 -
1856 0.0 -bash
4912 0.0 wasadmin
3476 0.0 -
3392 0.0 -
1780 0.0 -bash
572 0.0 vww
932 0.0 vww
440 0.0 --noclear

[Haruhiko Davis]

That wasn't quite what I wanted.

Try this:

ps -eaf|awk '/(WebSphere)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'

[Anon Anon]

[root@e48l002v ~]# ps -eaf|awk '/(WebSphere)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'
678672 1.3 RNAVal0-s2
673360 1.3 RNAVal0-s1
1058816 2.1 -Dpecos.url=
1173432 2.3 -Xjit:dontInline={jav
1156016 2.3 -Xjit:dontInline={ja
1597480 3.2 REMAS-S1
2543512 5.1 nodeagent
638844 1.2 ACA-ACO-s2
657676 1.3 ACA-AC0-s1
366996 0.7 MACBIS-S1
354104 0.7 HATS-e48l002v-1
427064 0.8 RNSGUI-WVE-V0_node-01
1024004 2.0 EES-S1
1026724 2.0 EES-S2
571904 1.1 CPC-002V-s1
1633160 3.3 PVMV0-002v-S1
492780 0.9 RACS-s1
1023308 2.0 -Djava.sec
1026428 2.0 -Djava.sec
986312 1.9 -Dnppes.useridurl=https://nppes7.cm
1036648 2.0 -Djava.security.auth.l
1014616 2.0 -Dnppes.useridurl=https://nppes7.cm
1028416 2.0 -Dnppes.useridurl=https://nppes7.cms.cmsval/I
195516 0.3 nodeagent
505568 1.0 RNSGUI-V0-s1
473476 0.9 RNSGUI-V0-s2
9086888 18.3 -appl
9140624 18.4 -no
920792 1.8 ESMD-WEB-APP-s1
951416 1.9 ESMD-WEB-APP-s2

[Haruhiko Davis]

We're getting the values now, but not correctly getting the process name.

Let me work on this.

[Anon Anon]

Is your script capable of defining each of the process name by our own? Otherwise the names will confuse our customers. For example, there are 2 jvms for our Application A. Can we name this two processes as JVM_1_APPLICATION A and JVM_2_APPLICATION A, which will be clear when we check the performance data from metric groupings or generate reports based on it.

[Haruhiko Davis]

The names are defined in WAS, and they appear in the output of the "ps -aef" command. I'm grabbing that as the node name.

[Anon Anon]

Yes. I know the name is from the "ps -aef"... I was asking if the script is able to customize the name based on the output of the "ps -aef" command.

[Haruhiko Davis]

You could if you wanted to add more conditional statements in the program.

[Haruhiko Davis]

Try this:

ps -eaf|awk '/(WebSphere.*\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'

[Anon Anon]

[root@e48l002v ~]# ps -eaf|awk '/(WebSphere.*\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'
678768 1.3 RNAVal0-s2
675436 1.3 RNAVal0-s1
1058832 2.1 -Dpecos.url=
1173588 2.3 -Xjit:dontInline={jav
1156056 2.3 -Xjit:dontInline={ja
1597480 3.2 REMAS-S1
2543612 5.1 nodeagent
638840 1.2 ACA-ACO-s2
657676 1.3 ACA-AC0-s1
367016 0.7 MACBIS-S1
354128 0.7 HATS-e48l002v-1
427080 0.8 RNSGUI-WVE-V0_node-01
1024008 2.0 EES-S1
1026776 2.0 EES-S2
571920 1.1 CPC-002V-s1
1633176 3.3 PVMV0-002v-S1
492784 0.9 RACS-s1
1033184 2.0 -Djava.sec
1072212 2.1 -Djava.sec
988236 1.9 -Dnppes.useridurl=https://nppes7.cm
1055836 2.1 -Djava.security.auth.l
1017448 2.0 -Dnppes.useridurl=https://nppes7.cm
1066708 2.1 -Dnppes.useridurl=https://nppes7.cms.cmsval/I
195516 0.3 nodeagent
505568 1.0 RNSGUI-V0-s1
473640 0.9 RNSGUI-V0-s2
9086900 18.3 -appl
9140636 18.4 -no
920852 1.8 ESMD-WEB-APP-s1
951404 1.9 ESMD-WEB-APP-s2


Still having issue.

[Haruhiko Davis]

This is the completed output? No missing JVMs in the list?

[Anon Anon]

Yes, this is complete output.

[Haruhiko Davis]

Try this:

ps -eaf|awk '/(WebSphere\/EntAppServer.*\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'

I think we look for some other unique value in the string.

[Anon Anon]

Looks the output is same


[root@e48l002v ~]# ps -eaf|awk '/(WebSphere\/EntAppServer.*\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'
678944 1.3 RNAVal0-s2
679628 1.3 RNAVal0-s1
1058896 2.1 -Dpecos.url=
1173704 2.3 -Xjit:dontInline={jav
1156056 2.3 -Xjit:dontInline={ja
1597476 3.2 REMAS-S1
2543616 5.1 nodeagent
638840 1.2 ACA-ACO-s2
657676 1.3 ACA-AC0-s1
367024 0.7 MACBIS-S1
354128 0.7 HATS-e48l002v-1
427080 0.8 RNSGUI-WVE-V0_node-01
1024020 2.0 EES-S1
1026812 2.0 EES-S2
571920 1.1 CPC-002V-s1
1633192 3.3 PVMV0-002v-S1
492784 0.9 RACS-s1
1033388 2.0 -Djava.sec
1072232 2.1 -Djava.sec
988476 1.9 -Dnppes.useridurl=https://nppes7.cm
1056088 2.1 -Djava.security.auth.l
1017492 2.0 -Dnppes.useridurl=https://nppes7.cm
1066748 2.1 -Dnppes.useridurl=https://nppes7.cms.cmsval/I
195516 0.3 nodeagent
505568 1.0 RNSGUI-V0-s1
473632 0.9 RNSGUI-V0-s2
9086904 18.3 -appl
9140648 18.4 -no
920852 1.8 ESMD-WEB-APP-s1
951404 1.9 ESMD-WEB-APP-s2

[Haruhiko Davis]

If you match the output of these JVMs:

988476 1.9 -Dnppes.useridurl=https://nppes7.cm
1056088 2.1 -Djava.security.auth.l
1017492 2.0 -Dnppes.useridurl=https://nppes7.cm
1066748 2.1 -Dnppes.useridurl=https://nppes7.cms.cmsval/I

with the output you get from running:

ps -eaf|awk '/(WebSphere\/EntAppServer.*\/java)/&&!/grep/{print $2}'|xargs -n1 ps vww

can you show me just the last column of data (from the 'ps') for that first JVM (ends with 'https://nppes7.cm)? That's basically the arguments from WAS for that node.

[Anon Anon]

/opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -server -Dwas.status.socket=57886 -Declipse.security -Dosgi.install.area=/opt/IBM/WebSphere/EntAppServer7-64 -Dosgi.configuration.area=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties:/opt/IBM/WebSphere/EntAppServer7-64/properties:/opt/IBM/WebSphere/EntAppServer7-64/lib/startup.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/bootstrap.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/jsf-nls.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/lmproxy.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/urlprotocols.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -verbose:gc -Xms512m -Xmx1792m -Dws.ext.dirs=/opt/IBM/WebSphere/EntAppServer7-64/java/lib:/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/classes:/opt/IBM/WebSphere/EntAppServer7-64/classes:/opt/IBM/WebSphere/EntAppServer7-64/lib:/opt/IBM/WebSphere/EntAppServer7-64/installedChannels:/opt/IBM/WebSphere/EntAppServer7-64/lib/ext:/opt/IBM/WebSphere/EntAppServer7-64/web/help:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/EntAppServer7-64/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/EntAppServer7-64/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/EntAppServer7-64 -Dpython.cachedir=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -DADDRESS_VRFCTN_ENBLD=Y -DDMEPOS_ENBLD=Y -DEMAIL_SEND_ENBLD=Y -DSSN_VLDTN_ENBLD=N -Dapp.id=pecos -Dapp.ldap.suffix=ou=Users,ou=nppes -Dcomm.secure.ldap=true -Dhost.name=v3035 -Dhost.port=636 -Dldap.app.admin.dn=uid=nppesadmin,ou=enumerators,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap.app.admin.pwd=14D18B2C7F02BCF9B76D982288646D50 -Dmax.retry.count=3 -Dnppes.url=https://nppes7.cms.cmsval/NPPES/ -Dpay.gov.activityfile.dir=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/logs/PECOSPI-S2/activity/activity -Dpay.gov.activityfile.password=XXXX -Dpay.gov.activityfile.url=http://158.73.175.150:8787/ActivityDownload/ActivityDownload -Dpay.gov.activityfile.username=XXXX -Dpay.gov.application.name=Medicare Application Fee VAL -Dpay.gov.oci.url=https://qa.pay.gov/paygov/OCIServlet?agency_id=1568 -Dretry.wait.time=1200 -Dws.endpoint=http://e48l002v:4012/NPPESIA/services/NPPESIAService http://e48l002v:4013/NPPESIA/services/NPPESIAService http://e48l002v:4014/NPPESIA/services/NPPESIAService -Dws.endpoint.delay.time=240000 -Dnpi.verification.ws.url=http://e48l002v:4012/NPPES/services/NPIVerificationService;http://e48l002v:4013/NPPES/services/NPIVerificationService;http://e48l002v:4014/NPPES/services/NPIVerificationService -Dnpi.checkdigit.sw=Y -Dnpi.verification.sw=N -Dcts.document.ws.url=http://ecm-cts-val.cmssvc.local/cts/CTSWebService -Dcts.document.ws.authBlock=AnwgMgIBAACppKcNd2_a1t4oqa_OqIryxwlDgfme49kLEbcUKLhqmpuufDsQ9svvHYXb6m3KJPGv7GC0R4ic-2dn_YL3112dRUAUC4kWuCIQFIVUmThdlw -Dcts.document.sw=true -DJNDI_NPPES_IA_WS_ENDPOINT=jms:jndi:jms/datawas_nppes_auth_request?jndiConnectionFactoryName=jms/PecosConnectionFactory&targetService=NPPESIAService&replyToName=jms/appwas_nppes_auth_reply&messageType=TEXT -Dnppes.useridurl=https://nppes7.cm

[Haruhiko Davis]

Okay...

The "ps" command is working as expected now. The problem I see if that for the above example, the very last string was the URL; we are expecting the WAS node name.

So my question to you is: should we ignore the JVMs that do not report their node names, or I could write a skeleton branching statement that you can replicate to add your own application names for those not picked up by the command.

I'm sorry this has taken so long and so many iterations. I needed to be sure we're getting the same output results so I can run tests from my RHEL server.

[Anon Anon]

Thank You for your help on my issue. My answer is that we cannot just ignore the jvms that do not report their node names correctly. For example -Dnppes.useridurl=https://nppes7.cm, this is one of our important applications that is required for CPU Memory info. So could you plz write a skeleton branching statement plz?
,

[root@e48l002v ~]# ps -eaf|awk '/(WebSphere)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$NF;t+=$7;c+=$11}'
678672 1.3 RNAVal0-s2
673360 1.3 RNAVal0-s1
1058816 2.1 -Dpecos.url=
1173432 2.3 -Xjit:dontInline={jav
1156016 2.3 -Xjit:dontInline={ja
1597480 3.2 REMAS-S1
2543512 5.1 nodeagent
638844 1.2 ACA-ACO-s2
657676 1.3 ACA-AC0-s1
366996 0.7 MACBIS-S1
354104 0.7 HATS-e48l002v-1
427064 0.8 RNSGUI-WVE-V0_node-01
1024004 2.0 EES-S1
1026724 2.0 EES-S2
571904 1.1 CPC-002V-s1
1633160 3.3 PVMV0-002v-S1
492780 0.9 RACS-s1
1023308 2.0 -Djava.sec
1026428 2.0 -Djava.sec
986312 1.9 -Dnppes.useridurl=https://nppes7.cm
1036648 2.0 -Djava.security.auth.l
1014616 2.0 -Dnppes.useridurl=https://nppes7.cm
1028416 2.0 -Dnppes.useridurl=https://nppes7.cms.cmsval/I
195516 0.3 nodeagent
505568 1.0 RNSGUI-V0-s1
473476 0.9 RNSGUI-V0-s2
9086888 18.3 -appl
9140624 18.4 -no
920792 1.8 ESMD-WEB-APP-s1
951416 1.9 ESMD-WEB-APP-s2

[Anon Anon]

Actually if it's possible that you adjust your script so that I can pick whatever processes I am interested in to show? For example this two processes are not important ACA-ACO-s1 and ACA-ACO-s2 even though the process names are correct. On the other hand, the other one we were working, that process is quite important for us to monitor.

678672 1.3 RNAVal0-s2
673360 1.3 RNAVal0-s1
1058816 2.1 -Dpecos.url=
1173432 2.3 -Xjit:dontInline={jav
1156016 2.3 -Xjit:dontInline={ja
1597480 3.2 REMAS-S1
2543512 5.1 nodeagent
638844 1.2 ACA-ACO-s2
657676 1.3 ACA-AC0-s1
366996 0.7 MACBIS-S1
354104 0.7 HATS-e48l002v-1
427064 0.8 RNSGUI-WVE-V0_node-01
1024004 2.0 EES-S1
1026724 2.0 EES-S2
571904 1.1 CPC-002V-s1
1633160 3.3 PVMV0-002v-S1
492780 0.9 RACS-s1
1023308 2.0 -Djava.sec
1026428 2.0 -Djava.sec

[Haruhiko Davis]

I'm capturing them all when I run the "ps" command, then iterating through the array. You can very easily skip JVMs inside the "foreach" loop.

[Anon Anon]

OK.. Could you write a skeleton branching statement that we can replicate to add our own application names for those not picked up by the command then as you mentioned previously?

[Haruhiko Davis]

I will try to do that.

I see from the last example that have a property called "-Dpay.gov.application.name=". Is there a common property for these applications that do not show their WAS application names?

[Haruhiko Davis]

Let's make sure we can grab the application nodes that report properly.

Run the program and post the results.

Afterwards, we'll work on getting some names attached to the remaining application nodes.

UPDATE: I found a typo in the code; I've attached the latest version for you to try.

Attachment(s)

psWASforLinux.zip   971 B 1 version

[Anon Anon]

Davis,
The new script is working properly. A snapshot is attached. Thank you very much for your help~!!!!!

[Haruhiko Davis]

I'm glad the first part of the process has worked out.

We still need to figure out what to do about the remaining nodes that are not reporting their application names.
The program is ignoring any row in the array that's starts with "-D" or "-X". I also see from the screenshot that there are some application names starting with "-[a-z]".

I began to wonder if this is due to a string length limitation. I remember seeing this before and don't quite remember what the workaround was.

Did you see my previous post to you about trying to figure out if there's a common property that we could key on to use as the application name?

[Anon Anon]

Yes. But seems that's not a common property. I am not sure about that.

[Haruhiko Davis]

How would you like to handle these "unknowns"?

We can't use PIDs because it changes upon startup/restart.

We need to find a system property in that "ps" output to latch onto like that application name from your sample.

[Anon Anon]

For our case, I think the easiest way is to standardise the JVMs' argument of those "unknowns" since the regex in your script works for most of our applications. But issue is that I am not the one who is able to adjust this arguments. I need other team's help, which I know will take time... I am not sure how they define the arguments... Like the example 986312 1.9 -Dnppes.useridurl=https://nppes7.cm, it's one of our key application that needs to be monitored, but I didn't find the key pattern like "nppes-S1" in the whole process name... It might because of the string length limitation.

[Haruhiko Davis]

I don't we need to do anything drastic like changing arguments.

What I'd to see if the "ps -aef" output for each of the applications we are not getting the node name. Let's include those that are showing up as "-[a-z]".

Please attach those in a text file so I can scan them. I'm sure we can find something to use.

[Anon Anon]

Attached is the output.

Attachment(s)

all.txt   102 KB 1 version

[Haruhiko Davis]

Thanks.

I'm analyzing the output now to figure out what we can do with it.

[Haruhiko Davis]

I think I've found a common value.

The WAS log directory "-DSERVER_LOG_ROOT=" provides the actual node name at the end, which I can then search and parse for.

I'll let you know when I'm ready for you to try it. I'm probably going to make it a different program as I'll have to do this scan & parse ahead of everything else.

[Haruhiko Davis]

Looks like only a few have the property. I'll continue to search for a property.

Please run this command and post the output:

ps -eaf|awk '/(SERVER_LOG_ROOT)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$37;t+=$7;c+=$11}'

There are still some processes that do not have the property so I'm still looking for ways to get those.

[Anon Anon]

[z2b3@e48l002v ~]$ ps -eaf|awk '/(SERVER_LOG_ROOT)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$37;t+=$7;c+=$11}'
9125328 18.4 -Dcom.ibm.security.jgss.debug=off
9113088 18.4 -Dserver.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01

[Haruhiko Davis]

Let's try again:

ps -eaf|awk '/(SERVER_LOG_ROOT)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$47;t+=$7;c+=$11}'

Not sure why we got two different results; they should show the same information regardless. At least I thought so. I may need to recount the lines.

[Anon Anon]

[z2b3@e48l002v ~]$ ps -eaf|awk '/(SERVER_LOG_ROOT)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$47;t+=$7;c+=$11}'
9128404 18.4 -Djavax.net.ssl.trustStorePassword=changeit
9111684 18.4 -Djavax.net.ssl.trustStore=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/config/cells/ENT7-CELL/e48l002vesmdAdapt.jks

[Haruhiko Davis]

Just for your information, you might want to point out to your WAS admins that there's a typo in the arguments list for ESMD-CONN-ADPT-s2:

-Dhttps.protocols=TLSv1-Djavax.net.ssl.keyStore=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/config/cells/ENT7-CELL/e48l002vesmdAdapt.jks

should be:

-Dhttps.protocols=TLSv1 -Djavax.net.ssl.keyStore=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/config/cells/ENT7-CELL/e48l002vesmdAdapt.jks

[Haruhiko Davis]

Okay so here's what I've found in comparing two of the JVMs:

The output would match exactly if we can get your WAS admins to adjust the placement of "-verbose:gc" property to the same location in the arguments.
If the command could be placed after "-Dibm.websphere.internalClassAccessMode=allow", that would place "-DSERVER_LOG_ROOT" at tab position 48.
This will make for consistent gathering of information for the JVMs using the property.
The reason why I want to use the property is because it provides the JVM name from WAS in its value.

Please run this command and display the output:

ps -eaf|awk '/(SERVER_LOG_ROOT)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$48;t+=$7;c+=$11}'

I'm still working out what to do about the ones that are not using this property.

[Anon Anon]

[z2b3@e48l002v ~]$ ps -eaf|awk '/(SERVER_LOG_ROOT)/&&!/grep/{print $2}'|xargs -n1 ps vww|awk 'NF>5&&!/PID/{printf "%s\t%s\t%s\n",$8,$9,$48;t+=$7;c+=$11}'
9125696 18.4 -DSERVER_LOG_ROOT=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/logs/ESMD-CONN-ADPT-s1
9111136 18.4 -Djavax.net.ssl.trustStorePassword=changeit

[Haruhiko Davis]

Thanks; I can now setup a second filter for "-DSERVER_LOG_ROOT".

[Haruhiko Davis]

Here is the updated program.

I've also added a filter to not report the metrics from the EPA JVM itself.

If you could just run the program from a shell and put the output in a text file for validation, I'd appreciate it.

I'm still not sure what to do about the remaining JVMs. Are they IBM HTTP Servers? We should probably ignore those and monitor it with the PowerPack for Web Servers.

UPDATE: I've updated the program and reattached the program.

Attachment(s)

psWASforLinux.zip   1 KB 1 version

[Haruhiko Davis]

It looks like we do have another identifier for those remaining servers.

They have a common property called "-Dname.server". It appears at tab stop 57. The only time it doesn't appear at the position is because some enabled debugging on NPPES-s1. This pushed the value to tab stop 59.
So assuming that is what they're doing, we can plan to grab that value at 57 for production monitoring.

[Haruhiko Davis]

Did you try my last update? Can we mark this thread resolved?

[Anon Anon]

Sorry for my late response. I did try your latest script. But there was no much differences btn this one and previous one...

[Haruhiko Davis]

Please run the script on its own and paste the XML to a file and post it here.

[Anon Anon]

Sorry for my late response. Here it is. I dont't see our PECOS JVMs info, which is our important app to monitor..

[root@e48l002v websphere]# perl psWASforLinuxNew.pl
<metric type="IntCounter" name="JVM Stats|REMAS-S1:CPU (%)" value="3" />
<metric type="IntCounter" name="JVM Stats|REMAS-S1:Memory" value="1625912" />
<metric type="IntCounter" name="JVM Stats|EES-S1:CPU (%)" value="2" />
<metric type="IntCounter" name="JVM Stats|EES-S1:Memory" value="1108848" />
<metric type="IntCounter" name="JVM Stats|EES-S2:CPU (%)" value="2" />
<metric type="IntCounter" name="JVM Stats|EES-S2:Memory" value="1128024" />
<metric type="IntCounter" name="JVM Stats|PVMV0-002v-S1:CPU (%)" value="3" />
<metric type="IntCounter" name="JVM Stats|PVMV0-002v-S1:Memory" value="1676096" />
<metric type="IntCounter" name="JVM Stats|MACBIS-S1:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|MACBIS-S1:Memory" value="394812" />
<metric type="IntCounter" name="JVM Stats|REMAS-S3:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|REMAS-S3:Memory" value="424280" />
<metric type="IntCounter" name="JVM Stats|HATS-e48l002v-1:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|HATS-e48l002v-1:Memory" value="373592" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-WVE-V0_node-01:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-WVE-V0_node-01:Memory" value="429812" />
<metric type="IntCounter" name="JVM Stats|nodeagent:CPU (%)" value="5" />
<metric type="IntCounter" name="JVM Stats|nodeagent:Memory" value="2535916" />
<metric type="IntCounter" name="JVM Stats|RACS-s1:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|RACS-s1:Memory" value="490132" />
<metric type="IntCounter" name="JVM Stats|CPC-002V-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|CPC-002V-s1:Memory" value="600740" />
<metric type="IntCounter" name="JVM Stats|nodeagent:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|nodeagent:Memory" value="185740" />
<metric type="IntCounter" name="JVM Stats|ACA-AC0-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ACA-AC0-s1:Memory" value="646692" />
<metric type="IntCounter" name="JVM Stats|ACA-ACO-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ACA-ACO-s2:Memory" value="634612" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s2:Memory" value="717736" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s1:Memory" value="709220" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s1:Memory" value="543896" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s2:Memory" value="545252" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s1:Memory" value="943308" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s2:Memory" value="946484" />
<metric type="IntCounter" name="JVM Stats|ESMD-CONN-ADPT-s1:CPU (%)" value="18" />
<metric type="IntCounter" name="JVM Stats|ESMD-CONN-ADPT-s1:Memory" value="9121736" />
<metric type="IntCounter" name="JVM Stats:CPU (%)" value="18" />
<metric type="IntCounter" name="JVM Stats:Memory" value="9078316" />
<metric type="IntCounter" name="JVM Stats|https_//ehrincentives7.cms.cmstest/:CPU (%)" value="2" />
<metric type="IntCounter" name="JVM Stats|https_//ehrincentives7.cms.cmstest/:Memory" value="1090808" />
<metric type="IntCounter" name="JVM Stats|https_//ehrincentives7.cms.cmstest/:CPU (%)" value="2" />
<metric type="IntCounter" name="JVM Stats|https_//ehrincentives7.cms.cmstest/:Memory" value="1103048" />
<metric type="IntCounter" name="JVM Stats|8080:CPU (%)" value="2" />
<metric type="IntCounter" name="JVM Stats|8080:Memory" value="1053948" />
<metric type="LongCounter" name="JVM Stats:Total Resident Memory (KB)" value="38108960" />





npsqv51 26976 1 0 Aug28 ? 00:20:08 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -server -Dwas.status.socket=60158 -Declipse.security -Dosgi.install.area=/opt/IBM/WebSphere/EntAppServer7-64 -Dosgi.configuration.area=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties:/opt/IBM/WebSphere/EntAppServer7-64/properties:/opt/IBM/WebSphere/EntAppServer7-64/lib/startup.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/bootstrap.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/jsf-nls.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/lmproxy.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/urlprotocols.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms512m -Xmx1536m -Xcompressedrefs -Dws.ext.dirs=/opt/IBM/WebSphere/EntAppServer7-64/java/lib:/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/classes:/opt/IBM/WebSphere/EntAppServer7-64/classes:/opt/IBM/WebSphere/EntAppServer7-64/lib:/opt/IBM/WebSphere/EntAppServer7-64/installedChannels:/opt/IBM/WebSphere/EntAppServer7-64/lib/ext:/opt/IBM/WebSphere/EntAppServer7-64/web/help:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/EntAppServer7-64/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/EntAppServer7-64/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/EntAppServer7-64 -Dpython.cachedir=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -DJVM_ID=SERVER2 -Dbase.dir.logs=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/logs/ -Dcd.apiport.appwas=1363 -Dcd.apiport.datawas=1363 -Dcd.connect.protocol=TCPIP -Dcd.nodename.appwas=e48l002v -Dcd.nodename.datawas=e49l105v -Dcd.passwd.appwas=1835ACB6538C0C088E1B239BAF8D19BE -Dcd.passwd.datawas=1835ACB6538C0C088E1B239BAF8D19BE -Dcd.userid.appwas=npsqv51 -Dcd.userid.datawas=npsqv51 -Demail.from.npi=[e-mail] -Demail.from.npi.efi.rep.nameid=[e-mail] -Demail.subject.npi=National Provider Identifier-CMSVALZW7 -Dgroup1_server_port=8080 -Dgroup1_server_url=v3010.cmssvc.local -Dhitech.url=https://ehrincentives7.cms.cmstest/ -Dis.internet.appwas=Y -DisDataWas=false -Dldap_nppesadmin_dn=uid=nppesadmin,ou=enumerators,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_nppesadmin_pwd=14D18B2C7F02BCF9B76D982288646D50 -Dldap_password_policy_dn_enumerator=cn=nppespasswordpolicyenumerator,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_password_policy_dn_intermediary=cn=nppespasswordpolicyintermediary,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_password_policy_dn_provider=cn=nppespasswordpolicyprovider,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_suffix=ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_unsecure_url=ldap://v3035.cms.hhs.gov:636/ou=nppes,dc=cms,dc=hhs,dc=gov/ -Dldap_url=ldap://v3035.cms.hhs.gov:636/ou=nppes,dc=cms,dc=hhs,dc=gov/ -Dname.server=NPPES-s2 -Dnpi.duplicates.list.info.file=/NPSFileSpace/dupnpi -Dnppes.efi.was1.file.response.zippath=/NPSFileSpace/response/zip/ -Dnppes.efi.was1.file.upload.path=/NPSFileSpace/upload/zip/ -Dnppes.efi.was1.file.upload.watched.path=/NPSFileSpace/upload/watchedzip/ -Dpecos.url=https://pecos7.cms.cmsval/pecos/login.do -DrunController=true -DENV=CMS-VAL -DUseInvalidatedId=false -verbosegc -Xjit:dontInline={ja
npsqv51 27072 1 0 Aug28 ? 00:26:00 /opt/IBM/WebSphere/EntAppServer7-64/java/bin/java -server -Dwas.status.socket=43037 -Declipse.security -Dosgi.install.area=/opt/IBM/WebSphere/EntAppServer7-64 -Dosgi.configuration.area=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Dwas.debug.mode=true -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777 -Xbootclasspath/p:/opt/IBM/WebSphere/EntAppServer7-64/lib/dertrjrt.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/properties:/opt/IBM/WebSphere/EntAppServer7-64/properties:/opt/IBM/WebSphere/EntAppServer7-64/lib/startup.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/bootstrap.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/jsf-nls.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/lmproxy.jar:/opt/IBM/WebSphere/EntAppServer7-64/lib/urlprotocols.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/EntAppServer7-64/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms512m -Xmx1536m -Xcompressedrefs -Dws.ext.dirs=/opt/IBM/WebSphere/EntAppServer7-64/java/lib:/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/classes:/opt/IBM/WebSphere/EntAppServer7-64/classes:/opt/IBM/WebSphere/EntAppServer7-64/lib:/opt/IBM/WebSphere/EntAppServer7-64/installedChannels:/opt/IBM/WebSphere/EntAppServer7-64/lib/ext:/opt/IBM/WebSphere/EntAppServer7-64/web/help:/opt/IBM/WebSphere/EntAppServer7-64/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/EntAppServer7-64/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/EntAppServer7-64/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/EntAppServer7-64 -Dpython.cachedir=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -DJVM_ID=SERVER1 -Dbase.dir.logs=/opt/IBM/WebSphere/EntAppServer7-64/profiles/profile01/logs/ -Dcd.apiport.appwas=1363 -Dcd.apiport.datawas=1363 -Dcd.connect.protocol=TCPIP -Dcd.nodename.appwas=e48l002v -Dcd.nodename.datawas=e49l105v -Dcd.passwd.appwas=1835ACB6538C0C088E1B239BAF8D19BE -Dcd.passwd.datawas=1835ACB6538C0C088E1B239BAF8D19BE -Dcd.userid.appwas=npsqv51 -Dcd.userid.datawas=npsqv51 -Demail.from.npi=[e-mail] -Demail.from.npi.efi.rep.nameid=[e-mail] -Demail.subject.npi=National Provider Identifier-CMSVALZW7 -Dgroup1_server_port=8080 -Dgroup1_server_url=v3010.cmssvc.local -Dhitech.url=https://ehrincentives7.cms.cmstest/ -Dis.internet.appwas=Y -DisDataWas=false -Dldap_nppesadmin_dn=uid=nppesadmin,ou=enumerators,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_nppesadmin_pwd=14D18B2C7F02BCF9B76D982288646D50 -Dldap_password_policy_dn_enumerator=cn=nppespasswordpolicyenumerator,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_password_policy_dn_intermediary=cn=nppespasswordpolicyintermediary,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_password_policy_dn_provider=cn=nppespasswordpolicyprovider,ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_suffix=ou=nppes,dc=cms,dc=hhs,dc=gov -Dldap_unsecure_url=ldap://v3035.cms.hhs.gov:636/ou=nppes,dc=cms,dc=hhs,dc=gov/ -Dldap_url=ldap://v3035.cms.hhs.gov:636/ou=nppes,dc=cms,dc=hhs,dc=gov/ -Dname.server=NPPES-s1 -Dnpi.duplicates.list.info.file=/NPSFileSpace/dupnpi -Dnppes.efi.was1.file.response.zippath=/NPSFileSpace/response/zip/ -Dnppes.efi.was1.file.upload.path=/NPSFileSpace/upload/zip/ -Dnppes.efi.was1.file.upload.watched.path=/NPSFileSpace/upload/watchedzip/ -Dpecos.url=

[Haruhiko Davis]

Here's the problem with NPPES-s1 and s2:

You have two additional entries at tab stop 11 & 12 for s1:

 -Dwas.debug.mode=true
 -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777

This pushed the tab stop index for "-Dname.server" from 56 to 58.

We need the parameters we're parsing for to be in the same location at all times.
Ask your WAS admins to move those parameters after "-Dname.server".

[Haruhiko Davis]

UPDATE: I made a slight change to a couple of the array names. The file has a timestamp of 08/31/2013 12:47 AM.

This should be the last update.

This will pick up NPPES-s2, but not s1 because of my previous post. Please see if you can get your WAS admins to move those two runtime parameters after "-Dname.server".

Please post your XML output to a file and attach to the thread.


-Hiko

Attachment(s)

psWASforLinux.zip   1 KB 1 version

[Anon Anon]

[root@e48l002v websphere]# perl psWASforLinuxLatest.pl
Global symbol "$property2" requires explicit package name at psWASforLinuxLatest.pl line 80.
Global symbol "$nodeName2" requires explicit package name at psWASforLinuxLatest.pl line 80.
Global symbol "$nodeName2" requires explicit package name at psWASforLinuxLatest.pl line 84.
Global symbol "$nodeName2" requires explicit package name at psWASforLinuxLatest.pl line 90.
Global symbol "$property3" requires explicit package name at psWASforLinuxLatest.pl line 104.
Global symbol "$nodeName3" requires explicit package name at psWASforLinuxLatest.pl line 104.
Global symbol "$nodeName3" requires explicit package name at psWASforLinuxLatest.pl line 108.
Global symbol "$nodeName3" requires explicit package name at psWASforLinuxLatest.pl line 114.
Execution of psWASforLinuxLatest.pl aborted due to compilation errors.

[Haruhiko Davis]

Silly mistake on my part.

Attachment(s)

psWASforLinux.zip   1 KB 1 version

[Haruhiko Davis]

Please let me know if we've resolved the issue and close out this thread.

[Anon Anon]

Here is the output of the latest script. The challenge is that our WAS team keeps changing the argument format so that we have to keep changing our script. One more thing, I have been running this agent for days and saw the error metric shows a lot of errors. I am not sure if it has any affects to the monitoring?

[root@e48l002v websphere]# perl psWASforLinuxLatest.pl
<metric type="IntCounter" name="JVM Stats|CPC-002V-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|CPC-002V-s1:Memory" value="584188" />
<metric type="IntCounter" name="JVM Stats|nodeagent:CPU (%)" value="5" />
<metric type="IntCounter" name="JVM Stats|nodeagent:Memory" value="2554596" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s1:Memory" value="615372" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNAVal0-s2:Memory" value="632296" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s1:Memory" value="500452" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|RNSGUI-V0-s2:Memory" value="529988" />
<metric type="IntCounter" name="JVM Stats|REMAS-S1:CPU (%)" value="3" />
<metric type="IntCounter" name="JVM Stats|REMAS-S1:Memory" value="1625060" />
<metric type="IntCounter" name="JVM Stats|ACA-ACO-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ACA-ACO-s2:Memory" value="726884" />
<metric type="IntCounter" name="JVM Stats|ACA-AC0-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ACA-AC0-s1:Memory" value="723840" />
<metric type="IntCounter" name="JVM Stats|nodeagent:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|nodeagent:Memory" value="196064" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s1:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s1:Memory" value="919316" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s2:CPU (%)" value="1" />
<metric type="IntCounter" name="JVM Stats|ESMD-WEB-APP-s2:Memory" value="929120" />
<metric type="IntCounter" name="JVM Stats|HATS-e48l002v-1:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|HATS-e48l002v-1:Memory" value="354156" />
<metric type="IntCounter" name="JVM Stats|PVMV0-002v-S1:CPU (%)" value="3" />
<metric type="IntCounter" name="JVM Stats|PVMV0-002v-S1:Memory" value="1563468" />
<metric type="IntCounter" name="JVM Stats|RACS-s1:CPU (%)" value="0" />
<metric type="IntCounter" name="JVM Stats|RACS-s1:Memory" value="485300" />
<metric type="StringEvent" name="JVM Stats:CPU (%)" value="18.0" />
<metric type="IntCounter" name="JVM Stats:Memory" value="8948600" />
<metric type="StringEvent" name="JVM Stats|ESMD-CONN-ADPT-s1:CPU (%)" value="18.0" />
<metric type="IntCounter" name="JVM Stats|ESMD-CONN-ADPT-s1:Memory" value="8946912" />
<metric type="StringEvent" name="JVM Stats:CPU (%)" value="1.9" />
<metric type="IntCounter" name="JVM Stats:Memory" value="963196" />
<metric type="StringEvent" name="JVM Stats|v3010.cmssvc.local:CPU (%)" value="2.1" />
<metric type="IntCounter" name="JVM Stats|v3010.cmssvc.local:Memory" value="1066112" />
<metric type="StringEvent" name="JVM Stats|v3010.cmssvc.local:CPU (%)" value="2.1" />
<metric type="IntCounter" name="JVM Stats|v3010.cmssvc.local:Memory" value="1060300" />
<metric type="LongCounter" name="JVM Stats:Total Resident Memory (KB)" value="33925220" />

[Haruhiko Davis]

You're going to continue to get errors because you can't get the WAS admins to stop moving the JVM arguments around.
It's not impacting performance of EPA; it's just an XML parsing error.

[Haruhiko Davis]

This also affects the metric "Total Resident Memory" because we're missing the "RSS" value on the bad parsing.

[Haruhiko Davis]

Original thread resolved.

The new thread can be found at 101796322.

[Haruhiko Davis]

The output from the "ps" command should look like this (it's detailed in my program):

#29272   0.4     proc_1
#94428   0.4     proc_2
#45820   0.5     nodeagent

Where col1 is RSS, col2 is CPU, col3 processname

[Haruhiko Davis]

When you talk about "remote agent", is this something you setup or are you talking about the Remote Unix Agent FieldPack developed by CA Services?

[Anon Anon]

Hi Davis,
Yes, that's what I mean to say. Remote agent = Remote Unix Agent fieldpack. I want to see the performance data as same as the one we got from Remote agent fieldpack for zlinux. The reason why we need to try it out from EPAgent was because there is a internal issue ( "negative value" ) I mentioned in my previous reply in that fieldpack.

Zhang