Automic Workload Automation

Expand all | Collapse all

Monitoring AE server performance using AE scripting

  • 1.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 10:30 AM
    I am looking for a way to automate the monitoring of AE performance. The   SYS_BUSY_01() function is capable of returning the current usage of the WP where the command happens to run, but there is no way to specify the name of the server process. Is there a way to use AE scripting to monitor the performance of all AE server processes?


  • 2.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 10:34 AM
    Hi Michael_Lowry,

    Some time ago I posted this : https://community.automic.com/discussion/6927/system-healthcheck

    One way to have SYS_BUSY run for all sever processes is to use a SQL statement nested in SQLI to get the list of server processes, then use the list as an input parameter in PREP_PROCES_VAR.

    Hope this helps.

    Best regards,
    Antoine



  • 3.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 10:42 AM
    Thanks, Antoine_Sauteron, but I do not think that does what I want. It does not appear to show the usage percentage of individual AE server processes.

    What I am doing now is something like this:
    :SET &counter# = 0
    :WHILE &counter# < 200
    :  SET &AE_Usage# = SYS_INFO(MQPWP, BUSY, "1")
    :  SET &AE_Usage# = FORMAT(&AE_Usage#)
    :  PRINT "AE usage: &AE_Usage#"
    :  WAIT 1
    :  SET
    &counter# = &counter + 1
    :ENDWHILE
    This shows the usage of the AE server (I guess this is either the PWP or some sort of average.) I’m interested in the individual WPs & CPs.


  • 4.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 10:48 AM
    Hmmm ... I use SYS_BUSY_01 in scripts, and in all honesty, you just made me conciously aware for the first time about that caveat. Yes, the documentation actually states that SYS_BUSY_01 and friends only report for the Worker Process they run on :(

    Or, I read that part in the past and thought "this must be a mistake, they can't be serious".

    Maybe Automic can comment whether that's actually true, or a documentation bug. Guess this could explain why my server monitor script has a comment to self, which reads that the results of SYS_BUSY_01 sometimes seem off ...

    I also use SYS_INFO(various_MQ_names, COUNT) and SYS_INFO(various_MQ_names, LENGTH, 1) for monitoring the server, that stuff goes straight into a Zabbix diagram, but I am not aware of any way to force the WP for the SYS_BUSY family of functions.




  • 5.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 10:51 AM
    Follow-Up:

    There is SYSINFO(MQPWP, BUSY, PERIOD), which seems to fetch the overall UC4 system load:

    "Supplies the Automation Engine's workload in percent."

    It appears from the documentation the BUSY argument can only be used with the MQPWP queue. Not sure what the significance of that is though.




  • 6.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 10:54 AM
    Upon scouring my source code: I am actually using that in my server monitor, too.

    !PWP is the only one that has load averages for it's MQ
    :SET &MQ_PWP_BUSY_01# = SYS_INFO(MQPWP, BUSY, 1)
    :SET &MQ_PWP_BUSY_01_FMT# = FORMAT(&MQ_PWP_BUSY_01#)

    :SET &MQ_PWP_BUSY_10# = SYS_INFO(MQPWP, BUSY, 10)
    :SET &MQ_PWP_BUSY_10_FMT# = FORMAT(&MQ_PWP_BUSY_10#)

    :SET &MQ_PWP_BUSY_60# = SYS_INFO(MQPWP, BUSY, 60)
    :SET &MQ_PWP_BUSY_60_FMT# = FORMAT(&MQ_PWP_BUSY_60#)


    I get alerts when this goes above certain limits, and it at least seemed to work decently over the recent year.


  • 7.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 02:07 PM
    Hahaaaa :-)

    Yes, SYS_INFO DOES return an average over all Processes so its not useful if you want the load of just one specific WP...

    The only way I would know (which is one more of eternal workarounds) is:
    parsing the WP logs for U00021601 and U00021602 which show the statistics per WP:
    (sorry for the bad formatting)

    20171023/152608.706 - U00021601                 AVGR.   1:00  2:00  3:00  4:00  5:00  6:00  7:00  8:00  9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 00:00
    20171023/152608.706 - U00021602 'Client:   0000 00,16  00,00 00,00 00,00 00,00 00,00 00,00 00,00 01,48 00,14 00,16 00,15 00,16 00,14 00,16 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00'
    20171023/152608.706 - U00021602 'Client:   0001 00,01  00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,25 00,00 00,00 00,01 00,01 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00'
    20171023/152608.706 - U00021602 'Client:   0003 00,00  00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,22 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00'
    20171023/152608.706 - U00021602 'Client:   0004 00,00  00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,01 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00'
    20171023/152608.706 - U00021602 'Client:   All  00,17  00,00 00,00 00,00 00,00 00,00 00,00 00,00 01,95 00,14 00,16 00,16 00,18 00,14 00,16 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00 00,00'


    the second only way that gets to my mind is catching the OS PID if every WP and use OS functions for getting the workload.





  • 8.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 02:17 PM
    short addon to the second only way:

    with the following (T-)SQL you can check for the currently running WPs and CPs:
    select ah_name, AH_ProcessId, ah_timestamp1  from AH
    where ah_otype = 'SERV'
    and AH_TimeStamp4 is NULL

    JWP is taken into consideration but at least with MS SQL Server the Process PID has another format that the other "real" WPs....


  • 9.  Monitoring AE server performance using AE scripting

    Posted 10-24-2017 05:21 PM
    The AE logs server process usage averaged over three different time periods: 1 second, 10 seconds, and 60 seconds. There are two ways to fetch these data: an average for the whole AE server, or the usage of the individual server process that happens to run the script.

    AE server average

    These AE scripting functions take no arguments, and simply return the usage of the particular WP that happens to run the script. There is no way to specify the AE server process for which the usage should be returned.
    • SYS_INFO(MQPWP, BUSY, "1") — Average AE server usage over the last 1 second
    • SYS_INFO(MQPWP, BUSY, "10") — Average AE server usage over the last 10 seconds
    • SYS_INFO(MQPWP, BUSY, "60") — Average AE server usage over the last 60 seconds

    Individual server process

    These AE scripting functions simply return the usage of the particular WP that happens to run the script.
    • SYS_BUSY_01() — Single AE server process usage over the last 1 second
    • SYS_BUSY_10() — Single AE server process usage over the last 10 seconds
    • SYS_BUSY_60() — Single AE server process usage over the last 60 seconds
    The functions take no arguments. There appears to be no way to specify the name of an individual AE server process whose usage should be returned.

    Are these statistics perhaps stored somewhere in the AE DB?


  • 10.  Monitoring AE server performance using AE scripting

    Posted 10-25-2017 03:01 AM

    These data can be accessed via the ServerListItem Java API class. Here’s a bit of sample code:

    private void getServerList(){
         ServerList serverList = new ServerList();
         localUC4Object.sendRequestAndWait(serverList);
         getMsgBox(serverList);
         String processTypeDesc = "";
         String processRoleDesc = "";
         for (Iterator<ServerListItem> serverListIter = serverList.iterator(); serverListIter.hasNext(); ){
              ServerListItem serverListItem = serverListIter.next();
              boolean processIsActive = serverListItem.isActive();
              if (processIsActive) {
                   String processName = serverListItem.getName();
                   char processType = serverListItem.getType();
                   String processRole = serverListItem.getRole();
                   String processVersion = serverListItem.getVersion();
                   DateTime processTime = serverListItem.getServerTime();
                   int processB01 = serverListItem.getB01();
                   switch (processType){
                        case 'P': processTypeDesc = "Primary work process";
                        break;
                        case 'W': processTypeDesc = "Work process";
                        break;
                        case 'D': processTypeDesc = "Dialog work process";
                        break;
                        case 'C': processTypeDesc = "Communications process";
                        break;
                        case 'J': processTypeDesc = "Java work process";
                        break;
                        case 'A': processTypeDesc = "Java communications process";
                        break;
                        default: processTypeDesc = "Unknown";
                        break;
                   }
                   log.writeLine(String.format("AE process : %s", processName));
                   if (! "".equals(processRole)) {
                        switch (processRole) {
                             case "O": processRoleDesc = "Output handler";
                             break;
                             case "R": processRoleDesc = "Resource calculations";
                             break;
                             default: processRoleDesc = "Unknown";
                        }
                        log.writeLine(String.format("Type : %s (Role: %s)", processTypeDesc,processRoleDesc));
                   } else{
                        log.writeLine(String.format("Type : %s", processTypeDesc));
                   }
                   log.writeLine(String.format("Version : %s", processVersion));
                   log.writeLine(String.format("Date & time : %s", processTime.toString()));
                   log.writeLine(String.format("B01 utilization : %s", processB01));
                   log.writeLine("");
              }
         }
    }

    The ServerList.iterator method returns the results in no particular order. Here is some sample output:

    AE process             : UC4_EXP#WP001
    Type                   : Primary work process
    Version                : 12.0.3+build.18019
    Date & time            : 2017-10-25 17:48:36
    B01 utilization        : 100

    AE process             : AE_TEST1#WP012
    Type                   : Work process (Role: Output handler)
    Version                : 12.0.3+build.18019
    Date & time            : 2017-10-25 17:44:47
    B01 utilization        : 100

    AE process             : AE_TEST1#WP011
    Type                   : Java work process
    Version                : 12.0.3+build.3912
    Date & time            : 2017-10-25 17:44:47
    B01 utilization        : 0

    AE process             : AE_TEST1#CP003
    Type                   : Communications process
    Version                : 12.0.3+build.1291
    Date & time            : 2017-10-25 17:44:47
    B01 utilization        : 0

    AE process             : AE_TEST1#WP041
    Type                   : Dialog work process
    Version                : 12.0.3+build.18019
    Date & time            : 2017-10-25 17:44:47
    B01 utilization        : 4

    AE process             : AE_TEST1#WP054
    Type                   : Work process
    Version                : 12.0.3+build.18019
    Date & time            : 2017-10-25 17:44:47
    B01 utilization        : 100

    AE process             : AE_TEST1#WP013
    Type                   : Work process (Role: Resource calculations)
    Version                : 12.0.3+build.18019
    Date & time            : 2017-10-25 17:44:47
    B01 utilization        : 100

    I think this will meet our needs.



  • 11.  Monitoring AE server performance using AE scripting

    Posted 10-25-2017 05:33 AM
    Michael Lowry said:
    Are these statistics perhaps stored somewhere in the AE DB?
    Unfortunately not, its in the server`s cache only.
    THX for the info regarding Java API class!


  • 12.  Monitoring AE server performance using AE scripting

    Posted 10-25-2017 11:55 AM
    I updated my comment above with some sample code.