Automic Workload Automation

  • Private Community
 View Only

  • 1.  Delays when JSCH prepares schedule for new day

    Posted Aug 09, 2022 12:17 PM

    I have run into an issue where the timing of preparing certain workflows is extending the time for our new processing day to start, and hoping for a few ideas on where I can look to locate what may be causing the issue.

    At 20 minutes this delay may seem small, but it causes several other processes to be off by the same time, which can be a snowball effect

    Our Automic support if through Fiserv, and they forward it on to Broadcom, so this has proved to take much longer to get resolved.

    Prior to July 12th
    - JSCH runs at 12:05 am
    - workflows for new day are up and running within 3 minutes

    As of July 12th
    - JSCH runs at 12:05 am
    - on weekends, workflows for new day are up and running within 3 minutes
    - on weekdays, workflows for new day are up and running after 20+ minutes 

    I am aware of approximately 20 new objects added to one workflow (JOBP.DAILY.ACH) and the sub JOBP's on July 11th.  The majority are EVNT's

    Yesterday I set JOBP.DAILY.ACH to inactive so I could see if that was causing the delays, and it proved to be true.  The normal remaining workflows took two minutes to prepare and start

    At 12:46 am I started JOBP.DAILY.ACH, and the objects


    - Preparing shows to have stopped on this for a few minutes, which I don't think is normal

    - I answered that, and the workflow still show as Preparing


    - 12:50 am, three of the four JOBP's on the right still show as preparing

    - 12:59 am, and we are down to two showing as preparing



    - 1:06 am and the workflow has finally started

    I have searched the boards and found that another client had an issue with an Agent not being available, but that isn't the case here.



  • 2.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 09, 2022 12:40 PM
    Hi Rick and hi all,
    I opened a case for Broadcom's support because, in a randomic and unpredictable way, sometimes JOBP in our production Automic system remain in the state of "preparing", without a specific reason.
    This is a problem because when this happens during the night and for critical JOBPs, we can only restart the launch in the morning during the monitoring checks.
    E.g. yesterday i noticed a JOBP in the preparing status about from the 4th of August, I read in its activation report:

    U00000063 Partner '*CP004#00000554' is not connected to the server.

    but I don't know the meaning of this message.

    Monica
    Original Message


  • 3.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 09, 2022 12:55 PM

    Thanks Monica

    I checked the Activation and I can see where it slows down.  I may drill down a little more on this

    2022-08-09 00:45:56 - U00007000 'JOBP.DNA.GET.ACH.J06.0030' activated with RunID '0003889626'.
    2022-08-09 00:45:56 - U00007000 'JOBP.DNA.GET.ACH.J11.0330' activated with RunID '0003889627'.
    2022-08-09 00:45:56 - U00007000 'JOBP.DNA.GET.ACH.J06.0330' activated with RunID '0003889628'.
    2022-08-09 00:45:56 - U00007000 'JOBP.DNA.GET.ACH.J11.1130' activated with RunID '0003889629'.
    2022-08-09 00:45:56 - U00007000 'JOBP.DNA.GET.ACH.J06.1130' activated with RunID '0003889630'.
    2022-08-09 00:45:57 - U00007000 'JOBP.DNA.GET.ACH.J11.1530' activated with RunID '0003889631'.
    2022-08-09 00:45:58 - U00007000 'JOBP.DNA.GET.ACH.J06.1530' activated with RunID '0003894374'.
    2022-08-09 00:45:58 - U00007000 'JOBP.DNA.GET.ACH.J11.1730' activated with RunID '0003896278'.
    2022-08-09 00:45:59 - U00007000 'JOBP.DNA.GET.ACH.J06.1730' activated with RunID '0003896281'.
    2022-08-09 00:48:00 - U00007000 'JOBP.DNA.ACH.0030' activated with RunID '0003887640'.
    2022-08-09 00:54:03 - U00007000 'JOBP.DNA.ACH.0330' activated with RunID '0003890512'.
    2022-08-09 00:58:06 - U00007000 'JOBP.DNA.ACH.1130' activated with RunID '0003890529'.
    2022-08-09 01:02:10 - U00007000 'JOBP.DNA.ACH.1530' activated with RunID '0003890543'.
    2022-08-09 01:06:14 - U00007000 'JOBP.DNA.ACH.1730' activated with RunID '0003895149'.


    Original Message


  • 4.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 09, 2022 01:08 PM

    and that lead me to 

    2022-08-09 00:48:00 - U00020206 Variable '&TD_TODAY#' was stored with value '08-09-2022'.

    2022-08-09 00:48:00 - U00007002 'EVNT.ACH.FILE.0330' has earliest start time: '+00/03:30'

    2022-08-09 00:48:00 - U00007002 'JOBS.WIN.COPY.ACH.0330' has earliest start time: '+00/03:30'

    2022-08-09 00:48:00 - U00007000 'EVNT.ACH.FILE.0330' activated with RunID '0003890513'.

    2022-08-09 00:48:00 - U00007000 'EVNT.ACH.IAT.FILE.0330' activated with RunID '0003890514'.

    2022-08-09 00:54:01 - U00007000 'JOBS.WIN.COPY.ACH.0330' activated with RunID '0003890515'.

    2022-08-09 00:54:01 - U00007000 'EVNT.ACH.FILE' activated with RunID '0003891541'.

    2022-08-09 00:54:01 - U00007000 'JOBP.FOREACH.ACH_FILELOADER' activated with RunID '0003891542'.

    2022-08-09 00:54:01 - U00007000 'JOBS.DNA.EF_ERROR0_606' activated with RunID '0003896287'.

    2022-08-09 00:54:01 - U00007000 'JOBS.DNA.PS_FWPDAT' activated with RunID '0003896288'.

    2022-08-09 00:54:02 - U00007000 'JOBS.DNA.AH_CLEAR_657_CREDITS' activated with RunID '0003896289'.

    2022-08-09 00:54:02 - U00007000 'EVNT.ACH.IAT.J06.2FILES.0330' activated with RunID '0003896290'.

    2022-08-09 00:54:02 - U00007000 'JOBP.IAT.PROCESSING.1' activated with RunID '0003896291'.

    2022-08-09 00:54:02 - U00007000 'CALL.ACH.POST.DEBITS' activated with RunID '0003890524'.

    2022-08-09 00:54:02 - U00007000 'JOBS.DNA.AH_CLEAR_657_DEBITS' activated with RunID '0003890525'.

    2022-08-09 00:54:02 - U00007000 'JOBS.DNA.AH_PENDING80_5137' activated with RunID '0003890526'.

    2022-08-09 00:54:03 - U00007000 'CALL.ACH.PENDING.UPDATE' activated with RunID '0003890527'.

    2022-08-09 00:54:03 - U00007000 'JOBS.DNA.AH_PENDING90_5137' activated with RunID '0003890528'.

    2022-08-09 03:34:11 - U00020237 The object variable '&TODAY_GT_2_BUSINESS_DAY#' in object: 'JOBS.DNA.PS_FWPDAT(1)', line: '00006' (RunID: '0003896288') has been created with the value '08-11-2022' by  using the command :PSET.

    But that is where I leave off.  I cant locate anything that shows why it would take so long between those steps


    Original Message


  • 5.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 10, 2022 06:23 AM

    Found a few unneeded date calcs in some EVNT's.  That shouldn't make any difference, but we removed them anyways.  

    Also noticed the following EVNT's dont have a login listed.  We have several other EVNT's setup this way, and those dont seem to be causing issues

    Any other suggestions, please let me know

    We are still waiting to hear back from Broadcom via Fiserv (our support)


    Original Message


  • 6.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 10, 2022 06:56 AM
    Do you have a lot of UC4 script within the objects?  Depending on your requirements, maybe switch the objects to Generate at runtime.  That way they generate when each is ready to run instead of everything at the same time.
    Original Message


  • 7.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 10, 2022 07:27 AM

    Jared,

    There really isnt much for scripting in them.  Staff added around 20 EVNT's to the workflows on July 11th, and then the delays started when the workflows were rebuilt on the 12th.

    It is strange because adding some EVNT's shouldnt cause the 20 minute delay.  Just doesnt make sense.

    I was thinking about Generate at Runtime yesterday.  It may be the next thing I attempt to change. 


    Original Message


  • 8.  RE: Delays when JSCH prepares schedule for new day

    Posted Aug 12, 2022 07:02 AM

    Doesnt make sense to me, but the EVNT's were causing the delays.

    There are five JOBP's that had these EVNT's, and each one of them were taking 4+ minutes to generate, so that ended up being the 20+ min delay.

    Switched these to "Generate at Runtime" yesterday, and the 20+ delay didn't occur when todays workflows were built.



    Original Message