Automic Workload Automation

Expand all | Collapse all

PREP_PROCESS_REPORT only working on DB reports by design?

  • 1.  PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-13-2019 07:59 AM

    Hi.

     

    Suppose I do this in a JOBS:

     

    "Process" tab:

    echo "Flamingo"
    echo "Chicken"
    echo "Flamingo Chick"

     

    "Post-Process" tab:

    :SET &HND# = PREP_PROCESS_REPORT("JOBS",, "REP", "*lamin*")
    :PROCESS &HND#
    : SET &RET# = GET_PROCESS_LINE(&HND#)
    : PRINT "from report: &RET#"
    :ENDPROCESS

     

    Then my post process report looks like this:

     

    2019-05-13 13:48:45 - U00020408 from report: Flamingo
    2019-05-13 13:48:45 - U00020408 from report: Flamingo Chick

     

    So far, so good.

     

    Now when I switch the JOBS to store its runtime report NOT in the database, but in a file (via the checkboxes in the UNIX tab), this immediately fails. &HND# will not contain a valid file handle for PREP_PROCESS_REPORT to read out of.

     

    I heard someone(tm) say this is by design, but can't find any written evidence of that.

     

    Does anyone know if this is indeed by design, and if so, is it written anywhere? Or should rectifying this go into an "idea" ...?

     

    p.s. if it is by design, this is dumb. I told a bunch of people to store their excessively long reports in the file system instead of shoveling jiggabytes into the database and potentially causing muchos problemos in the process, but this immediately backfired because of this nugget 

     

    p.p.s. sadly, the users can not use a simple output scan instead

     

    p.p.p.s. the 11.2 documentation says:

     

    " PREP_PROCESS_REPORT

    Comments

    This script function provides the report contents of executable objects for further processing with AE Script. The report is read from the AE database and prepared as data sequence.

     

    But this is not really conclusive and still dumb, and the 12.2 documentation contains no such thing.

     

    Thanks!



  • 2.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-13-2019 02:57 PM

    The 11.2 documentation also says "The report is read from the AE database...".  Seems pretty explicit to me.

     

    We require our applications to write voluminous reports (or csv files, or whatever) as a separate file onto a file-server, rather than letting them be stored on our application servers.    We usually have UC4 passing the target folder/file specification as a parameter so that the file name can include the UC4 runid, which has turned out to be quite useful for us.



  • 3.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-13-2019 02:58 PM

    And when UC4 is passing the folder/file name, it can then use PREP_PROCESS_FILE() against that output. 



  • 4.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-13-2019 03:00 PM

    We also use the REGISTER_OUTPUTFILE() function to logically attach the output file to the UC4 statistics entry.  Then you can search the report with an OUTPUT_SCAN too if you want.



  • 5.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-14-2019 04:12 AM

    Pete,

     

    Cheers for the hints, I'll look into REGISTER_OUTPUTFILE.

     

    Either way, our users voiced they will not change their scripts, and I have no authority to enforce that. They have reverted to putting loads of data into the DB, despite load problems. I have to somewhat agree with their reasoning: It should not be neccessary to change the scripts to totally different commands when changing the report storage option. The report retrieval is transparent when you open a report from AWA, it should be transparent when using scripting, too.

     

    The report is read from the AE database...".  Seems pretty explicit to me.


    Yeah ... that may be so, but they exorcised this from the 12.x docs, so it's not canon anymore.

     

    Br,



  • 6.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-13-2019 04:14 PM

    Hi

    Job - WIN tab:

    Database

    The job report is handled by AE. After a job has been processed, the Automation Engine takes the process protocol which has been stored on the agent computer and stores it to the AE database. Doing so is only possible if a protocol can be created. It is not possible if a command has been used.

    File

    The job report is handled by AE. After the job execution, the process protocol is available as a file in the target system. This only works if a protocol can be created. It is not possible if a command has been used.

    Automic 

     

    If the Job report is not to be stored into the automic DB because of the "save report in DB checkbox", its on the agent only (or nowhere, because not created). So where should AE get its data from?

     

    This behavior has always been this way (could be a sentence from CA DEV :-)   ).

     

    A possible workaround would be "On error only".

     

    cheers, Wolfgang



  • 7.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-14-2019 04:21 AM

    Wolfgang old (methaphorically) friend!

     

    So where should AE get its data from

     

    Where does it get it from when displaying a report stored on the agent? It manages to get the matching report for displaying it in AWA with no problems whatsoever, so why can't it call the exact same existing "sub-routine" to fetch it when running PREP_PROCESS_REPORT?

     

    A possible workaround would be "On error only".

    Sadly, they can't do that due to their logic. They want to parse the output for number of files processed, then do arithmetic on that, then set status codes based on that.

     

    Cheers,

    Carsten



  • 8.  Re: PREP_PROCESS_REPORT only working on DB reports by design?



  • 9.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-14-2019 06:54 AM

    Wolfgang old (methaphorically) friend!

    Sometime I feel like an UC4 dinosaur...

     

    Where does it get it from when displaying a report stored on the agent?

    => Job messenger sends the Report to the AE (=DB)

     

    Usually (simplified)

    user presses "run", AE generates JCL (a batch or SH file), transfers it to the agent, the agent executes it and stores the report in a textfile, job messenger sends the TXT file and the status  it back to AE (=DB).

     

    Hope I didnot get you wrong...

     

    They want to parse the output for number of files processed, then do arithmetic on that, then set status codes based on that.

    What about using means of OS to count the files in your report-file and use REGISTER_VARIABLE ?

    Its a hard way but it might work....

     

    cheers, Wolfgang



  • 10.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-14-2019 07:41 AM

    Job messenger sends the Report to the AE (=DB)

    I highly doubt the "=DB" part for when your job is told to store it in the agent file system exclusively

     

    Run a simple job and store the report on the agent file system. Open report in AWI. It will display it, and I highly doubt it gets pushed into the DB first when you merely display it in AWI. And even it it would, that would not change anything. Quite on the contrary, if it were that way, then there would already be code in AWA to pull it back into the DB (and there would already be useless redundancy), so even less reason not to have PREP_PROCESS_REPORT operate on it.

     

    Bottom line, AWA can locate the proper report for a given job run even when it's in the file system on an agent machine, so there is, to me, no reason not to make PREP_PROCESS_REPORT work the same way, no matter where the report physically is.

     

    What about using means of OS to count the files in your report-file and use REGISTER_VARIABLE ?

    Thanks. But that was in fact my very first suggestion to them (as you probably know, I'm always advocating to do stuff in shell script instead of AutomicScript ).

    But they don't want to change their jobs, and I have no authority (nor desire) to force this, nor dabble in their job objects. So any proposals for alternative coding solutions are unfortunately moot.

    I would agree that this may seem ridiculous at first, but this is how my company wants it to be: ~60 part-time Automic "users" all creating and running their own jobs (and at some point, Automic has also sold their interfaces as facilitating exactly that) and I have no final authority over what they create.

     

    And as I pointed out repeatedly: While I tell them when I think something doesn't make sense in their jobs, in this case I have to agree with them. There is no good reason why a different storage location should neccessitate a different script function.

     

    I think Automic should facilitate that, so yeah, the idea is there

     

    Cheers,

    Carsten



  • 11.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-14-2019 12:57 PM

    If it is decided that you will be required to store very large reports in the AE database, then you'll also need to study the MAX_REPORT_SIZE setting inside of the UC_HOSTCHAR_DEFAULT variable.  This setting restricts what the largest size of report will be accepted into the AE database.  When this limit is exceeded, the report is left on the application server and it will not be imported into the AE database.

     

    There is always an upper limit.  This products primary mission is to schedule and run tasks, not to be a report repository.



  • 12.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-15-2019 04:20 AM

    Cheers, will keep it in mind. And yes, totally agree on the "Automic is not a document management server" but alas, again, the powers that be direct what the powers that be direct.

     

    It's also somewhat sad that one can not freely use MAX_REPORT_SIZE as intended. We recently discovered that PREP_PROCESS_XML also uses MAX_REPORT_SIZE for it's limit, hence we had to increase MAX_REPORT_SIZE for a user who demanded to be able to process large XML files with Automic . The dual role of that variable is, however, officially declared as "as designed".



  • 13.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-15-2019 04:23 AM

    Oh, but the good news is, MAX_REPORT_SIZE is also a UC_HOSTCHAR-capable variable. So one can set it different for certain hosts, so that may be a way arround some issues.



  • 14.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted 05-16-2019 03:32 PM

    maybe i can add some facts here:

     

    1) prep-process-report is AE (wp) function and using SQL to read data from the DB

    2) a job messenger is not sending any report 

    2.1) the report is sent by the agent - if store in db is used

    2.2) the report transfer is triggered by the AE (wp) 

    3) because of 1) this will only work if 2.1) is used

    4) prep-process-file is exactly the script function what is doing the expected (working with file content on agents)

    4.1) but keep in mind if the data is stored on the disk - someone must remove these files

    4.2) i can remember lots of customer tickets in the past - where this kind of logic filled the disk in a very short time...

    5) the UI/AWI is able to display the data because of the feature to look into running job outputs

    5.1) but this is not touching the AE (wp) at all

    5.2) this simple get-file function is going from the UI/AWI to the connected CP and is routed from that CP to the CP where the agent is connected and requesting a file - this file is then passed back from the agent through the CP's to the UI to display the content

    5.3) as mentioned before this feature was implemented that a user can have a look inside the output of long-running tasks - where/what is within the output - no need to login to this box and look into the report in question.

    6) if the report is stored in file and a prep-process-report would do the transfer implicit what is the benefit -> still the data must be transferred to the DB and if all jobs have this command it makes no difference if the flag is set or not - data is stored in DB for processing

    7) the only thing that might make sense is to enhance the prep-process-report with a flag - if the report is not within DB use prep-process-file - but this can already be achieved today with a simple if - do the report function or the file function

    7.1) but still the data is then transferred from the agent to the DB again for further processing

    7.2) this logic will only reduce the data to be transferred - but data is still written to DB

    7.3) as written here counts are searched - so data will always be transferred - the benefit would not be too big...

    7.4) if just counts are needed - use the  :REGISTER-VARIABLE function and no need to transfer any report

    8) but the file version might lead than very often into 4.2)

    9) 4.2) leads in a complete outage of the app server

    ...

     

    hope this helps and can be passed on to your uc4-scripting developers

    holli