Automic Workload Automation

Expand all | Collapse all

How to reduce time to move data from Automic vara to Text file

  • 1.  How to reduce time to move data from Automic vara to Text file

    Posted 07-04-2018 07:36 AM

    In Automic vara I have 18,000 records which I need to move to text file.

    I am using Automic scripting which is taking more than 4 hours to move data to Text file

     

    :SET &QUERY_VAR# = &RESULT_VARA#

    :SET &QUERY_VARA_COLUMN# = "2"
    :SET &COLUMN#="COLNAME"
    :SET &HND1# = CREATE_PROCESS(NEW)
    :SET &LINE# = "LEGACY_PART_NUMBER"
    :SET &LINE1# = ""
    :SET &RET# = PUT_PROCESS_LINE(&HND1#,&LINE#)
    :SET &RET# = PUT_PROCESS_LINE(&HND1#,&LINE1#)
    :SET &PREFIX# = ""
    :SET &HND2#=PREP_PROCESS_VAR(&QUERY_VAR#)
    :PROCESS &HND2#
    : PSET &VAL# = GET_PROCESS_LINE(&HND2#, &QUERY_VARA_COLUMN#)

    : PSET &VAL2# = "&VAL#"
    : SET &RET# = PUT_PROCESS_LINE(&HND1#,&VAL2#)


    :ENDPROCESS
    : PSET &FILEPATH# = "D:\AutomicFileStore\Spare Parts list active in CONA &DATE1#.txt"

    :SET &RET# = WRITE_PROCESS(&HND1#,&FILEPATH#,AE_WIN01, AE01_WIN.LOGIN,OVERWRITE)

     

     

     

    I am using the above scripting which is taking more than 4 hours.

     

    Is there any logic so that time can be reduced to minutes in the above scripting.



  • 2.  Re: How to reduce time to move data from Automic vara to Text file

    Posted 07-05-2018 07:28 AM

    Hi

     

    out of my guts I would guess it could be a performance issue of your DB.

    What is stated in your WP log with U number U00003533 ? Thats above the line

    U00003544 UCUDB: Reference values tested with Windows 2003 on XEON 1500 MHz: CPU 813865, DB 470

     

    What do you exactly want to do -just write the content of a VARA object into a file?

     

    cheers, Wolfgang



  • 3.  Re: How to reduce time to move data from Automic vara to Text file

    Posted 07-11-2018 02:59 AM

    Yes , I want to transfer the the data from Automic vara to a text file.

    The vara contains 5000 records which is taking nearly 40 minutes to get transferred to a text file.

    So , Is there any code so that I can reduce the time from 40 minutes to 5 minutes or less than 5 minutes for 5000 records.



  • 4.  Re: How to reduce time to move data from Automic vara to Text file

    Posted 07-12-2018 07:29 AM

    Out of curiosity:

     

    I made a VARA with 14.000 lines. Then I used this code to write it to a text file:

     

    :SET &HND#=PREP_PROCESS_VAR(VARA.DC1.BIGVAR_FOR_WRITEOUT_TEST)
    :PROCESS &HND#
    :ENDPROCESS
    :SET &RET# = WRITE_PROCESS(&HND#,"/tmp/flamingo",myagent, mylogin)
    :CLOSE_PROCESS &HND#

    This job worked in 1 second on my test system (which is even in compatibility mode from the last botched ZDU).

     

    Lessons learned:

     

    • large (for Automic) VARAs can be written out surprisingly fast
    • when I had the WRITE_PROCESS ahead of the :ENDPROCESS, it feeds the ENTIRE variable each time, so it tried to write 14000 times 14000 lines. That indeed took 20 minutes, maybe that's the problem?
    • (unrelated to this): when you set such a job to "generate at activation time", the AWI waits for generation, is unusable for a minute or two, then reports a "timeout" while job generation happily continues in the engine. That could be considered a bug.

     

    But regardless of the 1 second performance: Working with such large VARAs in AWI is a pain, and my personal, non-official recommendation is this: If you need to do processing on somewhat large data sets, if you maybe not just need to print the VARA as-is, but parse it or otherwise manipulate it: don't use Automics scripting language.

     

    Use perl or python.

     

    Hth.



  • 5.  Re: How to reduce time to move data from Automic vara to Text file

    Posted 07-12-2018 08:54 AM

    Yo Brotha,

     

    just a performance optimizing hint - the loop

    :PROCESS &HND#
    :ENDPROCESS

    is only necessary if you want to do (whatever) line per line.

     

    If you onlywant to use write_process its enough:

     

    :SET &HND#=PREP_PROCESS_VAR(VARA.DC1.BIGVAR_FOR_WRITEOUT_TEST)
    :SET &RET# = WRITE_PROCESS(&HND#,"/tmp/flamingo",myagent, mylogin)
    :CLOSE_PROCESS &HND#

    cheers, Wolfgang