How to reduce time to move data from Automic vara to Text file

[MohammedNawazuddin613134]

In Automic vara I have 18,000 records which I need to move to text file.

I am using Automic scripting which is taking more than 4 hours to move data to Text file

 

:SET &QUERY_VAR# = &RESULT_VARA#

:SET &QUERY_VARA_COLUMN# = "2"
:SET &COLUMN#="COLNAME"
:SET &HND1# = CREATE_PROCESS(NEW)
:SET &LINE# = "LEGACY_PART_NUMBER"
:SET &LINE1# = ""
:SET &RET# = PUT_PROCESS_LINE(&HND1#,&LINE#)
:SET &RET# = PUT_PROCESS_LINE(&HND1#,&LINE1#)
:SET &PREFIX# = ""
:SET &HND2#=PREP_PROCESS_VAR(&QUERY_VAR#)
:PROCESS &HND2#
: PSET &VAL# = GET_PROCESS_LINE(&HND2#, &QUERY_VARA_COLUMN#)

: PSET &VAL2# = "&VAL#"
: SET &RET# = PUT_PROCESS_LINE(&HND1#,&VAL2#)

:ENDPROCESS
: PSET &FILEPATH# = "D:\AutomicFileStore\Spare Parts list active in CONA &DATE1#.txt"

:SET &RET# = WRITE_PROCESS(&HND1#,&FILEPATH#,AE_WIN01, AE01_WIN.LOGIN,OVERWRITE)

 

 

 

I am using the above scripting which is taking more than 4 hours.

 

Is there any logic so that time can be reduced to minutes in the above scripting.

[FrankMuffke]

Hi

 

out of my guts I would guess it could be a performance issue of your DB.

What is stated in your WP log with U number U00003533 ? Thats above the line

U00003544 UCUDB: Reference values tested with Windows 2003 on XEON 1500 MHz: CPU 813865, DB 470

 

What do you exactly want to do -just write the content of a VARA object into a file?

 

cheers, Wolfgang

[MohammedNawazuddin613134]

Yes , I want to transfer the the data from Automic vara to a text file.

The vara contains 5000 records which is taking nearly 40 minutes to get transferred to a text file.

So , Is there any code so that I can reduce the time from 40 minutes to 5 minutes or less than 5 minutes for 5000 records.

[Carsten Schmitz]

Out of curiosity:

 

I made a VARA with 14.000 lines. Then I used this code to write it to a text file:

 

:SET &HND#=PREP_PROCESS_VAR(VARA.DC1.BIGVAR_FOR_WRITEOUT_TEST)
:PROCESS &HND#
:ENDPROCESS
:SET &RET# = WRITE_PROCESS(&HND#,"/tmp/flamingo",myagent, mylogin)
:CLOSE_PROCESS &HND#

This job worked in 1 second on my test system (which is even in compatibility mode from the last botched ZDU).

 

Lessons learned:

 

• large (for Automic) VARAs can be written out surprisingly fast
• when I had the WRITE_PROCESS ahead of the :ENDPROCESS, it feeds the ENTIRE variable each time, so it tried to write 14000 times 14000 lines. That indeed took 20 minutes, maybe that's the problem?
• (unrelated to this): when you set such a job to "generate at activation time", the AWI waits for generation, is unusable for a minute or two, then reports a "timeout" while job generation happily continues in the engine. That could be considered a bug.

 

But regardless of the 1 second performance: Working with such large VARAs in AWI is a pain, and my personal, non-official recommendation is this: If you need to do processing on somewhat large data sets, if you maybe not just need to print the VARA as-is, but parse it or otherwise manipulate it: don't use Automics scripting language.

 

Use perl or python.

 

Hth.

[FrankMuffke]

Yo Brotha,

 

just a performance optimizing hint - the loop

:PROCESS &HND#
:ENDPROCESS

is only necessary if you want to do (whatever) line per line.

 

If you onlywant to use write_process its enough:

 

:SET &HND#=PREP_PROCESS_VAR(VARA.DC1.BIGVAR_FOR_WRITEOUT_TEST)
:SET &RET# = WRITE_PROCESS(&HND#,"/tmp/flamingo",myagent, mylogin)
:CLOSE_PROCESS &HND#

cheers, Wolfgang