Automic Workload Automation

 View Only
Expand all | Collapse all

File Transfer Job (JOBF)

  • 1.  File Transfer Job (JOBF)

    Posted Jan 31, 2020 05:16 PM
    Edited by Kenneth Hutchins Jan 31, 2020 05:16 PM
    Is there an option within a File Transfer Job to allow only new files place in the source directory to transfer to the destination directory?


    We do not want the files to be deleted from the source directory and have tried all three options for If file exists at destination

    • Cancel transfer

    If the file already exists in the target system, the transfer is canceled.

    • Overwrite file

    If the file already exists in the target system, the file is replaced by the one that is being transferred.

    • Append file
    If the file already exists in the target system, the file that is being transferred is appended to the existing one.

    Cancel transfer works, it doesn't copy the existing files but it throws errors which cause the UC4 Job to fail.

    Overwrite/Append cause the job to run too long and also increases the size of the existing files moved.




  • 2.  RE: File Transfer Job (JOBF)

    Posted Jan 31, 2020 07:25 PM
    For speed and efficiency, I sometimes use the DOS robocopy instruction.  It has a "mirror" feature that assures the content of the source and target folders match.  It will only copy files that are missing or have been changed.  And it will also delete files that no longer exist in the source folder.

    I don't know if that would work for your use-case though.  It requires the source and target be available as network shares.

    ------------------------------
    Pete
    ------------------------------



  • 3.  RE: File Transfer Job (JOBF)

    Posted Jun 16, 2022 09:45 AM
    hey PETER ,

    is there any way to use robocopy in automic , 

    i use xcopy and it's take long time

    thanks


  • 4.  RE: File Transfer Job (JOBF)

    Posted Feb 02, 2020 05:05 AM
    Dear Kenneth,

    instead of processing the file transfer on the whole directory, you can use an exec VARA to process only the files that have changed since the last run. The file transfer then runs on individual files by using a parameter and being started from a  script or a FOR EACH workflow.

     Alternatively you can copy the files changed after the last run into another directory and get them from there by your file transfer.

    In order to remember the last run you can set the current date and time stamp in a VARA.

     To be sure to get all files you might add some time difference and use the overwrite option, but since you don't process all files the performance should be better.

    I hope this helps,

    ------------------------------
    Regards, Nicole
    ------------------------------



  • 5.  RE: File Transfer Job (JOBF)

    Posted Feb 03, 2020 02:29 AM
    Hi Kenneth,

    I would use 2 VARA.FILELIST that compare source and target directory. Next, put them into a VARA.MULTI and use "Difference". Make sure the source VARA is the first one in your VARA.MULTI selection. Than use the VARA.MULTI as source in the JOBP.FOREACH start object. The variable &item# contains the current file name for each loop. &item# will be automatically passed to the child objects of your JOBP.FE. Now simply put your JOBF into the JOBP.FE, set the source and target path and add &item# as filename. That's it.






  • 6.  RE: File Transfer Job (JOBF)

    Posted Feb 03, 2020 10:52 AM
    Thanks this appears to have worked.


  • 7.  RE: File Transfer Job (JOBF)

    Posted 22 days ago

    Hello christoph, i tried it but it doesnt copy the file existing in Folder A to Folder b

    i want to copy files from a to b from the last day thats it




  • 8.  RE: File Transfer Job (JOBF)
    Best Answer

    Posted Feb 03, 2020 05:25 AM
    Edited by Diane Craddock Feb 03, 2020 10:25 AM
    Hi Kenneth,

    "...Cancel transfer works, it doesn't copy the existing files but it throws errors which cause the UC4 Job to fail…"

    Good question!

    I just tried to solve this problem, because i could need such "feature" too in the future.

    What about this solution? Just copy this code and paste it in PostProces of your FileTransfer Job!

    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    ! Problem:
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    ! With following setting in JOBF
    !      If file exists at destination = Cacel Transfer 
    ! existing files will be NOT overwritten at destination, BUT UC4 Job will fail
    !
    ! Assumption: The file "/tmp/test.txt" exists at destination already.
    ! Result:     You'll find following rows in the REP report:
    !   ...
    !   2020-02-03 11:14:14 - U00011134 ERROR '0' bytes, incorrect transfer of '0' records for file 'C:\temp\test.txt'->'/tmp/test.txt'. Duration  '00:00:00'.
    !   2020-02-03 11:14:14 - U00011404 File '/tmp/test.txt' already exists.
    !   ...
    !   2020-02-03 11:14:14 - U00011409 FT '1307040': FileTransfer ended abnormally.
    !
    !
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    ! Objective: Do NOT fail UC4 job if only "FileAlreadyExist errors" occurred
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    !
    !
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    ! Possible solution:
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    !
    ! Use Post Process to check REP report:
    !  1. Count amount of rows in REP report containing    "* ERROR *" occurred during FileTransfer
    !  2. Count amount of rows in REP report containing    "*U00011404 File * already exists.*" occurred during FileTransfer
    !                                              or just "* already exists.*"
    !  3. If    "Amount of ERRORs" EQ "Amount of FileAlreadyExists"
    !     Then  Change ReturnCode to 0 using MOFDIFY_STATE ==> Task ends with ENDED_OK
    ! 
    ! 
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    ! Result:
    !++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    !  In cases when all ERRORs are caused by "FileAlreadyExists", your task ends with "ENDED_OK - ended normally" and you'd get the POST Report:
    !
    !     2020-02-03 11:14:14 - U00020512 'Status_Text' modified from ' ' to 'count(ERRORs)=count(FileExists)'.
    !     2020-02-03 11:14:14 - U00020512 'Retcode' modified from '0000000000' to '0000000000'.
    !
    !  In cases when ERROR(s) occurred    AND    NOT all ERRORs are caused by "FileAlreadyExists", there will be no POST Report at all and you'd get the "original" Status
    !
    
    
    
    :set &ERROR_COUNTER#=0
    :set &ALREADY_EXISTS_COUNTER#=0
    
    :SET &HND# = PREP_PROCESS_REPORT(, , "REP", "* ERROR *", , )
    !:SET &HND# = PREP_PROCESS_REPORT(, , "REP", "*ERROR * incorrect transfer of*", , )
    :PROCESS &HND#
    : SET &ERROR_COUNTER# = ADD (&ERROR_COUNTER#,1)
    :ENDPROCESS
    :CLOSE_PROCESS &HND#
    
    :SET &HND# = PREP_PROCESS_REPORT(, , "REP", "*U00011404 File * already exists.*", , )
    :PROCESS &HND#
    : SET &ALREADY_EXISTS_COUNTER# = ADD (&ALREADY_EXISTS_COUNTER#,1)
    :ENDPROCESS
    :CLOSE_PROCESS &HND#
    
    :IF &ERROR_COUNTER# EQ &ALREADY_EXISTS_COUNTER#
    :  MODIFY_STATE STATUS_TEXT="count(ERRORs)=count(FileExists)"
    :  MODIFY_STATE RETCODE=0
    :ENDIF



  • 9.  RE: File Transfer Job (JOBF)

    Posted Feb 03, 2020 12:14 PM
    Edited by Kenneth Hutchins Feb 03, 2020 12:15 PM
    I tired this in the post process and it didnt work. 

    I was able to use the solution provided by Christoph.


  • 10.  RE: File Transfer Job (JOBF)

    Posted 22 days ago

    i believe we can use outputscan feature and change job status to success accordingly