Automic Automation Engine V21 FTP Validation and Transfer Script

[Michael Kennedy]

Hi Community,

I am very new to Automic but I've been trying to create a validation script to add to a workflow we have. This script should verify a file is larger than 27KB and that the file is actually there. Then SFTP from one Linux machine to another Windows machine. The reason for this is to solve an issue where our workflows are dying do to files not always existing or being empty.  

I believe I am pretty close with the below script but I found something saying I should be able to use FTP_CONNECT with the connection object name but I can't seem to get that syntax to work. Any advice would be very welcomed! 

!Defines the variables to validate the files
:DEFINE &remoteFilePathColl#, string
:DEFINE &localFilePathColl#, string
:DEFINE &collFileValid#, unsigned
:SET &remoteFilePathColl# = "[filepath]"
:SET &localFilePathColl# = "[filepath]"
:SET &remoteFileSizeColl# = 0
:SET &collFileValid# = 0

!Defines the variables to validate the files
:DEFINE &remoteFilePathHS#, string
:DEFINE &localFilePathHS#, string
:DEFINE &hsFileValid#, unsigned
:SET &remoteFilePathHS# = "[filepath]"
:SET &localFilePathHS# = "[filepath]"
:SET &remoteFileSizeHS# = 0
:SET &hsFileValid# = 0

!Defines the variables to validate the files
:DEFINE &localFilePathMat#, string
:DEFINE &matFileValid#, unsigned
:SET &remoteFilePathMat# = "[filepath]"
:SET &localFilePathMat# = "[filepath]"
:SET &remoteFileSizeMat# = 0
:SET &matFileValid# = 0
​
:PROCESS &remoteFilePathColl#
:PSET &remoteFileSizeColl# = :REMOTE_FILESIZE(&remoteFilePathColl#)
​
!Confirms the file size is greater than 27KB
:IF &remoteFileSizeColl# GT 27648
:SET &localDownloadStatusColl# = :FTP_DOWNLOAD(&remoteFilePathColl#, &localFilePathColl#)
​
!Confirms the file exists and prints a message relaying the state of the file
:IF &localDownloadStatusColl# EQ 0
:SET &collFileValid# = 1
:PRINT "File {&remoteFilePathColl#} exists and is larger than 27 KB."
​
! Add the unzip step here when collFileValid is equal to 1
:IF &collFileValid# EQ 1
:SET &unzipStatus# = :SHELL("unzip " &localFilePathColl# & "*.zip -o " &localFilePathColl#)
:IF &unzipStatus# EQ 0
:PRINT "Unzip successful for {&remoteFilePathColl#}."
:ELSE
:PRINT "Error unzipping {&remoteFilePathColl#}."
:ENDIF
:ENDIF
:ELSE
:PRINT "Error downloading file to {&localFilePathColl#}."
:ENDIF
:ELSE
:PRINT "File {&remoteFilePathColl#} does not exist or is not larger than 27 KB."
:ENDIF
:ENDPROCESS
​
:PROCESS &remoteFilePathHS#
:PSET &remoteFileSizeHS# = :REMOTE_FILESIZE(&remoteFilePathHS#)
​
!Confirms the file size is greater than 27KB
:IF &remoteFileSizeHS# GT 27648
:SET &localDownloadStatusHS# = :FTP_DOWNLOAD(&remoteFilePathHS#, &localFilePathHS#)
​
!Confirms the file exists and prints a message relaying the state of the file
:IF &localDownloadStatusHS# EQ 0
:SET &hsFileValid# = 1
:PRINT "File {&remoteFilePathHS#} exists and is larger than 27 KB."
​
! Add the unzip step here when hsFileValid is equal to 1
:IF &hsFileValid# EQ 1
:SET &unzipStatus# = :SHELL("unzip " &localFilePathHS# & "*.zip -o " &localFilePathHS#)
:IF &unzipStatus# EQ 0
:PRINT "Unzip successful for {&remoteFilePathHS#}."
:ELSE
:PRINT "Error unzipping {&remoteFilePathHS#}."
:ENDIF
:ENDIF
:ELSE
:PRINT "Error downloading file to {&localFilePathHS#}."
:ENDIF
:ELSE
:PRINT "File {&remoteFilePathHS#} does not exist or is not larger than 27 KB."
:ENDIF
:ENDPROCESS
​
:PROCESS &remoteFilePathMat#
:PSET &remoteFileSizeMat# = :REMOTE_FILESIZE(&remoteFilePathMat#)
​
!Confirms the file size is greater than 27KB
:IF &remoteFileSizeMat# GT 27648
:SET &localDownloadStatusMat# = :FTP_DOWNLOAD(&remoteFilePathMat#, &localFilePathMat#)
​
!Confirms the file exists and prints a message relaying the state of the file
:IF &localDownloadStatusMat# EQ 0
:SET &matFileValid# = 1
:PRINT "File {&remoteFilePathMat#} exists and is larger than 27 KB."
​
! Add the unzip step here when matFileValid is equal to 1
:IF &matFileValid# EQ 1
:SET &unzipStatus# = :SHELL("unzip " &localFilePathMat# & "*.zip -o " &localFilePathMat#)
:IF &unzipStatus# EQ 0
:PRINT "Unzip successful for {&remoteFilePathMat#}."
:ELSE
:PRINT "Error unzipping {&remoteFilePathMat#}."
:ENDIF
:ENDIF
:ELSE
:PRINT "Error downloading file to {&localFilePathMat#}."
:ENDIF
:ELSE
:PRINT "File {&remoteFilePathMat#} does not exist or is not larger than 27 KB."
:ENDIF
:ENDPROCESS
​

[Christoph Rekers]

Hi @Michael Kennedy,

Your script will not work for at least 2 reasons:

:PROCESS always requires a data sequence - supplied in most cases by a PREP_PROCESS-function, e.g.:

:set &HND# = PREP_PROCESS_xxx()
:PROCESS &HND#
...
:ENDPROCESS &HND#

Secondly:  a string like this 
:SET &unzipStatus# = :SHELL("unzip " &localFilePathMat# & "*.zip -o " &localFilePathMat#)
needs to be set into quotes, otherwise the Automic Server tries to interpret the colon as his own script command - which it isn't.

Cheers

Christoph 

[Michael Kennedy]

I have adjusted my script to include these items but still not able to start. Does anyone have any other suggestions or any training that goes beyond the high-level. 

[Christoph Rekers]

Hi @Michael Kennedy
I can't see your adjustments...

Cheers

Christoph 

[Michael Kennedy]

@Christoph Rekers Sorry I forgot to attach the new updates,

:set &HND# = PREP_PROCESS_MULTIFILE_vERIFY_UNZIP
:PROCESS &HND#

!Defines the variables to validate the slate_undergrad_dom_coll_transc_export_*.zip files
:DEFINE &remoteFilePathColl#, string
:DEFINE &localFilePathColl#, string
:DEFINE &collFileValid#, unsigned
:SET &remoteFilePathColl# = "/tst_share/kennedytest/slate_undergrad_dom_coll_transc_export_*.zip"
:SET &localFilePathColl# = "/tst_share/kennedytest/unzipped"
:SET &remoteFileSizeColl# = 0
:SET &collFileValid# = 0

!Defines the variables to validate the /apps/prod/slate_undergrad_dom_hs_transc_export_*.zip files
:DEFINE &remoteFilePathHS#, string
:DEFINE &localFilePathHS#, string
:DEFINE &hsFileValid#, unsigned
:SET &remoteFilePathHS# = "/tst_share/kennedytest/slate_undergrad_dom_hs_transc_export_*.zip"
:SET &localFilePathHS# = "/tst_share/kennedytest/unzipped"
:SET &remoteFileSizeHS# = 0
:SET &hsFileValid# = 0

!Defines the variables to validate the slate_undergrad_dom_mat_export_*.zip files
:DEFINE &remoteFilePathMat#, string
:DEFINE &localFilePathMat#, string
:DEFINE &matFileValid#, unsigned
:SET &remoteFilePathMat# = "/tst_share/kennedytest/slate_undergrad_dom_mat_export_*.zip"
:SET &localFilePathMat# = "/tst_share/kennedytest/unzipped"
:SET &remoteFileSizeMat# = 0
:SET &matFileValid# = 0

:PROCESS &remoteFilePathColl#
:PSET &remoteFileSizeColl# = :REMOTE_FILESIZE(&remoteFilePathColl#)

!Confirms the file size is greater than 27KB
:IF &remoteFileSizeColl# GT 27648
:SET &localDownloadStatusColl# = :FTP_DOWNLOAD(&remoteFilePathColl#, &localFilePathColl#)

!Confirms the file exists and prints a message relaying the state of the file
:IF &localDownloadStatusColl# EQ 0
:SET &collFileValid# = 1
:PRINT "File {&remoteFilePathColl#} exists and is larger than 27 KB."

! Add the unzip step here when collFileValid is equal to 1
:IF &collFileValid# EQ 1
:SET &unzipStatus# = ':SHELL("unzip " &localFilePathColl# & "*.zip -o " &localFilePathColl#)'
:IF &unzipStatus# EQ 0
:PRINT "Unzip successful for {&remoteFilePathColl#}."
:ELSE
:PRINT "Error unzipping {&remoteFilePathColl#}."
:ENDIF
:ENDIF
:ELSE
:PRINT "Error downloading file to {&localFilePathColl#}."
:ENDIF
:ELSE
:PRINT "File {&remoteFilePathColl#} does not exist or is not larger than 27 KB."
:ENDIF
:ENDPROCESS

:PROCESS &remoteFilePathHS#
:PSET &remoteFileSizeHS# = :REMOTE_FILESIZE(&remoteFilePathHS#)

!Confirms the file size is greater than 27KB
:IF &remoteFileSizeHS# GT 27648
:SET &localDownloadStatusHS# = :FTP_DOWNLOAD(&remoteFilePathHS#, &localFilePathHS#)

!Confirms the file exists and prints a message relaying the state of the file
:IF &localDownloadStatusHS# EQ 0
:SET &hsFileValid# = 1
:PRINT "File {&remoteFilePathHS#} exists and is larger than 27 KB."

! Add the unzip step here when hsFileValid is equal to 1
:IF &hsFileValid# EQ 1
:SET &unzipStatus# = ':SHELL("unzip " &localFilePathHS# & "*.zip -o " &localFilePathHS#)'
:IF &unzipStatus# EQ 0
:PRINT "Unzip successful for {&remoteFilePathHS#}."
:ELSE
:PRINT "Error unzipping {&remoteFilePathHS#}."
:ENDIF
:ENDIF
:ELSE
:PRINT "Error downloading file to {&localFilePathHS#}."
:ENDIF
:ELSE
:PRINT "File {&remoteFilePathHS#} does not exist or is not larger than 27 KB."
:ENDIF
:ENDPROCESS

:PROCESS &remoteFilePathMat#
:PSET &remoteFileSizeMat# = :REMOTE_FILESIZE(&remoteFilePathMat#)

!Confirms the file size is greater than 27KB
:IF &remoteFileSizeMat# GT 27648
:SET &localDownloadStatusMat# = :FTP_DOWNLOAD(&remoteFilePathMat#, &localFilePathMat#)

!Confirms the file exists and prints a message relaying the state of the file
:IF &localDownloadStatusMat# EQ 0
:SET &matFileValid# = 1
:PRINT "File {&remoteFilePathMat#} exists and is larger than 27 KB."

! Add the unzip step here when matFileValid is equal to 1
:IF &matFileValid# EQ 1
:SET &unzipStatus# = ':SHELL("unzip " &localFilePathMat# & "*.zip -o " &localFilePathMat#)'
:IF &unzipStatus# EQ 0
:PRINT "Unzip successful for {&remoteFilePathMat#}."
:ELSE
:PRINT "Error unzipping {&remoteFilePathMat#}."
:ENDIF
:ENDIF
:ELSE
:PRINT "Error downloading file to {&localFilePathMat#}."
:ENDIF
:ELSE
:PRINT "File {&remoteFilePathMat#} does not exist or is not larger than 27 KB."
:ENDIF
:ENDPROCESS

:ENDPROCESS &HND#

[Markus Embacher]

Hi Michael,

please let us know what you are trying to achieve with your script. As Christoph pointed out :SHELL and :REMOTE_FILESIZE are not valid Automic script commands, but because they start with a colon Automic tries to interpret them. That will fail. Further you are using :SHELL and :REMOTE_FILESIZE like functions and you try to capture the return code. That's not going to work for multiple reasons: Automic does not know those functions and if those are functions of your FTP software then Automic would not execute them on the ftp server.

Next is the PREP_PROCESS loop. There is not PREP_PROCESS_MULTIFILE in Automic. GET_PROCESS_LINE is also missing. 

Please explore FILELIST Vara objects and PREP_PROCESS or BACKEND VARAs. Or code a shell script in a UNIX job (or powershell in and windows job) to do the checking.

Regards, Markus

[Michael Kennedy]

@Markus Embacher We are having issues with one of our workflows failing. This workflow has 4 overall steps RAFTP, UNZIP, SFTP, and ARCH. The step we are failing on is the UNZIP due to two reasons.

First we don't have a file at all or the file is empty. We have contacted the party who creates the file and they should be suppressing the empty files in the future. The second is because this is looking for potentially 3 different files which may or may not available for that day. If that file isn't available that day it fails. I am trying to create a way to verify the file size is greater than 27 KB (ensuring the file isn't empty), and verify the file is there before trying to unzip the file. With the ultimate goal of the workflow being able to complete successfully and avoid daily monitoring/manual intervention of the workflow. 

I believe I adjusted the shell command to include quotes as suggested by Christoph unless he meant I should put quotes around the entire command and not just the shell portion. The red ' is my adjustment, :SET &unzipStatus# = ':SHELL("unzip " &localFilePathHS# & "*.zip -o " &localFilePathHS#)' 

Ultimately I am hoping solving this problem will help expand my understanding and ability to automate our daily requirements. I suspect it will be with some form of scripting but I am unsure of the optimal method to use within Automic.

Thanks,
Michael