AutoSys Workload Automation

Expand all | Collapse all

check if a file exists in a ftp server before dowloading it

Jump to Best Answer
  • 1.  check if a file exists in a ftp server before dowloading it

    Posted 05-02-2018 04:46 PM

    I have a SFTP remote server which I have to inspect every now and then for a particular file to appear before attempt to download it (File Watch funcionality)

     

    I can not install a WA Agent on this server, and the only access I have is using an FTP user + password provided.

     

    What are your suggestions to solve this?



  • 2.  Re: check if a file exists in a ftp server before dowloading it

    Posted 05-03-2018 08:23 AM

    We had a similar situation.  However, in our case, the developer had to resort to doing this almost exclusively in scripts.  The result is a set of Unix jobs and tasks.  The first job logs in and checks for the existence of the file.  If the file exists, it exits with a return code of 0.  So it completes, and the next set of jobs runs.  They connect to the server and download the file.  If the file does not exist, the initial job exits with a 99, and releases a task that forces the other jobs complete.  We have this process set up to repeat hourly throughout the day using a template.  We're doing this in ESP.  I can add sample code if you like.



  • 3.  Re: check if a file exists in a ftp server before dowloading it

    Posted 05-03-2018 11:21 AM

    Hi, jonathan_calloway,

     

    I was afraid we'll forced to do so. Could you be so kind to share the sample code you offered?



  • 4.  Re: check if a file exists in a ftp server before dowloading it

    Posted 05-04-2018 07:50 AM

    Hi dennis.naranjo,

     

    Here is CLANG from our proc.  I don't have any of the code from the scripts involved:

     

    TEMPLATE DSNP270_HOURLY (1,OFFSET)                     
                                                           
      UNIX_JOB ZN470X02.%OFFSET                            
        EXITCODE 0 SUCCESS                                 
        EXITCODE 99 SUCCESS                                
        RUN WORKDAYS                                       
        NORUN TNSTATEHOLIDAY                               
        DELAYSUB 06:00 PLUS %OFFSET HOURS                  
        AGENT <agent>
        USER <user>
        SCRIPTNAME <script>
        ENVAR JOBNAME=%ESPAPJOB                            
        RELEASE ADD(BYPASS_DSNP270) COND(RC(99))           
        RELEASE ADD(ZN470Y01.%OFFSET) COND(RC(0))          
      ENDJOB                                               

     

      UNIX_JOB ZN470Y01.%OFFSET                                      
        RUN WORKDAYS                                                 
        NORUN TNSTATEHOLIDAY                                         
        AGENT <agent>
        USER  <user>
       SCRIPTNAME <user>        
       ARGS %USER1                                                   
       ENVAR JOBNAME=%ESPAPJOB                                       
      ENDJOB                                                         
                                                                     
      JOB BYPASS_DSNP270 LINK PROCESS                                
        RUN WORKDAYS                                                 
        NORUN TNSTATEHOLIDAY                                         
        ESPNOMSG AJ ZN470Y01.%OFFSET BYPASS APPL(%ESPAPPL..%ESPAPGEN)
      ENDJOB                                                         
                                                                     
    ENDTEMPL                                                         
     DSNP270_HOURLY 0                                                
     DSNP270_HOURLY 1                                                
     DSNP270_HOURLY 2                                                
     DSNP270_HOURLY 3                                                
     DSNP270_HOURLY 4       

    DSNP270_HOURLY 5 
    DSNP270_HOURLY 6 
    DSNP270_HOURLY 7 
    DSNP270_HOURLY 8 
    DSNP270_HOURLY 9 
    DSNP270_HOURLY 10
    DSNP270_HOURLY 11
    DSNP270_HOURLY 12
    DSNP270_HOURLY 13
    DSNP270_HOURLY 14                                          

     

    Explanation: If ZN470X02 ends with a return code of 99, then it releases a task (BYPASS_DSNP270) which bypasses ZN470Y01.  This process uses a template that ensure this occurs every hour for 15 hours.  You can make this more complex by using AFTER ADD statements to add other jobs to the stream.

     

    Hope this helps!



  • 5.  Re: check if a file exists in a ftp server before dowloading it

    Posted 05-04-2018 01:07 PM

    Thank you, Jonathan.



  • 6.  Re: check if a file exists in a ftp server before dowloading it
    Best Answer

    Posted 05-12-2018 11:19 PM

    cURL could be used.

     

    $ curl -u ftpuser:ftppass -O ftp://ftp_server/public/

     

    cURL will list all the files and directories under the given ftp URL. One could use the -n or --netrc option to cause cURL to fetch the FTP user name and password from the ~/.netrc file

     

    Man page for cURL is here

    curl - How To Use 

     

    I hope this helps!

     

    Cheers,

    Chandru



  • 7.  Re: check if a file exists in a ftp server before dowloading it

    Posted 05-04-2018 07:55 AM

    Can the SFTP remote server PUT the file on your server with an Agent?




  • 8.  Re: check if a file exists in a ftp server before dowloading it

    Posted 05-04-2018 08:40 AM

    Hi,

    The CA Agent can only act as an SFTP client.  The Agent can download/upload the file from your remote SFTP server.  If you want remote server to login to the host where agent is then you will need SSH/SFTP service.    

    The CA Agent can however act as an FTP client and server.  You can start FTP server and use FTPS or FTP over SSL for better security than plain FTP.

     

    Thank you,

    Nitin Pande

    CA Technologies



  • 9.  Re: check if a file exists in a ftp server before dowloading it

    Posted 06-18-2018 01:47 PM

    Dennis, 

     

    It looks like you got a few suggestions here -- did they help you? Or do you still need help with this?



  • 10.  Re: check if a file exists in a ftp server before dowloading it

    Posted 06-18-2018 02:04 PM

    Lenn,

     

    'Till now, the CURL option sound better than the restm but I'm currently flooded with work so I'm not able to test it right now...



  • 11.  Re: check if a file exists in a ftp server before dowloading it

    Posted 06-18-2018 02:11 PM

    Once you have a chance to test it, please come back and let us know!