Using the File Transfer object

[Anon Anon]

Hi All,

My company is very interested in using the File Transfer Object as a managed file transfer, we were under the impression that it is a very robust tool in that it can start where it stopped if the transfer goes down. That's definitely useful for us, but I was wondering if it can do a couple of other things that I haven't figured out by playing with it.

We're trying to get it so that it will:

1. Check to see if there are new files in a specific directory and ONLY copy the new ones (or ones that have been modified recently)
2. Delete files from the destination directory if the files have been deleted from the source directory (like mirroring in robocopy)
3. Replace the destination files if they already exist (this is a basic overwrite which I saw it should have that functionality)

Anyway, #3 I'm 95% sure it can do, I just need to test it out a bit but I'm not sure about #s 1 & 2 and was wondering if anyone had ever tried to use it for that purpose before (and if so, how did they manage it). We're ideally trying to do this not using scripting (just using the functionality of the transfer object).

On a side note, the Automic community has been ludicrously helpful and I've learned tons from posting questions here and reading through other threads so thank you for being awesome.

-Bertie

[Anon Anon]

We have similar requirements but needed to use scripting as the base attributes of FT objects are insufficient to handle it alone.

If you would like to know our basic approach then I'd be happy to describe it. 

[Anon Anon]

I'm not sure if you got my email...I don't know if replying to the email notifications from this site will go to the person in question. Anyway, what I said was that the higher powers here want to do it all through an FT object, I'm relatively confident that's impossible so I'll have to convince them of it.

I have some ideas of how to do it in script form but I'd definitely be interested in how you've managed to do it.

[Anon Anon]

Bertie:

Have not received an email or notification of a message, I also checked my spam folder just to be certain.  I don’t see anything in my profile that should prevent my receiving emails and I have gotten some from other forum members. 

We use a two-step approach.  

Step 1:  This, for us, is usually a Script object.  It performs your steps 1 & 2 using Script statements like PREP_PROCESS_FILENAME to list the files in question and GET_FILESYSTEM to acquire the needed attributes and decide what needs to be done.  If files need to be deleted we will usually use a PREP_PROCESS statement.  

Once we decide that a transfer needs to occur we use the PSET function to pass the needed attributes to a “generic” File Transfer object.  Those attributes are the required ones that are unique to this particular file such as Login, To/From Hosts, To/From File names, etc.  Then the ACTIVATE_UC_OBJECT with the PASS_VALUES parameter is used to start the File Transfer object, once for each qualifying file.  

Step 2:  The Process tab of the File Transfer contains the PUT_ATT statements that sets the passed attributes from Step 1 and the “miracle” occurs as it performs your step 3.

As is usually the case with this type of coding that devil is in the details.  However, we have many executions per day of this type of processing and it works with about as much absolute certainty as we have come to expect from UC4.  The transfers are mostly between Windows and Unix hosts.  I might add that we are on V8.

Let me know if you need any more specific information.

[Legacy User]

If you need to sync from source A "/tmp/srcDir" to destination B "/tmp/dstDir" but you don't want to overwrite files that already exist in /tmp/dstDir (because of big files for example):

1. Create filelist VARA (F1) on A with /tmp/srcDir
2. Create filelist VARA (F2) on B with /tmp/dstDir
3. Create multi VARA (M1) with F1 & F2 and "difference" method

You now have all the files within the M1 VARA that exist on F1 but not on F2.

Now you might use a foreach workflow onto the M1 VARA to copy files from A to B that are not yet existing on B.

Be warned - it won't transfer files with different content, it will just add files that are not at already in the destination directory. Oh and.. I just figured out a new feature request . Having filelist VARAs to show more information on the files. If we now have more values available, this could be really cool.

[Anon Anon]

Mark_Hadler_430 - I don't really understand your step 2, why would using PUT_ATT with the pass values overwrite the destination files if they exist already?

But your step 1 is most helpful, one quick question about that, though, are you using this script inside a workflow? I ask because you say you use PSET, but I thought that only stored variables across jobs when in a workflow...or is that needed in order to use the PASS_VALUES key?

[Anon Anon]

joel_wiesmann_automic - I'm not sure what you mean by "using the difference method"? What is the difference method...maybe I'm too green to UC4 to get that, haha.

Also, the first variable you mention there, does it contain just the path to the correct directory? Or is it a list of the files contained within the directory? If it's just a path, I don't understand how UC4 would then see all the files contained in the directory qualified by the path.

[Legacy User]

6979faefbacb4dee06d02fb1aad9714d if you create a VARA of type "MULTI" and open it, it should spring in your eye what I meant :). The VARA can be used in 3 ways, one of them shows the difference between 2 specified VARAs (in my example VARA F1 and F2). The F1 / F2 VARA objects contain Server, Login & Path. The path might also contain a file-filter (*.txt). The MULTI object compares the VARAs contents, so basically it compares the directory contents of both servers. Best is to just try it on your UC4 system and things will get clear.

[Anon Anon]

Bertie:

My step 2 uses a file transfer object that does not have the mentioned attributes set, it just contains the default ones from the template.  For example it has a Host of <UNIX> and a File of ScrFile.  Since this single object is going to be executed multiple times, once for each file, it will need to have its attributes set to the required values each time it is activated.  It would be up to you to either pass the FT_EXISTING_DST_FILE value along with the other attribute values or, if you always want to overwrite then you could just as well set that value in the object so that it will already have the File exists attribute.

The Step 1 is, in our environment, usually either in a Process Flow or in the ! Process tab of time Event.  Virtually all of our scheduled tasks execute from within a flow.  As you appear to know, PSET has a few potential issues of what it does to script variable values that are passed as they can affect them both up and down the flow.  For us, since we choose the variable names knowing this behavior it has not been an issue.  The PSET is not limited to just job objects but any task in the flow, including sub flows.  If PSET is an issue for then you could consider creating a "temporary" Variable object that contains the attribute values and have the File Transfer read it to perform the PUT_ATT statements.

Though I didn't mention it, the file transfer (Step 2) itself executes as a single task in its own process flow; in reality we activate the flow and not the transfer.  We do this for a couple of reasons.  The main one being that we can easily avail ourselves of the Result Properties and use the Else Repeat option.  The majority of the logic of these types of processes exists to handle exceptions and failures.  Like I said, it's that devil!   >:)

[Anon Anon]

joel_wiesmann_automic Hi Joel, I was just trying out your method because once I understood it, it seemed like a good way to solve one part of what I need to get done, but as I was just testing the filename vara, I noticed it only seems to import 426 file names from the directory.

Is the limit of how many rows this vara can have really that low? That seems crazy if it is, this directory has about 1500 files in it.

[Anon Anon]

I wish I could delete that...haha, I was being a dummy. Ignore it!

[Anon Anon]

joel_wiesmann_automic This time is a more legitimate question! Do you know of any way to modify the working directory for a filelist variable?

Because in this script, it will need to look at one specific directory, compare to another directory and then move onto a different directory. So like on the first iteration, it will look at \\server\location1\*.jpg and then on the next iteration, it would have to look at \\server\location2\*.jpg and so on up to about 200 locations.

I have a list of locations, obviously, so it would be easy to read that list into UC4 but I'm not sure it would be possible, in the script to modify the directory of the filelist vara.

Let me know if I need to clarify anything there!
Thanks,
Bertie

[Anon Anon]

@Mark Hadler Hey Mark, I'm attempting to work with your advice on this to have it check to see if files have been modified and if they've been modified recently, have it perform the FT, but using GET_FILESYSTEM doesn't seem to give me any information about the last modification date on a file. Maybe I'm missing where you said your system will do that (and so was looking at the wrong place).

[Anon Anon]

Hopefully I didn't say or mean to imply that everything that you may need to know about a file is available via a Script function.  Sadly this particular information is not available with GET_FILESYSTEM function; maybe that is a subject of a Product Idea.

What I have done when I require this type of information is to use PREP_PROCESS to issue the appropriate operating system command to display that information and process the resulting data sequence extracting what is needed.

The only functionality regarding last modification of a file that I am aware of is in a File Event's File attribute regarding the within: parameter.  Though I have never used this particular method it appears that that information is available with a GET_EVENT_INFO function.  It seems likely that attempting to use an Event for your needs is rather impractical.

[Anon Anon]

Oh okay, yeah, my plan was to use PREP_PROCESS to get the modified date, let me give that a try now.

Thanks, Mark!

[Legacy User]

Hi @"Bertie Wright" ! I'm in vacation so out of my head I can't say whether it is possible to change the filelist backend setting but I tend to say no.

However what still can be done is to use static vara instead of filelist varas. By using static varas you need to set the logic in place that they contain the filenames. This isn't so difficult to achieve. I might give an example once I'm back and have access to the AE setup (end of month).