Select multiple files (with the same namespace) for File Transfer on Windows

[Andre Eymann]

Customer requirement
A Windows application is constantly creating output files. The total amount (may differ) and the names have (partly) the same namespace. The namespace identifies the "set of files" as a unique customer sequence.

Examples of the filenames:
VC_180223_092213_customersessionid_2_6572151.mp4
VC_180223_092213_customersessionid_2_6572151-NewIdCardDE_Back.png
VC_180223_092213_customersessionid_2_6572151-NewIdCardDE_Covering.png
VC_180223_092213_customersessionid_2_6572151-SessionHistory.xml

The string above "180223_092213_customersessionid_2_6572151" (part of the filename) is unique any may be used to identify all files with that string.

Automic should identify all files with the given namespace, wait for the files to be stable (e.g. 3 minutes) and then tranfers those files with File Transfer to a target while deleting on the source location. The files in the target then should be processed with a batch program.

Question
How can Automic manage to identify all files "of a set" to transfer them into the target?

Any help would be great! Thanks in advance.

[Anon Anon]

Hi Andre,

Thats a bit tricky because you do not know the amount of files AND when they are finished writing from your application.

Basically I would do a check for the amount of files matching the "key phrase *180223_092213_customersessionid_2_6572151* AND the size of them.

Possibly in a VARA object - for every KEY phrase one VARA object which holds the filenames as key and the file size in every VAL column (this means 5 checks are allowed)

if the file size AND amount of files does not change since last check the JOBF can be started with *180223_092213_customersessionid_2_6572151* as wildcard.

Yes its a lot of scripting.

And yes its a lot to think about in case the whole thing crashes and should resume or re-ran....

cheers, Wolfgang

[Carsten Schmitz]

Hi.

You can make a file event like this:

https://us.v-cdn.net/5019921/uploads/editor/i4/eg852r6sya3n.png" width="921">
From some quick tests, it behaves mostly as you'd expect:

Zero files matching wildcard: event does not trigger (but prints a silly error message into log and report, because hey, it's still Automic after all)

One more more matching files: event only triggers if the newly arrived files are stable over the set peroid. Adding new files that match the wildcard, and changing file sizes of any file appears *) to reset the event so it only triggers if all files matching the wildcard are stable.


*) this is all the result of some quick testing on my part, I suggest you test this more extensively yourself before using it for anything important.

Or, you do the whole magic in shell script or Powershell instead (md5sum over the files), and just write a trigger file once your shell script or Powershell thing determines file stability, then just react to that trigger file with a more simplified Automic event. Much more flexibility that way.

Hth,
Carsten

[Andre Eymann]

Wolfgang Brueckler said:

Hi Andre,

Thats a bit tricky because you do not know the amount of files AND when they are finished writing from your application.

Basically I would do a check for the amount of files matching the "key phrase *180223_092213_customersessionid_2_6572151* AND the size of them.

Possibly in a VARA object - for every KEY phrase one VARA object which holds the filenames as key and the file size in every VAL column (this means 5 checks are allowed)

if the file size AND amount of files does not change since last check the JOBF can be started with *180223_092213_customersessionid_2_6572151* as wildcard.

Yes its a lot of scripting.

And yes its a lot to think about in case the whole thing crashes and should resume or re-ran....

cheers, Wolfgang

Thanks Wolfgang for your suggestions!

[Andre Eymann]

Carsten Schmitz said:

Hi.

You can make a file event like this:

https://us.v-cdn.net/5019921/uploads/editor/i4/eg852r6sya3n.png" width="921">
From some quick tests, it behaves mostly as you'd expect:

Zero files matching wildcard: event does not trigger (but prints a silly error message into log and report, because hey, it's still Automic after all)

One more more matching files: event only triggers if the newly arrived files are stable over the set peroid. Adding new files that match the wildcard, and changing file sizes of any file appears *) to reset the event so it only triggers if all files matching the wildcard are stable.


*) this is all the result of some quick testing on my part, I suggest you test this more extensively yourself before using it for anything important.

Or, you do the whole magic in shell script or Powershell instead (md5sum over the files), and just write a trigger file once your shell script or Powershell thing determines file stability, then just react to that trigger file with a more simplified Automic event. Much more flexibility that way.

Hth,
Carsten

Hallo Carsten,

thanks a lot for your input! That basically looks good to me. This may work so far und seems to be a valid approach. Your ideas using Windows PowerShell on the source side is something i already discussed with the customer. My recommendation was to use PowerShell to identify and move all stable files of a set to different directory for Automic. Or even better deliver the whole set of files in an archive (ZIP) so we know that these files are a compete sequence of data.

But by now i would implement your idea and try to test it out.

Cheers, André