I need to monitor arrival of files in AWS S3 bucket storage.

[Suren_V]

I need to monitor arrival of files in AWS S3 bucket storage.

 

IF I mount the S3 bucket to linux server where autoys agent is installed, whether i can able to run the file watcher jobs in S3 mounted file system

 

In S3 file system linux commands wont work as it is a different type of file system.

 

Still autosys agent can monitor arrival of file in S3 storage?

Does anyone implemented it ? Please share your thoughts?

[Legacy User]

Hello

If these files are on an NFS mounted file system, you can configure the agent to run file triggers as separate processes, which let the agent act as follows:

• Scan files using a specified user ID on a UNIX or Linux system. As a result, the agent can scan NFS file systems that the local root user does not have access to.
• Resolve file names to be scanned on both UNIX and Windows.
• Steps to configure the System Agent are described Here

   

Does it help?

Let us know

 

Thanks and regards

Jean Paul

[Chandru.Venkataraman]

Hi Suren,

 

Yes, Filewatcher (FW) can watch for files on AWS S3 bucket mounted on Linux.

While I do not know what tool your shop might be using to mount S3 on Linux, I have tested using S3FS.

You need to sort out the permissions (ACLs) on the Linux mount point, though. For example, if you mount S3 bucket as one user on Linux (say root) and the FW job runs as a different user (say awsuser), then you might get permission denied errors.

Here is a simple run down of what I did to test:

1. Create bucket in AWS S3

2. Mount the S3 bucket on Linux using the S3FS tool.

3. Ensure the mountpoint is readable:

3. Define a FW job:

:~> jr AWS_S3_Bucket_Watch -q

 

/* ----------------- AWS_S3_Bucket_Watch ----------------- */

 

insert_job: AWS_S3_Bucket_Watch   job_type: FW
machine: I3914
owner: root
permission:
date_conditions: 0
alarm_if_fail: 0
watch_file: "/mnt/s3/chandru/folder1/junk.txt"
watch_interval: 90

 

4. Execute the job:

 

You may notice that my job owner is root, its because my mount is owned by root. In realworld, you may not be able to run jobs as root user on *NIX. However, I believe tools available to mount S3 on Linux allow one to mount the filesystem as non-root user and you can manage permissions/ACLs within AWS as well. Please check with your Linux Admin on this.

 

Disclaimer: S3FS was used in this example only to prove the point that FW jobs can monitor files in S3 bucket when mounted on Linux/UNIX. I am NOT recommending S3FS or similar tools. And, do NOT attempt to run job as root user on *NIX systems. The foregoing example, again, is only to prove the technical possibility. I am NOT responsible for any damage you or your business may incur while trying to follow the instructions in my response.

 

 

Thanks,

Chandru

[Steve Carrobis]

Chandru

 

when you give an example its always best to keep it generic. otherwise .. people will say. well so and so said it can only be done this way .,\

 

as it is you didnt set profile: 

so that env variables could be used 

 

insert_job: GRP-MyFW   job_type: FW
machine: machinename
owner: userid
permission:
date_conditions: 0
alarm_if_fail: 0
watch_file: "$FileName"
watch_interval: 90

watch_file_min_size: 0 or greater 

profile: /path/Profile.sh

 

 

just my 3 cents 

 

Steve C.

[Suren_V]

Hi Chandru,

 

Thanks for your response. We have considered S3FS option earlier, however confused whether Autosys agent can monitor files since file system type is different.

 

Now I could see autosys can monitor files in S3 bucket, Great..:) 

[Steve Carrobis]

Suren,

 

as long as the file system is visible and the file is visible FW/FT should work . if it doesn't then its a bug :-)

prior to R11 it never mattered and it still shouldnt matter 

 

Good luck 

 

Steve C.