XCOM Data Transport

 View Only

How to Use XCOM Data Transport with AWS Mountpoint for Sending Files to S3 Storage

By Shivaramakrishna Chakravarthula posted Oct 30, 2024 09:00 AM

  

In today's world of cloud storage, Amazon S3 (Simple Storage Service) is one of the popular solutions for storing large amounts of data securely, affordably, and reliably. Interacting with S3 storage can be done via several methods.  One of the most efficient ways to move files seamlessly from local storage to S3 storage is through AWS Mountpoint for S3, a lightweight, high-performance file system client designed to make S3 buckets accessible like regular local directories. 


As an XCOM Data Transport for Linux user, if you want to interact with S3 storage to send/receive files, you can leverage software like AWS mountpoint. This blog post will guide you through installing AWS Mountpoint and using it with XCOM Data Transport to transfer files to an S3 bucket.


 


What Is AWS Mountpoint?


AWS Mountpoint for S3 is an open-source, POSIX-like file system that makes it easy to map Amazon S3 storage to a local directory on your machine, allowing you to read and write files using native Linux commands. This makes interacting with your S3 bucket feel as if you're working with a native file system on your local Linux server.


 


Key features of AWS Mountpoint:




  • Seamless access to your S3 buckets like local directories




  • High-performance file transfer




  • POSIX-like file handling for large-scale data movements




 


This blog provides step-by-step instructions for installing AWS Mountpoint and using it with XCOM Data Transport on a Redhat Linux Server. Though we have used Redhat Linux in this blog, AWS Mountpoint can be installed on any other Linux distribution. For more details, refer to https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint-installation.html


 


Prerequisites




  • AWS account with access to S3




  • AWS CLI installed and configured on your machine




  • RHEL 8 or later




  • A valid subscription or access to Red Hat repositories for installing necessary packages




  


Step 1: Install AWS Mountpoint


AWS Mountpoint software is available as a precompiled binary or it can be built from the source.


Here's how to install and configure it on Redhat Linux.



  • Download the latest release that is appropriate for your architecture (for example, x86_64) and install it:


wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
yum install ./mount-s3.rpm



  • Verify the installation:  



mount-s3 --version


You should see output similar to the following:



mount-s3 1.5.0

 


Step 2: Configure AWS CLI


If you haven't already configured the AWS CLI, you will need to set it up before proceeding.


 



aws configure

 


You'll need to provide your:




  • AWS Access Key ID




  • AWS Secret Access Key




  • Default region name (for example, us-west-1)




  • Default output format (for example, json)




  


Step 3: Mount the S3 Bucket Locally


 


Now that AWS Mountpoint is installed, mount an S3 bucket as a directory on your local machine.


 




  • Create a directory that will serve as your mount point:




mkdir /tmp/s3-bucket

 




  • Use the mount-s3 command to map your S3 bucket to the folder. Replace my-bucket with the name of your S3 bucket:




mount-s3 --allow-delete --allow-overwrite --allow-other my-bucket /tmp/s3-bucket

Note: Ensure the directory is empty before mounting the bucket. Otherwise, mount-s3 fails to mount the S3 bucket. In case of a system reboot, You need to mount the file system again.         



At this point, you should see the contents of your S3 bucket inside the /tmp/s3-bucket folder. You can interact with these files as if they were local files within the limitations of Mountpoint software.


  


Step 4: Transfer Files to the S3 Bucket Using XCOM Data Transport


 


With the bucket mounted, you can transfer files from any of your servers with XCOM installed on it to S3 storage using XCOM transfer jobs.


 


To upload files from your local system to the S3 bucket, use the mounted  /tmp/s3-bucket as your remote file location. The local system could be any of the platforms supported by XCOM. 


For example, 




  • To upload a file named C:\temp\testfile.txt from a Windows Server to an S3 bucket that is mounted to /tmp/s3-bucket/ on a remote XCOM Linux server with IP address 192.168.1.2, create an XCOM configuration file named sendfiletos3.cnf with the following contents: 




TRANSFER_TYPE=3
LOCAL_FILE=C:\temp\testfile.txt
REMOTE_FILE=/tmp/s3-bucket/testfile.txt
REMOTE_SYSTEM=192.168.1.2
FILE_OPTION=CREATE
USERID=<userid>
PASSWORD=<password>

Note: You can not use REPLACE or APPEND options with the FILE_OPTION parameter when transferring to a mounted S3 bucket. 



You can submit the transfer job using the command: 


xcomtcp -f sendfiletos3.cnf

 




  • To download and receive the testfile.txt file that is stored under the S3 bucket mounted to /tmp/s3-bucket on a remote XCOM Linux server with IP address 192.168.1.2 and optionally replace the file on the local Windows server if it already exists, create an XCOM configuration file named receivefilefroms3.cnf with the following contents: 




 


TRANSFER_TYPE=4
LOCAL_FILE_RF=C:\temp\testfile.txt
REMOTE_FILE_RF=/tmp/s3-bucket/testfile.txt
REMOTE_SYSTEM_RF=192.168.1.2
FILE_OPTION_RF=REPLACE
USERID=<userid>
PASSWORD=<password>

 


You can submit the transfer job using the command: 


xcomtcp -f receivefilefroms3.cnf

 


(Optional) Step 5: Unmount the S3 Bucket


 


After you've finished transferring files, unmount the S3 bucket to ensure no more data is written to it unintentionally.


 


To unmount the S3 bucket:


sudo umount /tmp/s3-bucket

 

Troubleshooting



If you encounter issues while using AWS Mountpoint on Red Hat, here are a few common troubleshooting tips:




  • Permission Issues: If you’re unable to mount the bucket, check that the user has the necessary permissions to access the S3 bucket.




  • AWS CLI Configuration: Ensure that the AWS CLI is configured with the correct credentials, especially if your system uses multiple profiles.



  • Issue the below command to check the messages related to mount-s3 in SYSLOG. 


journalctl -e SYSLOG_IDENTIFIER=mount-s3


For Mountpoint logging details, refer to  https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint-troubleshooting.html.



Conclusion


 


AWS Mountpoint makes it easier to interact with Amazon S3 by enabling applications like XCOM Data Transport to access S3 buckets directly from the Linux file system. Whether you're dealing with large-scale data transfers or automating backups, Mountpoint offers a high-performance and user-friendly way to access your S3 storage.


 


By following the steps above, you can now mount your S3 buckets, transfer files using XCOM seamlessly, and integrate S3 storage into your existing XCOM Data Transport transfer jobs.


 


Key takeaways:




  • AWS Mountpoint provides native-like file system access to S3 buckets on Red Hat Linux.




  • It allows seamless file transfers using existing XCOM software and configurations.




  • Installation is simple, requiring AWS CLI, making it a flexible tool for Linux users.



0 comments
43 views

Permalink