In today's world of cloud storage, Amazon S3 (Simple Storage Service) is one of the popular solutions for storing large amounts of data securely, affordably, and reliably. Interacting with S3 storage can be done via several methods. One of the most efficient ways to move files seamlessly from local storage to S3 storage is through AWS Mountpoint for S3, a lightweight, high-performance file system client designed to make S3 buckets accessible like regular local directories.
As an XCOM Data Transport for Linux user, if you want to interact with S3 storage to send/receive files, you can leverage software like AWS mountpoint. This blog post will guide you through installing AWS Mountpoint and using it with XCOM Data Transport to transfer files to an S3 bucket.
What Is AWS Mountpoint?
AWS Mountpoint for S3 is an open-source, POSIX-like file system that makes it easy to map Amazon S3 storage to a local directory on your machine, allowing you to read and write files using native Linux commands. This makes interacting with your S3 bucket feel as if you're working with a native file system on your local Linux server.
Key features of AWS Mountpoint:
Seamless access to your S3 buckets like local directories
High-performance file transfer
POSIX-like file handling for large-scale data movements
This blog provides step-by-step instructions for installing AWS Mountpoint and using it with XCOM Data Transport on a Redhat Linux Server. Though we have used Redhat Linux in this blog, AWS Mountpoint can be installed on any other Linux distribution. For more details, refer to https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint-installation.html.
Prerequisites
AWS account with access to S3
AWS CLI installed and configured on your machine
RHEL 8 or later
A valid subscription or access to Red Hat repositories for installing necessary packages
Step 1: Install AWS Mountpoint
AWS Mountpoint software is available as a precompiled binary or it can be built from the source.
Here's how to install and configure it on Redhat Linux.
- Download the latest release that is appropriate for your architecture (for example, x86_64) and install it:
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
yum install ./mount-s3.rpm
mount-s3 --version
You should see output similar to the following:
mount-s3 1.5.0
Step 2: Configure AWS CLI
If you haven't already configured the AWS CLI, you will need to set it up before proceeding.
aws configure
You'll need to provide your:
AWS Access Key ID
AWS Secret Access Key
Default region name (for example, us-west-1)
Default output format (for example, json)
Step 3: Mount the S3 Bucket Locally
Now that AWS Mountpoint is installed, mount an S3 bucket as a directory on your local machine.
mkdir /tmp/s3-bucket
mount-s3 --allow-delete --allow-overwrite --allow-other my-bucket /tmp/s3-bucket
Note: Ensure the directory is empty before mounting the bucket. Otherwise, mount-s3 fails to mount the S3 bucket. In case of a system reboot, You need to mount the file system again.
At this point, you should see the contents of your S3 bucket inside the /tmp/s3-bucket folder. You can interact with these files as if they were local files within the limitations of Mountpoint software.
Step 4: Transfer Files to the S3 Bucket Using XCOM Data Transport
With the bucket mounted, you can transfer files from any of your servers with XCOM installed on it to S3 storage using XCOM transfer jobs.
To upload files from your local system to the S3 bucket, use the mounted /tmp/s3-bucket as your remote file location. The local system could be any of the platforms supported by XCOM.
For example,
TRANSFER_TYPE=3
LOCAL_FILE=C:\temp\testfile.txt
REMOTE_FILE=/tmp/s3-bucket/testfile.txt
REMOTE_SYSTEM=192.168.1.2
FILE_OPTION=CREATE
USERID=<userid>
PASSWORD=<password>
Note: You can not use REPLACE or APPEND options with the FILE_OPTION parameter when transferring to a mounted S3 bucket.
You can submit the transfer job using the command:
xcomtcp -f sendfiletos3.cnf
To download and receive the testfile.txt file that is stored under the S3 bucket mounted to /tmp/s3-bucket on a remote XCOM Linux server with IP address 192.168.1.2 and optionally replace the file on the local Windows server if it already exists, create an XCOM configuration file named receivefilefroms3.cnf with the following contents:
TRANSFER_TYPE=4
LOCAL_FILE_RF=C:\temp\testfile.txt
REMOTE_FILE_RF=/tmp/s3-bucket/testfile.txt
REMOTE_SYSTEM_RF=192.168.1.2
FILE_OPTION_RF=REPLACE
USERID=<userid>
PASSWORD=<password>
You can submit the transfer job using the command:
xcomtcp -f receivefilefroms3.cnf
(Optional) Step 5: Unmount the S3 Bucket
After you've finished transferring files, unmount the S3 bucket to ensure no more data is written to it unintentionally.
To unmount the S3 bucket:
sudo umount /tmp/s3-bucket
Troubleshooting
If you encounter issues while using AWS Mountpoint on Red Hat, here are a few common troubleshooting tips:
Permission Issues: If you’re unable to mount the bucket, check that the user has the necessary permissions to access the S3 bucket.
AWS CLI Configuration: Ensure that the AWS CLI is configured with the correct credentials, especially if your system uses multiple profiles.
- Issue the below command to check the messages related to mount-s3 in SYSLOG.
journalctl -e SYSLOG_IDENTIFIER=mount-s3
For Mountpoint logging details, refer to https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint-troubleshooting.html.
Conclusion
AWS Mountpoint makes it easier to interact with Amazon S3 by enabling applications like XCOM Data Transport to access S3 buckets directly from the Linux file system. Whether you're dealing with large-scale data transfers or automating backups, Mountpoint offers a high-performance and user-friendly way to access your S3 storage.
By following the steps above, you can now mount your S3 buckets, transfer files using XCOM seamlessly, and integrate S3 storage into your existing XCOM Data Transport transfer jobs.
Key takeaways:
AWS Mountpoint provides native-like file system access to S3 buckets on Red Hat Linux.
It allows seamless file transfers using existing XCOM software and configurations.
Installation is simple, requiring AWS CLI, making it a flexible tool for Linux users.