Syncing files from S3 bucket It is definitely advisable in order to have a redundancy in your production storage in case one of the S3 went down. With sync operation, source location S3’s objects will synchronise to the destination location.
In this illustration, we will be using AWS CLI as the main command line method to perform the operation. You may refer to the AWS documentation to setup AWS CLI if not installed.
We will be using s3 commands here and you may refer to here to illustrate the differences between s3 and s3api commands.
Method to Sync files from S3
- Single file Sync from S3
- Sync files from whole bucket to current directory
- Sync files from whole bucket to another bucket
Do note that there are some fundamental differences between cp and sync, please refer here to view the comparison.
Single file Sync from S3
In this example, we will sync only 1 file from S3 to local directory.
The command consist of a few segments. By default, sync will have the whole source bucket cloned to the destination. Hence, we need to use --exclude and --include to indicate.
# Sync only 'old-name.txt' located at 'my-bucket' to 'clone-bucket'
aws s3 sync s3://my-bucket s3://clone-bucket --exclude "*" --include "old-name.txt"Sync files from whole bucket to current directory
If there is a need to sync all the files to current directory, just add in a dot at the end. The files that does not exist in the source will be deleted if we append --delete to the command.
# Sync everything from 'my-bucket' to current directory
aws s3 sync s3://my-bucket . --deleteSync files from whole bucket to another bucket
If there is a need to sync all the files to another bucket, just add in the other bucket URL.
# Sync everything from 'my-bucket' to 'clone-bucket'
aws s3 sync s3://my-bucket s3://clone-bucket --deleteConclusion
With AWS CLI, it is easy and straight forward to sync files from S3 to current directory or other bucket. More information on how to use sync is in AWS documentation.
No Responses Yet