Transferring Data with AWS S3

This page explains how to transfer data to/from your AWS S3 buckets with a terminal. You can use the methods on this page for all S3 buckets, whether you created them on the ACTIVATE platform or outside the platform.

To transfer data to/from S3 bucket storage, you’ll use the AWS Command-Line Interface (CLI).

The AWS CLI is pre-installed on cloud clusters provisioned by ACTIVATE, so you can enter commands directly into the IDE after logging in to the controller of an active AWS cluster.

If you’re transferring data between S3 buckets and your local machine or an on-premises cluster, you’ll likely need to install the AWS CLI first.

Check for AWS CLI

Open a terminal or command line and navigate to your data’s destination. Enter which aws.

If the AWS CLI is installed, you’ll see a message that shows its location, such as /usr/local/bin/aws.

If the AWS CLI is not installed, you’ll see a message such as /usr/bin/which: no aws or aws not found.

Install AWS CLI

If you need to install the AWS CLI, we recommend following the AWS installation guide, which includes OS-specific instructions for Linux, macOS, and Windows as well as troubleshooting tips.

Export Your AWS Credentials

You can see our page Obtaining Credentials for information on finding your AWS credentials.

In your terminal, enter export AWS_ACCESS_KEY_ID="_____" with your AWS access key ID in the blank space.

Enter export AWS_SECRET_ACCESS_KEY="_____" with your AWS access key ID in the blank space.

Note

Please be sure to include the quotation marks on both ends of your key IDs. There are characters inside AWS key IDs that, without quotation marks, systems will try to read as commands.

List Files in S3

In your terminal, enter aws s3 ls s3://bucket_name to display the files in your bucket. For this guide, we used a small text file named test.txt, so our command returned this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test
2023-02-08 16:37:56         28 test.txt

Transfer a File To/From S3

The AWS CLI mimics the Linux cp command for transferring files. To transfer a file, enter aws s3 SOURCE DESTINATION in your terminal.

SOURCE and DESTINATION can be local files or S3 buckets. S3 buckets are formatted as s3://bucket_name/file_name.

Below is an example of the aws s3 command.

In your terminal, enter aws s3 cp s3://bucket_name/file_name ./ to download a file from S3 storage to your current directory. You’ll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./
download: s3://cloud-data-test/test.txt to ./test.txt

To download a file from S3 storage to a specific directory, enter its absolute or relative path (e.g., /home/username/ or ./dir_relative_to_current_dir) in place of ./ with the aws s3 cp command. In the example below, we used the storage directory, which is located inside the current working directory:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./storage
download: s3://cloud-data-test/test.txt to ./storage

To upload, simply reverse the order of SOURCE and DESTINATION in the aws s3 command.

Delete a File From S3

In your terminal, enter aws s3 rm s3://bucket_name/file_name to delete a file. You’ll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 rm s3://cloud-data-test/test_upload.txt
delete: s3://cloud-data-test/test_upload.txt

Check for AWS CLI​

Install AWS CLI​

Export Your AWS Credentials​

List Files in S3​

Transfer a File To/From S3​

Delete a File From S3​

Further Reading​