Transferring Data to/from AWS S3 Storage

Working With AWS CLI

To transfer data to/from S3 bucket storage, you’ll use the AWS Command Line Interface (CLI).

Because the AWS CLI is pre-installed on AWS clusters in the PW platform, you can enter commands directly into the IDE after logging in to the controller of an active AWS cluster.

If you’re transferring data between S3 buckets and a folder, an on-premise cluster, or a remote device, you’ll likely need to install the AWS CLI first.

Check for AWS CLI

Open a terminal or command line and navigate to your data’s destination. Enter which aws.

If the AWS CLI is installed, you’ll see a message that shows its location, such as /usr/local/bin/aws.

If the AWS CLI is not installed, you’ll see a message such as /usr/bin/which: no aws or aws not found.

Install AWS CLI

If you need to install the AWS CLI, we recommend following the AWS installation guide, which includes OS-specific instructions for Linux, macOS, and Windows as well as troubleshooting tips.

Transferring Your Data

Export Your AWS Credentials

In your terminal, enter export AWS_ACCESS_KEY_ID="_____" with your AWS access key ID in the blank space.

Enter export AWS_SECRET_ACCESS_KEY="_____" with your AWS access key ID in the blank space.

Note

Please be sure to include the quotation marks on both ends of your key IDs. There are characters inside AWS key IDs that, without quotation marks, systems will try to read as commands.

Validate Your AWS Credentials

In your terminal, enter aws sts get-caller-identity to check that the system correctly registered your AWS credentials.

If your login was successful, you’ll see your AWS UserID, Account, and ARN:

{
    "UserId": "ABCD1E2FG3H4IJKLMNOPQ",
    "Account": "012345678901",
    "Arn": "arn:aws:iam:: 012345678901:user/Admin"
}

If your login was not successful, enter your AWS access keys again with the export commands above.

You can also configure your login credentials manually with aws configure. For more information about that process, see the AWS documentation on environment variables and configuration files.

List Files in S3

In your terminal, enter aws s3 ls s3://bucket_name to display the files in your bucket. For this guide, we used a small text file named test.txt, so our command returned this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test
2023-02-08 16:37:56         28 test.txt

Transfer a File From S3

In your terminal, enter aws s3 cp s3://bucket_name/file_name ./ to download a file from S3 storage to your current directory. You’ll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./
download: s3://cloud-data-test/test.txt to ./test.txt

To download a file from S3 storage to a specific directory, enter its name after ./ with the aws s3 cp command. We used the storage directory in this example:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./storage
download: s3://cloud-data-test/test.txt to ./storage

Transfer a File to S3

In your terminal, enter aws s3 cp file_name s3://bucket_name/ to transfer a file to your S3 bucket. You'll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp test_upload.txt s3://cloud-data-test/
upload: ./test_upload.txt to s3://cloud-data-test/test_upload.txt

You can check your S3 files with aws s3 ls s3://bucket_name:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test/
2023-02-08 16:37:56         28 test.txt
2023-02-08 17:19:24         28 test_upload.txt

Delete a File From S3

In your terminal, enter aws s3 rm s3://bucket_name/file_name to delete a file. You’ll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 rm s3://cloud-data-test/test_upload.txt
delete: s3://cloud-data-test/test_upload.txt

You can check your S3 files with `aws s3 ls s3://bucket_name`:

demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test/
2023-02-08 16:37:56         28 test.txt