Transferring Data to/from AWS S3 Storage
Working With AWS CLI
To transfer data to/from S3 bucket storage, you’ll use the AWS Command Line Interface (CLI).
Because the AWS CLI is pre-installed on AWS clusters in the PW platform, you can enter commands directly into the IDE after logging in to the controller of an active AWS cluster.
If you’re transferring data between S3 buckets and a folder, an on-premise cluster, or a remote device, you’ll likely need to install the AWS CLI first.
Check for AWS CLI
Open a terminal or command line and navigate to your data’s destination. Enter which aws
.
If the AWS CLI is installed, you’ll see a message that shows its location, such as /usr/local/bin/aws
.
If the AWS CLI is not installed, you’ll see a message such as /usr/bin/which: no aws
or aws not found
.
Install AWS CLI
If you need to install the AWS CLI, we recommend following the AWS installation guide, which includes OS-specific instructions for Linux, macOS, and Windows as well as troubleshooting tips.
Transferring Your Data
Export Your AWS Credentials
In your terminal, enter export AWS_ACCESS_KEY_ID="_____"
with your AWS access key ID in the blank space.
Enter export AWS_SECRET_ACCESS_KEY="_____"
with your AWS access key ID in the blank space.
Please be sure to include the quotation marks on both ends of your key IDs. There are characters inside AWS key IDs that, without quotation marks, systems will try to read as commands.
Validate Your AWS Credentials
In your terminal, enter aws sts get-caller-identity
to check that the system correctly registered your AWS credentials.
If your login was successful, you’ll see your AWS UserID, Account, and ARN:
{
"UserId": "ABCD1E2FG3H4IJKLMNOPQ",
"Account": "012345678901",
"Arn": "arn:aws:iam:: 012345678901:user/Admin"
}
If your login was not successful, enter your AWS access keys again with the export
commands above.
You can also configure your login credentials manually with aws configure
. For more information about that process, see the AWS documentation on environment variables and configuration files.
List Files in S3
In your terminal, enter aws s3 ls s3://bucket_name
to display the files in your bucket. For this guide, we used a small text file named test.txt
, so our command returned this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test
2023-02-08 16:37:56 28 test.txt
Transfer a File From S3
In your terminal, enter aws s3 cp s3://bucket_name/file_name ./
to download a file from S3 storage to your current directory. You’ll see this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./
download: s3://cloud-data-test/test.txt to ./test.txt
To download a file from S3 storage to a specific directory, enter its name after ./
with the aws s3 cp
command. We used the storage
directory in this example:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./storage
download: s3://cloud-data-test/test.txt to ./storage
Transfer a File to S3
In your terminal, enter aws s3 cp file_name s3://bucket_name/
to transfer a file to your S3 bucket. You'll see this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp test_upload.txt s3://cloud-data-test/
upload: ./test_upload.txt to s3://cloud-data-test/test_upload.txt
You can check your S3 files with aws s3 ls s3://bucket_name
:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test/
2023-02-08 16:37:56 28 test.txt
2023-02-08 17:19:24 28 test_upload.txt
Delete a File From S3
In your terminal, enter aws s3 rm s3://bucket_name/file_name
to delete a file. You’ll see this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 rm s3://cloud-data-test/test_upload.txt
delete: s3://cloud-data-test/test_upload.txt
You can check your S3 files with `aws s3 ls s3://bucket_name`:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test/
2023-02-08 16:37:56 28 test.txt