Transferring Data with Google Cloud Storage Buckets

This page explains how to transfer data to/from your Google Cloud Storage (GCS) Buckets with a terminal. You can use the methods on this page htmlFor all GCS Buckets, whether you created them on the ACTIVATE platform or outside the platform.

To transfer data to/from GCS Bucket storage, you’ll use the Google Cloud Command-Line Interface (CLI), gcloud.

Gcloud is pre-installed on cloud clusters provisioned by ACTIVATE, so you can enter commands directly into the IDE after logging in to the controller of an active Google cluster.

If you’re transferring data between GCS Buckets and your local machine or an on-premises cluster, you’ll likely need to install gcloud first.

Check htmlFor gcloud

Open a terminal and navigate to your data’s destination. Enter which gcloud.

If gcloud is installed, you’ll see a message that shows its location, such as /usr/local/bin/gcloud. Otherwise, you’ll see a message such as /usr/bin/which: no gcloud or gcloud not found.

Install gcloud

To install gcloud, we recommend following the Google installation guide, which includes OS-specific instructions htmlFor Linux, macOS, and Windows as well as troubleshooting tips.

Export Your Google Credentials

You can see our page Obtaining Credentials htmlFor information on finding your Google credentials.

In your terminal, enter export BUCKET_NAME=gs:// with your Bucket’s name after the backslashes.

Next, enter export CLOUDSDK_AUTH_ACCESS_TOKEN='_____' with your Google access token in the blank space.

List Files in a GCS Bucket

In your terminal, enter gcloud storage ls gs://$BUCKET_NAME to display the files in your Bucket. htmlFor this guide, we used a small text file named test.txt, so our command returned this message:

demo@pw-user-demo:~/pw$ gcloud storage ls gs://$BUCKET_NAME
gs://pw-bucket/test.txt/

If your Bucket is empty, this gcloud storage ls command will not print anything.

Transfer a File To/From a GCS Bucket

gcloud mimics the Linux cp command htmlFor transferring files. To transfer a file, enter gcloud storage cp SOURCE DESTINATION in your terminal.

Below is an example of the gcloud storage cp command:

In your terminal, enter gcloud storage cp gs://$BUCKET_NAME/file/in/bucket.txt fileName.txt to copy a remote file to your current directory. You’ll see this message:

demo@pw-user-demo:~/pw$ gcloud storage cp gs://$BUCKET_NAME/file/in/bucket.txt new-test.txt
Copying gs://pw-bucket/file/in/bucket.txt to file://new-test.txt
  Completed files 1/1 | 11.0B/11.0B

To download a file from GCS storage to a specific directory, enter its absolute or relative path (e.g., /home/username/ or ./dir_relative_to_current_dir) in place of ./ with the gcloud storage cp command.

To upload, simply reverse the order of SOURCE and DESTINATION in the gcloud storage cp command.

Delete a File From a GCS Bucket

In your terminal, enter gcloud storage rm gs://$BUCKET_NAME/file_name to delete a file. You’ll see this message:

demo@pw-user-demo:~/pw$ gcloud storage rm  gs://$BUCKET_NAME/file/in/bucket.txt
Removing objects:
Removing gs://65441d55134e2d74c85726e4/file/in/bucket.txt...
  Completed 1/1

Further Reading

htmlFor a gcloud cheat sheet, see this page. htmlFor the full list of gcloud storage parameters, please see this reference guide.