Transferring Data with Google Cloud Storage Buckets

This page explains how to transfer data to/from your Google Cloud Storage (GCS) Buckets with a terminal. You can use the methods on this page for all GCS Buckets, whether you created them on the ACTIVATE platform or outside the platform.

To transfer data to/from GCS Bucket storage, you’ll use the Google Cloud Command-Line Interface (CLI), gcloud.

Gcloud is pre-installed on cloud clusters provisioned by ACTIVATE, so you can enter commands directly into the IDE after logging in to the controller of an active Google cluster.

If you’re transferring data between GCS Buckets and your local machine or an on-premises cluster, you’ll likely need to install gcloud first.

Check for gcloud

Open a terminal and navigate to your data’s destination. Enter which gcloud.

If gcloud is installed, you’ll see a message that shows its location, such as /usr/local/bin/gcloud. Otherwise, you’ll see a message such as /usr/bin/which: no gcloud or gcloud not found.

Install gcloud

To install gcloud, we recommend following the Google installation guide, which includes OS-specific instructions for Linux, macOS, and Windows as well as troubleshooting tips.

About gsutil

Google refers to gsutil commands as a legacy feature that is minimally maintained; instead, they recommend using gcloud commands. For this reason, we've used gcloud in this guide. Please see this page for Google's gsutil guide.

Export Your Google Credentials

You can see our page Obtaining Credentials for information on finding your Google credentials.

In your terminal, enter export BUCKET_NAME=gs:// with your Bucket’s name after the backslashes.

Next, enter export CLOUDSDK_AUTH_ACCESS_TOKEN='_____' with your Google access token in the blank space.

Note

Please be sure to include the quotes on both ends of your access token. There are characters inside Google tokens that, without quotation marks, systems will try to read as commands.

List Files in a GCS Bucket

In your terminal, enter gcloud storage ls gs://$BUCKET_NAME to display the files in your Bucket. For this guide, we used a small text file named test.txt, so our command returned this message:

demo@pw-user-demo:~/pw$ gcloud storage ls gs://$BUCKET_NAME
gs://pw-bucket/test.txt/

If your Bucket is empty, this gcloud storage ls command will not print anything.

Transfer a File To/From a GCS Bucket

gcloud mimics the Linux cp command for transferring files. To transfer a file, enter gcloud storage cp SOURCE DESTINATION in your terminal.

Below is an example of the gcloud storage cp command:

In your terminal, enter gcloud storage cp gs://$BUCKET_NAME/file/in/bucket.txt fileName.txt to copy a remote file to your current directory. You’ll see this message:

demo@pw-user-demo:~/pw$ gcloud storage cp gs://$BUCKET_NAME/file/in/bucket.txt new-test.txt
Copying gs://pw-bucket/file/in/bucket.txt to file://new-test.txt
  Completed files 1/1 | 11.0B/11.0B

To download a file from GCS storage to a specific directory, enter its absolute or relative path (e.g., /home/username/ or ./dir_relative_to_current_dir) in place of ./ with the gcloud storage cp command.

To upload, simply reverse the order of SOURCE and DESTINATION in the gcloud storage cp command.

Delete a File From a GCS Bucket

In your terminal, enter gcloud storage rm gs://$BUCKET_NAME/file_name to delete a file. You’ll see this message:

demo@pw-user-demo:~/pw$ gcloud storage rm  gs://$BUCKET_NAME/file/in/bucket.txt
Removing objects:
Removing gs://65441d55134e2d74c85726e4/file/in/bucket.txt...
  Completed 1/1

Check for gcloud​

Install gcloud​

Export Your Google Credentials​

List Files in a GCS Bucket​

Transfer a File To/From a GCS Bucket​

Delete a File From a GCS Bucket​

Further Reading​