Event-driven Transfer on Storage Transfer Service for Google Cloud Storage

Last updated on April 27, 2023

Last January 7, 2023, Google Cloud announced a new capability for Storage Transfer Service (STS). Now, users can do an event-driven transfer quickly to a Cloud storage whenever there are changes to a source bucket. The event-driven transfer is an execution mode on Storage Transfer Service that allows transfer to a destination using the events from the source as triggers. Google Cloud claims that the transfer rate is near-real-time between the source and its destination.

Here are some of the use cases of event-driven transfer:

Event-driven Analytics
Cloud Storage Replication/Data Aggregation
Disaster Recovery/High Availability
Cross-cloud Backup (AWS S3 backup to Cloud Storage)

Cross-region or Cross-project backup
Live migration

Not only can you use this for Cloud Storage buckets, but STS can also transfer objects from AWS S3 to Cloud Storage. When using the AWS S3 bucket as a source, you need to create an SQS queue, enable event notifications on the S3 bucket and set up the required permission. Check the detailed steps here.

Permission Required

When using setting up even-driven transfer within Google Cloud, ensure that the following permissions are correctly configured.

Description	Roles	Permission
Permission to read the source Cloud Storage bucket	roles/storage.legacyBucketReade roles/storage.objectViewer	storage.buckets.get and storage.objects.get
Permission to write on the destination Cloud Storage bucket	roles/storage.legacyBucketWriter	storage.objects.create
Permission to subscribe to the Pub/Sub subscription	roles/pubsub.subscriber	pubsub.subscriptions.consume

How to set up an event-driven transfer using STS?

An event should trigger an STS transfer job. Thus, a Pub/Sub subscription should be configured first. This Pub/Sub subscription will listen and get notifications whenever there are events from the Cloud Storage Bucket.

Create Pub/Sub notification for the Cloud Storage you wish to monitor

Create a pull subscription

After the Pub/Sub subscription is created, you can now create the Transfer Job from the STS. Select event-driven as the transfer execution mode and enter the Pub/Sub subscription name you have made.

Once all of these are configured, you will just wait for Pub/Sub subscription to get an event from the source bucket. After this, the transfer job will be triggered, and the replication will start between the source and the destination. The transfer details are available from the job details page on STS.

Reference:

Announcing launch of Event-driven transfer for Cloud Storage | Google Cloud Blog

Event-driven transfers | Cloud Storage Transfer Service Documentation | Google Cloud

Written by: Lervin John Obando

Lervin is a Cloud Technology enthusiast and an AWS Community Builder whose expertise expands from Systems Administration, Database to Cloud Infrastructure. He is a Multi-Cloud Certified professional holding multiple certifications from AWS, Google, and Oracle. When he is not in front of his computer, you can find him spending time with his wife and dog.