Ends in
00
days
00
hrs
00
mins
00
secs
ENROLL NOW

Get up to $10 DISCOUNT on our AWS Solutions Architect Associate Reviewers!

Event-driven Transfer on Storage Transfer Service for Google Cloud Storage

Last January 7, 2023, Google Cloud announced a new capability for Storage Transfer Service (STS). Now, users can do an event-driven transfer quickly to a Cloud storage whenever there are changes to a source bucket. The event-driven transfer is an execution mode on Storage Transfer Service that allows transfer to a destination using the events from the source as triggers. Google Cloud claims that the transfer rate is near-real-time between the source and its destination.

Here are some of the use cases of event-driven transfer:

  • Event-driven Analytics

  • Cloud Storage Replication/Data Aggregation

  • Disaster Recovery/High Availability

  • Cross-cloud Backup (AWS S3 backup to Cloud Storage)

  • Cross-region or Cross-project backup

  • Tutorials dojo strip
  • Live migration

Not only can you use this for Cloud Storage buckets, but STS can also transfer objects from AWS S3 to Cloud Storage. When using the AWS S3 bucket as a source, you need to create an SQS queue, enable event notifications on the S3 bucket and set up the required permission. Check the detailed steps here.

Permission Required

When using setting up even-driven transfer within Google Cloud, ensure that the following permissions are correctly configured.

Description

Roles

Permission

Permission to read the source Cloud Storage bucket

roles/storage.legacyBucketReade

roles/storage.objectViewer

storage.buckets.get and storage.objects.get

Permission to write on the destination Cloud Storage bucket

roles/storage.legacyBucketWriter

storage.objects.create

Permission to subscribe to the Pub/Sub subscription

roles/pubsub.subscriber

pubsub.subscriptions.consume

How to set up an event-driven transfer using STS?

Event-driven transfer on Storage Transfer Service for Google Cloud Storage

An event should trigger an STS transfer job. Thus, a Pub/Sub subscription should be configured first. This Pub/Sub subscription will listen and get notifications whenever there are events from the Cloud Storage Bucket.

Create Pub/Sub notification for the Cloud Storage you wish to monitor

Event-driven transfer on Storage Transfer Service for Google Cloud Storage

Create a pull subscription

Event-driven transfer on Storage Transfer Service for Google Cloud Storage

After the Pub/Sub subscription is created, you can now create the Transfer Job from the STS. Select event-driven as the transfer execution mode and enter the Pub/Sub subscription name you have made.

Event-driven transfer on Storage Transfer Service for Google Cloud Storage

Once all of these are configured, you will just wait for Pub/Sub subscription to get an event from the source bucket. After this, the transfer job will be triggered, and the replication will start between the source and the destination. The transfer details are available from the job details page on STS.

Reference: 

Tutorials Dojo portal

FREE AWS Exam Readiness Digital Courses

Enroll Now – Our Azure Certification Exam Reviewers

azure reviewers tutorials dojo

Enroll Now – Our Google Cloud Certification Exam Reviewers

Tutorials Dojo Exam Study Guide eBooks

tutorials dojo study guide eBook

Subscribe to our YouTube Channel

Tutorials Dojo YouTube Channel

FREE Intro to Cloud Computing for Beginners

FREE AWS, Azure, GCP Practice Test Samplers

Browse Other Courses

Generic Category (English)300x250

Recent Posts

Written by: Lervin John Obando

Lervin is a Cloud Technology enthusiast and an AWS Community Builder whose expertise expands from Systems Administration, Database to Cloud Infrastructure. He is a Multi-Cloud Certified professional holding multiple certifications from AWS, Google, and Oracle. When he is not in front of his computer, you can find him spending time with his wife and dog.

AWS, Azure, and GCP Certifications are consistently among the top-paying IT certifications in the world, considering that most companies have now shifted to the cloud. Earn over $150,000 per year with an AWS, Azure, or GCP certification!

Follow us on LinkedIn, YouTube, Facebook, or join our Slack study group. More importantly, answer as many practice exams as you can to help increase your chances of passing your certification exams on your first try!

View Our AWS, Azure, and GCP Exam Reviewers Check out our FREE courses

Our Community

~98%
passing rate
Around 95-98% of our students pass the AWS Certification exams after training with our courses.
200k+
students
Over 200k enrollees choose Tutorials Dojo in preparing for their AWS Certification exams.
~4.8
ratings
Our courses are highly rated by our enrollees from all over the world.

What our students say about us?