Google Cloud Storage (GCS)
- An object storage service that stores data within buckets.
- Below is a sample Cloud Storage integration:
- The data you upload on Cloud Storage are called objects.
- An object is an immutable piece of data consisting of a file in any format.
- You store objects inside containers called buckets.
- All buckets belong to a project.
- Each project can have multiple buckets.
- You can also configure a Cloud Storage bucket to host a static website for a domain you own.
- Life Cycle Management
- You can define conditions that trigger data deletion, or transition to a cheaper storage class with object life cycle management.
- Continue to store old copies of objects you store when they are deleted or overwritten.
- Retention Policies
- Define minimum retention periods that objects must be stored.
- Object holds
- Place a hold on an object to prevent deletion.
- Encryption keys
- Access Permissions
- Access Control List
- Uniform bucket level access
- Object and Bucket Level Permissions
- Standard Storage
- Good for hot data that is accessed frequently.
- Nearline Storage
- Good for use cases that need to store objects for at least 30 days.
- Ideal for data that you plan to access once per month or less.
- Coldline Storage
- Is a low-cost storage option for storing infrequently accessed data within 90 days.
- Archive Storage
- Is the coldest storage among the storage classes.
- Designed for storing archive data and disaster recovery data that is expected to be accessed once per 365 days or less.
- A Python application that enables you to manage your Cloud Storage from the command line.
- You can use gsutil to perform bucket and object management tasks like:
- creating and deleting buckets
- uploading, downloading, and deleting objects
- listing buckets and objects
- moving, copying, and renaming objects
- editing object and bucket ACL
- gsutil performs all operations using HTTPS and TLS
Uploading objects to GCS
You can send upload requests to Google Cloud Storage via the following methods:
- Simple Upload – utilize this if the file is small enough to upload again if the connection fails, and if there is no object metadata to send as part of the upload request.
- Multipart Upload – utilize this if the file is small enough to upload again if the connection fails, and you need to include object metadata as part of the upload request.
- Resumable Upload – utilize this for a more reliable transfer, which is especially important with large files.
- Parallel composite uploads – utilize if network and disk speed are not limiting factors. When doing parallel composite upload, a file is divided into up to 32 chunks and uploaded in parallel to temporary objects. The final object is recreated using the temporary objects, and the temporary objects are deleted.
- Alternatively, for uploading large volumes of data (from hundreds of terabytes up to 1 petabyte), you can utilize the Transfer Appliance. It is a hardware appliance you can use to securely migrate to Google Cloud Platform without disrupting business operations.
- Pricing for Cloud Storage services is based on what you use, including:
- the amount of data you store,
- the duration for which you store it,
- the number of operations you perform on your data,
- the network resources used when moving or accessing your data.
- For “cold” storage classes meant to store long-term, infrequently accessed data, there are also charges for retrieving data and early deletion of data.
- You can require accessors of your data to include a project ID to bill for network charges, operation charges, and retrieval fees.