AWS Lake Formation

  • A service for managing and building data lakes.

  • It stores and catalogs data from databases and object storage before transferring it to a new S3 data lake.

  • You can also use ML algorithms to clean and classify data and secure access to sensitive data with granular controls at the column, row, and cell levels.

How It Works

Tutorials dojo strip
  • Identify existing data stores, such as S3 or databases, and move the data to your data lake.

  • The data is then crawled, cataloged, and prepared for analytics.

  • Lastly, provide users with data access through their preferred analytics services.

Concepts

  • Data Lake

    • A persistent data stored in Amazon S3:

      • Structured and unstructured data

      • Raw data and transformed data

    • When you register an Amazon S3 location, the S3 path and all folders under that path are registered.

  • Data Catalog

    • Persistent metadata store.

    • A repository where various systems can store and find metadata to keep track of data in data silos and use that metadata to query and transform the data.

    • The AWS Glue Data Catalog maintains metadata about data lakes, data sources, transforms, and targets.

    • Metadata about data sources and targets are in the form of databases and tables.

      • Databases – collection of tables.

      • Tables – information about data in the data lake.

    • You can control access to databases and tables in the data catalog using permissions.

    • Each AWS account has one Data Catalog per AWS Region.

    • Governed tables are unique to AWS Lake Formation and has the following features:

      • ACID transactions

      • Automatic data compaction

      • Time-travel queries

    • Resource links

      • Links to shared databases and tables in the external accounts

      • It is used for cross-account access to data in the data lake.

  • Blueprint

    • A data management template to ingest data into a data lake.

    • You can use blueprints to configure the workflow by providing input such as the data source, data target, and schedule.

    • Types of blueprints:

      • Database snapshot

      • Incremental database

      • Log file

  • Workflow

    • Defines the data source and schedule for importing data into the data lake.

    • A container for a collection of AWS Glue jobs, crawlers, and triggers.

    • Uses AWS Glue to orchestrate the loading and updating of data.

    • It can be run on-demand or on a schedule.

    • With AWS Glue directed acyclic graph (DAG), you can monitor the progress of the workflow.

Security

  • Encrypt and decrypt data in Amazon S3 using AWS KMS.

  • Use AWS CloudTrail to capture all Lake Formation API calls.

  • Types of permissions:

    • Metadata access – for data catalog resources.

    • Underlying data access – for Amazon S3 locations.

  • You can use the credential vending API to provide temporary credentials to registered Amazon S3 locations based on effective permissions, allowing authorized engines to access data on users’ behalf.

  • With Querying API, you can retrieve data from Amazon S3, filter the results based on effective permissions, and then share it to query engines.

  • The service-linked role provides necessary permissions to call other AWS services on your behalf.

  • When you need to grant more permissions than the service-linked role provides, use a user-defined role.

  • You can specify, grant, and revoke permissions on tables in the data catalog.

  • Manage your AWS Glue Data Catalog objects and data locations in Amazon S3 using the LakeFormation permissions model.

  • Use LakeFormation tag-based access control for a large number of data catalog resources.

  • You can create a data filter to restrict access to certain data in query results and engines.

Pricing

  • You are charged for transaction requests and metadata storage.

  • You are charged for data filtering or the number of bytes scanned by the Storage API.

  • You are charged based on the number of bytes processed by the storage optimizer.

References:
https://aws.amazon.com/lake-formation/
https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html

Tutorials Dojo portal

Enroll Now – Our AWS Practice Exams with 95% Passing Rate

AWS Practice Exams Tutorials Dojo

FREE AWS Exam Readiness Digital Courses

Enroll Now – Our Azure Certification Exam Reviewers

azure reviewers tutorials dojo

Enroll Now – Our Google Cloud Certification Exam Reviewers

Tutorials Dojo Exam Study Guide eBooks

tutorials dojo study guide eBook

Subscribe to our YouTube Channel

Tutorials Dojo YouTube Channel

FREE Intro to Cloud Computing for Beginners

FREE AWS, Azure, GCP Practice Test Samplers

Browse Other Courses

Generic Category (English)300x250

Recent Posts

AWS, Azure, and GCP Certifications are consistently among the top-paying IT certifications in the world, considering that most companies have now shifted to the cloud. Earn over $150,000 per year with an AWS, Azure, or GCP certification!

Follow us on LinkedIn, YouTube, Facebook, or join our Slack study group. More importantly, answer as many practice exams as you can to help increase your chances of passing your certification exams on your first try!

View Our AWS, Azure, and GCP Exam Reviewers Check out our FREE courses

Our Community

~98%
passing rate
Around 95-98% of our students pass the AWS Certification exams after training with our courses.
200k+
students
Over 200k enrollees choose Tutorials Dojo in preparing for their AWS Certification exams.
~4.8
ratings
Our courses are highly rated by our enrollees from all over the world.

What our students say about us?