Amazon Managed Workflows for Apache Airflow (MWAA) Cheat Sheet

• Amazon MWAA is a service that helps you manage and automate your data workflows using Apache Airflow.
• Workflows are designed as Directed Acyclic Graphs (DAGs) using Python.

Use Cases

• Complex Data Workflows: Handles complex data processing tasks.
• ETL Jobs: Coordinates Extract, Transform, Load (ETL) processes.
• Machine Learning: Prepares datasets for machine learning models.
• Data Orchestration Across AWS Services: Easily integrates with S3, Redshift, Lambda, and RDS for orchestrated workflows.
• Event-Driven Workflows: Can trigger DAGs based on events from EventBridge or S3.
• Data Quality Checks: Automate validation and monitoring of datasets as part of pipelines.

Key Features

• Deployment: Easy setup and operation.
• Scaling: Automatically adjusts to match workload demands.
• Security: Built-in measures for data protection.
• Monitoring: Comprehensive tracking of workflows.
• Cost-Effectiveness: Reduces operational costs due to its managed nature.
• Plugin Support: Extend Airflow capabilities with custom operators, hooks, and sensors.
• Version Management: Supports multiple Airflow versions with minimal disruption.
• Integration with AWS IAM: Fine-grained access control for DAGs and resources.
• Logging and Auditing: Centralized logs stored in CloudWatch for auditing and troubleshooting.

Security

• Network Isolation: Operates within Amazon’s Virtual Private Cloud (VPC).
• Data Encryption: Uses AWS Key Management Service for automatic data encryption.
• IAM Roles for Tasks: Each task can assume a role to securely access AWS resources.
• Secrets Management: Integrates with AWS Secrets Manager for secure credential handling.
• Audit Trails: Logs all Airflow actions for compliance and auditing purposes.

Pricing

• Environment Pricing: Billed hourly based on environment size.
• Additional Worker Instance Pricing: Billed hourly based on task load.
• Additional Scheduler Instance Pricing: Billed hourly based on availability.
• Database Storage: Billed per GB-month for storage consumed.
• Data Transfer Costs: Charges may apply for data transferred between VPC and other AWS regions.
• Log Storage Costs: CloudWatch log storage and retrieval are billed separately.
• DAG Runs: Indirect cost based on the number of tasks executed and compute resources consumed.

Amazon Managed Workflows for Apache Airflow References:

https://aws.amazon.com/managed-workflows-for-apache-airflow/

https://aws.amazon.com/managed-workflows-for-apache-airflow/features/

https://aws.amazon.com/managed-workflows-for-apache-airflow/pricing/