Amazon Data Firehose Cheat Sheet

• Easily stream real-time data to AWS destinations without custom applications.

• Adjusts resources dynamically to match data volume.

• AWS Lambda supports data format conversion (e.g., Parquet, ORC) and custom transformations.

• Works seamlessly with Amazon S3, Redshift, OpenSearch, Splunk, and third-party HTTP endpoints.

Features

• Real-Time Data Delivery: Streams data with minimal latency to multiple destinations.

• Data Buffering: Configurable buffer sizes and intervals to control data flow.

• Compression & Encryption: Supports GZIP and Snappy compression; integrates with AWS KMS for encryption.



• Monitoring & Alerts: Amazon CloudWatch metrics and alarms monitor delivery stream health.

• Dynamic Partitioning: Organizes data into partitions for optimized query performance.

Use Cases

• Stream data into Amazon S3 for scalable storage and analysis.

• Load data into Amazon Redshift for high-performance querying.

• Deliver data to Amazon OpenSearch for real-time analytics and search.

• Stream logs to SIEM tools like Splunk for threat detection.

• Feed data into machine learning models for real-time predictions.

Security

• Encryption: Protects data at rest and in transit using AWS KMS.

• Access Control: Utilizes AWS IAM for detailed permission management.

• Audit Trails: Supports AWS CloudTrail for tracking API calls and stream activity.

• Compliance: Meets various standards such as GDPR and HIPAA for regulatory compliance.

Pricing

• Pay-As-You-Go: Charges based on the volume of data ingested and delivered.

• Free Tier: Includes a limited free tier for new users to get started.

• Additional Costs: Charges may apply for data transformation, compression, and extended data retention.

References:

https://aws.amazon.com/firehose/

https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html

https://aws.amazon.com/firehose/pricing/