Azure Big Data Cheat Sheet

• A service to store and process large amounts of data sets.
• Create big data clusters for Hadoop, Spark, and Kafka with Azure HDInsight.
  • Reduce costs by scaling your workloads up and down.
  • Monitor all your clusters with Azure Monitor.
  • Autoscaling adjusts cluster size based on workload demands.
  • Deploy clusters within a virtual network for enhanced security.
  • Monitor cluster health using Apache Ambari.
  • Interactive Query (LLAP) for low-latency SQL on large datasets.
• Azure Databricks is based on Apache Spark capabilities that provide an interactive workspace and streamlined workflows.
  • Enables you to read data from multiple sources and use Spark to create breakthrough insights.
  • Deploy apps directly from Git repositories (beta).
  • Apply custom query tags to SQL warehouses for cost attribution.
  • Set default SQL warehouses at workspace or user level.
  • View detailed warehouse activity reasons (query, sessions, idle).
• Azure Synapse Analytics includes dedicated SQL pool (formerly SQL DW) for enterprise data warehousing.
  • Use PolyBase T-SQL queries to import and analyze big data.

Note: Azure Data Lake Analytics is retired (Feb 2024). Use Azure Synapse Analytics or Azure Databricks for big data processing.

• You can use Azure Event Hubs for big data streaming and event ingestion service.
  • Enables you to receive and process millions of events per second.
  • Provides a Kafka-compatible endpoint.
  • Existing Kafka applications can connect without modification.

• Azure Stream Analytics provides you real-time analytics and a complex event-processing engine.
  • Simultaneously analyze and process large volumes of streaming data from multiple sources.

Note: .NET Standard user-defined functions and custom .NET deserializers were retired September 30, 2024. Migrate to JavaScript UDFs or built-in deserializers (JSON, AVRO, CSV).

Azure Big Data Cheat Sheet References:

https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-overview
https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-overview
https://docs.microsoft.com/en-us/azure/databricks/scenarios/what-is-azure-databricks
https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/sql-data-warehouse-overview-what-is
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about
https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-introduction