The MLA-C02 exam reflects the broadened scope of the machine learning engineer role, incorporating significant advancements in Artificial Intelligence. The updated exam includes:

• Generative AI Implementation: Leveraging Amazon Bedrock and designing Retrieval-Augmented Generation (RAG) architectures using vector databases such as OpenSearch Serverless and pgvector.
• Agentic AI: Orchestrating autonomous AI agents and managing complex, agentic workflows to optimize efficiency.
• Foundation Models (FMs) and LLMs: Expertise in model selection, fine-tuning, and the operationalization of Large Language Models (LLMs).
• Responsible AI Practices: Implementing guardrails, mitigating hallucinations, and managing drift across traditional ML and Generative AI (GenAI) solutions.
• Expanded Technical Scope: Increased focus on SageMaker AI pipelines, specialized data preparation for AI, and advanced model evaluation techniques.

For a more detailed and in-depth look at what’s new in the MLA-C02 exam, you can refer to our article here.

Domain 1: Data Preparation for ML and AI (28%)

This domain focuses on the ingestion, storage, and transformation of data required for high-quality modeling.

• Task 1.1: Collect and store data
  • Extract data from data sources: Extract data from Amazon S3, Amazon EBS, Amazon EFS, Amazon RDS, Amazon DynamoDB, and Amazon OpenSearch Service.
  • Storage decision-making: Configure storage services based on cost, performance, data structure, and compliance rules.
  • Debugging ingestion & storage: Troubleshoot and debug data ingestion and storage capacity and scalability issues.
  • Streaming data ingestion: Use streaming data sources like Amazon Kinesis, Apache Flink, and Apache Kafka.
  • Optimized formats: Write and ingest data using Apache Parquet, JSON, CSV, and ORC based on access patterns.
  • Data merging: Merge data from multiple sources using code, AWS Glue, and Apache Spark.
  • Vector databases for AI: Configure scalable vector databases for RAG and GenAI applications (OpenSearch Service, Amazon RDS with pgvector, Amazon S3).
  • Multimodal data types: Ingest and store diverse data types including text, images, and audio.
  • Feature Store ingestion: Ingest streaming and batch data into SageMaker Feature Store.

• Task 1.2: Perform data transformation, feature engineering, and pre-processing
  • Data transformation tools: Transform data using AWS Glue, AWS Glue DataBrew, Spark on Amazon EMR, and SageMaker Data Wrangler.
  • Feature management: Create and manage online and offline features using SageMaker Feature Store.
  • Streaming data transformations: Transform streaming data via AWS Lambda and Spark.
  • Feature engineering techniques: Perform scaling, standardization, feature splitting, binning, log transformation, and normalization.
  • Embeddings for AI: Configure and use embedding models to convert text and image assets into numerical vector representations.
  • Text pre-processing: Apply advanced text pre-processing techniques (tokenization, domain-specific data augmentation).
  • Document preparation for RAG: Prepare documents for Retrieval Augmented Generation applications using chunking strategies and metadata extraction.
  • Data privacy & masking: Mask, redact, and anonymize sensitive PII/PHI data.
  • Data prep for FMs: Prepare datasets for FM fine-tuning, continuous pre-training, and model distillation.
• Task 1.3: Validate data quality and manage bias
  • Data quality validation: Validate data quality using AWS Glue Data Quality and DataBrew.
  • Data labeling: Label and annotate datasets using human workflows (Amazon SageMaker Ground Truth).
  • Bias identification & mitigation: Identify and mitigate dataset bias using splitting, shuffling, and data augmentation techniques.
  • Multimodal bias metrics: Optimize multimodal data distributions by applying bias metrics across numeric, text, and image assets.
  • Class imbalance resolution: Resolve class imbalance across numeric, text, and image datasets.
  • AI training data integrity: Validate prompt-response pair integrity and perform content safety screening.
  • Data cleaning: Clean data by detecting outliers, imputing missing values, and removing duplicates.

Domain 2: ML Model and Foundation Model (FM) Development (24%)

This domain covers the selection, training, and analysis of both traditional and foundation models.

• Task 2.1: Choose appropriate modeling approaches for ML and AI solutions
  • Bedrock FM selection: Evaluate and select appropriate FMs from Amazon Bedrock based on task requirements and performance criteria.
  • Fine-tuning strategy selection: Identify fine-tuning strategies for pre-trained FMs to satisfy business needs.
  • Model & algorithm comparison: Compare and select traditional ML algorithms, GenAI models, and solution templates based on interpretability, performance, and latency.
  • Architectural tradeoffs: Evaluate tradeoffs between custom models, managed services, pre-trained models, and FMs.
  • RAG architecture patterns: Select Retrieval Augmented Generation (RAG) architecture patterns based on use case requirements.
  • Cost vs. performance tradeoffs: Assess tradeoffs between ML model accuracy/performance, training time, and cost.
  • Latency vs. cost tradeoffs: Assess tradeoffs between AI model performance, latency, and operational cost.
  • AWS AI services: Apply turnkey AWS AI services (Amazon Textract, Amazon Rekognition, Amazon Comprehend, Amazon Transcribe) for domain problems.
• Task 2.2: Train, fine-tune, and customize models for ML and AI solutions
  • Built-in algorithms & frameworks: Apply SageMaker AI built-in algorithms and standard ML libraries.
  • SageMaker Script Mode: Configure SageMaker AI script mode with supported frameworks (PyTorch, TensorFlow).
  • Hyperparameter optimization: Implement automatic model tuning (AMT) and hyperparameter search pipelines.
  • Training acceleration: Implement early stopping and distributed training strategies to reduce training duration.
  • Overfitting & catastrophic forgetting: Prevent overfitting, underfitting, and catastrophic forgetting during tuning.
  • Model ensembling: Combine multiple ML models to improve performance or reduce inference cost.
  • Hyperparameter tuning: Adjust fundamental hyperparameters including epoch, step count, and batch size.
  • Customization & prompt engineering: Apply AI customization techniques including task-specific prompt engineering and fine-tuning.
  • Retrieval & embedding optimization: Optimize retrieval components and embedding models for RAG architectures.
• Task 2.3: Analyze and evaluate the performance of ML and AI systems
  • Experiment tracking: Perform reproducible experiments using MLflow on SageMaker AI, Amazon Bedrock evaluations, and Bedrock Prompt Management.
  • Baselines & drift detection: Establish model performance baselines and implement drift detection.
  • Production variant testing: Compare performance of shadow variants against active production variants.
  • Explainability: Explain model outputs and detect post-training bias (using SageMaker Clarify).
  • Convergence debugging: Debug model convergence issues during training.
  • Evaluation techniques: Apply comprehensive evaluation techniques for both traditional ML models and GenAI models.
  • Human evaluation frameworks: Implement human-in-the-loop (HITL) workflows and text generation quality assessment.
  • NLP evaluation metrics: Apply NLP metrics including BLEU, ROUGE, BERTScore, and semantic similarity.
  • GenAI & LLM evaluation: Evaluate AI outputs using content quality validation, bias detection, and LLM-as-a-judge frameworks.
  • RAG system monitoring: Configure RAG system monitoring, including retrieval accuracy assessment.

Domain 3: Deployment and Orchestration of ML and AI Workflows (24%)

This domain focuses on moving models into production and automating the lifecycle.

• Task 3.1: Manage deployment infrastructure for ML and AI model types
  • Compute & target selection: Select compute environments and target platforms (real-time, batch, asynchronous, serverless).
  • Multi-model strategies: Select deployment orchestrators and multi-model/multi-container endpoint strategies.
  • Inference pattern selection: Choose model inference strategies based on real-time vs. batch latency constraints.
  • FM deployment selection: Evaluate and select appropriate foundation model (FM) hosting options.
  • External model import: Deploy third-party or custom models built outside of AWS into SageMaker AI and Bedrock Custom Model Import.
  • Agent deployment & protocols: Deploy and configure AI agents for specific tasks, service integrations, and communication protocols.
  • FM resource allocation: Configure FM deployment, model hosting, and GPU/CPU resource allocation.
  • RAG system configuration: Apply RAG system configurations including retrieval strategies and reranking models.
• Task 3.2: Provision and configure resources for ML and AI workloads based on existing architecture and requirements
  • Provisioning optimization: Balance on-demand vs. provisioned resources for cost efficiency and speed.
  • Automated stack provisioning: Automate infrastructure provisioning with integrated stack communications.
  • Container management: Build and maintain OCI-compliant container images for ML and AI workloads.
  • VPC network isolation: Configure SageMaker AI endpoints securely inside VPC private subnets.
  • Programmatic deployment: Deploy and host models programmatically using the SageMaker Python SDK, AWS CLI, and Boto3.
  • Auto scaling configuration: Select specific performance metrics for endpoint Auto Scaling.
  • Bedrock Knowledge Bases: Create and manage Bedrock knowledge bases with vector database configurations, document indexing, and retrieval optimization.
  • Retrieval pipeline implementation: Implement custom retrieval pipelines to satisfy business SLA requirements.
  • Agent state management: Implement state management systems for complex agentic workflows.
  • GPU resource scaling: Implement AI-specific resource scaling tailored to GPU workloads.
  • Agentic workflow infrastructure: Deploy and maintain infrastructure for agentic workflows.
• Task 3.3: Implement automated orchestration and continuous integration and continuous delivery (CI/CD) pipelines for MLOps and AI workloads
  • Deployment strategies: Implement automated deployment strategies (Blue/Green, Canary) and automated rollback actions.
  • AWS Developer Tools: Configure and troubleshoot AWS CodeBuild, AWS CodeCommit, AWS CodeDeploy, AWS CodePipeline, and AWS CodeConnections.
  • Job orchestration: Configure automated training, processing, and inference jobs.
  • Automated testing in CI/CD: Configure automated unit and integration testing strategies within MLOps pipelines.
  • Automated retraining: Build and integrate automated triggers to re-train models based on data/performance thresholds.
  • Model Versioning & Registry: Manage model versions for auditability using SageMaker Model Registry and MLflow on SageMaker AI.
  • Prompt Management: Manage versioning and testing of prompts using Amazon Bedrock Prompt Management.
  • Agent deployment pipelines: Implement automated agent deployment pipelines and agent version management.
  • AI testing frameworks: Implement AI model testing frameworks, including prompt testing.
  • FM deployment automation: Configure FM deployment automation with fine-tuned model versioning.
  • Knowledge Base refresh pipelines: Configure AI-specific pipeline orchestration for RAG system updates and knowledge base refresh cycles.

Domain 4: Operating, Monitoring, and Securing ML and AI Solutions (24%)

The final domain ensures the long-term health, cost-efficiency, and security of ML systems.

• Task 4.1: Monitor ML and AI model inference and performance
  • Production observability: Monitor model performance in production using Amazon CloudWatch generative AI observability, Amazon Bedrock Model Evaluation, and drift detection pipelines.
  • Anomaly & error detection: Monitor inference workflows to detect data drift, concept drift, or pipeline errors.
  • Data distribution shifts: Detect changes in input data distribution that impact accuracy.
  • A/B testing: Monitor live model performance in production using A/B testing and traffic splitting.
  • Agent coordination monitoring: Monitor and automate management of agent coordination (detecting coordination failures, truncated streaming, or tool call errors).
  • FM performance evaluation: Configure AI-specific performance monitoring for foundation models using Amazon Bedrock evaluations.
• Task 4.2: Optimize and manage ML and AI infrastructure costs and performance
  • Instance selection: Select optimal inference instance families (CPU, GPU, Inferentia, Graviton) for cost and throughput.
  • Observability & troubleshooting: Use CloudWatch, Amazon Bedrock AgentCore Observability, and AWS X-Ray to diagnose bottlenecks.
  • Monitoring dashboards: Set up CloudWatch and QuickSight dashboards to track operational metrics.
  • Capacity optimization: Optimize instance capacity for reliability, latency, and cost.
  • Cost controls & quotas: Set up cost quotas and manage spend using AWS Cost Explorer and Budgets.
  • Purchasing options: Reduce compute costs using Savings Plans, Reserved Instances, and Spot Instances.
  • FM inference cost analysis: Evaluate cost implications of deploying FMs for inference in production.
  • Agent resource patterns: Track agent resource consumption patterns and execution efficiency.
  • FM usage optimization: Manage FM inference costs via usage optimization and prompt tokens management.
  • AI-specific cost tracking: Monitor token usage, embedding computation costs, and vector database storage expenses.
• Task 4.3: Secure ML and AI workloads and model endpoints
  • Pipeline vulnerability scanning: Secure CI/CD pipelines by scanning for code and container image vulnerabilities (Amazon CodeGuru, Amazon Inspector).
  • Least privilege access: Configure fine-grained least privilege access to ML artifacts, models, and datasets.
  • IAM configuration: Configure IAM policies, execution roles, and resource controls for users and services.
  • Auditability & compliance: Configure CloudTrail, AWS Config, and AWS Trusted Advisor for auditing and compliance.
  • Security troubleshooting: Troubleshoot and debug IAM, network, or policy security issues.
  • Network isolation: Create VPCs, private subnets, security groups, and VPC endpoints for isolated network security.
  • Vulnerability mitigation: Identify and mitigate security risks and prompt injection vulnerabilities in AI systems.
  • Credential management: Select appropriate authentication mechanisms for FMs (Bedrock API keys, IAM credentials, Secrets Manager).
  • Guardrails & responsible AI: Implement safeguards and sensitive data protection (PII masking, toxicity filtering, hallucination guardrails) using Amazon Bedrock Guardrails.