Area Under the ROC Curve (AUC) Cheat Sheet

• AUC, short for Area Under the Curve of the Receiver Operating Characteristic (ROC), is a metric that evaluates how well a model can differentiate between different classes.
• A performance metric primarily used for binary classification models.
• Ranges from 0 to 1:
  • 1: Perfect model.
  • 0.5: Model performs no better than random guessing.
  • 0: Model inversely ranks positives and negatives.

Accurate Outcomes

• True Positive (TP): The model predicted 1, which matches the actual result.
• True Negative (TN): The model predicted 0, and the real outcome was also 0.

Misclassified Outcomes

• False Positive (FP): The model predicted 1, but the true result was 0.
• False Negative (FN): The model predicted 0, though the actual value was 1.

What is the ROC Curve?

• ROC Curve plots:
  • True Positive Rate (TPR): Sensitivity or Recall.
  • False Positive Rate (FPR): 1 – Specificity.
• Shows model performance across all classification thresholds.
• The optimal ROC curve closely follows the top-left boundary (point 0,1), signifying a high true positive rate and a low false positive rate.

Key Metrics for Model Evaluation

Accuracy

• Definition: The ratio of correct predictions out of all correct and incorrect predictions, used in both binary and multiclass classification. It ranges from 0 (completely wrong) to 1 (perfectly accurate).
• Formula:

Precision

• Definition: Measures the proportion of true positives among all positive predictions made by the algorithm.
• Formula:

Recall (Sensitivity or True Positive Rate)

• Definition: The proportion of actual positives that are correctly identified by the model.
• Formula:​

False Positive Rate (FPR)

• Definition: The proportion of actual negatives that are incorrectly classified as positive.
• Formula:​

Why AUC Matters?

• Threshold-agnostic: Assesses performance across every possible decision boundary.
• Handles imbalance well: Remains effective even when class distributions are uneven.
• Model-agnostic comparison: Enables fair evaluation of models regardless of their chosen thresholds.

Interpreting AUC Values

• 0.9 – 1.0: Excellent model.
• 0.8 – 0.9: Good model.
• 0.7 – 0.8: Fair model.
• 0.6 – 0.7: Poor model.
• 0.5: No better than random guessing.
• < 0.5: Worse than random guessing.

Limitations of AUC

• Does not account for precision or negative predictive value.
• Can give a false sense of performance on imbalanced data.
• Does not reflect the calibration of predicted probabilities.

When to Use AUC

• Comparing models to evaluate overall performance.
• Handling imbalanced classes where accuracy may be misleading.
• Choosing thresholds to find the optimal classification cutoff.

References:

https://docs.aws.amazon.com/machine-learning/latest/dg/binary-model-insights.html
https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-metrics-validation.html