07 — Evaluation and Model Selection

Measuring model performance, selecting between model candidates, and assessing model risk — from cross-validation and metric choice through to interpretability tools and governance.

Notes

• Evaluation and Validation — K-fold CV, temporal CV, classification/regression metrics, ROC/PR-AUC, calibration, model selection
• SHAP and Feature Attribution — Shapley values, TreeSHAP, DeepSHAP, LIME, integrated gradients, permutation importance
• PDP, ICE, and ALE — partial dependence plots, individual conditional expectation, accumulated local effects
• Model Risk Considerations — SR 11-7, EU AI Act, model validation framework, concept drift, governance
• Interpretability — Implementation (SHAP, PDP, Permutation Importance) — SHAP TreeExplainer/KernelExplainer, waterfall/beeswarm plots, PartialDependenceDisplay, permutation importance

Links

← 06 — Training and Regularization → Interpretability and Communication

• Statistical Learning Theory
• Monitoring and Observability