06 — Training and Regularization

Loss functions, optimisation algorithms, regularisation strategies, hyperparameter search, and early stopping — the decisions that govern how a model is trained.

Notes

• Optimization Algorithms — SGD, momentum, RMSProp, Adam, learning rate schedules, adaptive methods
• Loss Functions — MSE, MAE, Huber, cross-entropy, hinge, focal loss, task–loss alignment
• Regularization — L1/L2 penalties, weight decay, dropout, data augmentation, early stopping
• Early Stopping — validation monitoring, patience, checkpointing, overfitting detection
• Hyperparameter Tuning — grid search, random search, Bayesian optimisation, LR finder, Hyperband

Links

← 05 — Time Series → 07 — Evaluation and Model Selection

• Optimization Theory
• Deep Learning Theory