Optimization

Mathematical theory of finding minima and maxima of functions, with and without constraints. Answers the question: how do we find the parameters that minimize a loss function?

Notes

• Convex Optimization — convex sets, convex functions, KKT conditions, duality, common ML problems
• Gradient Descent and Variants — batch GD, SGD, momentum, Adam, AdamW, learning rate schedules
• Lagrangian and Constrained Optimization — Lagrange multipliers, KKT, dual problem, sensitivity, SVM dual

Links

← 03 — Probability & Statistics → 05 — Statistical Learning Theory

• Optimization Algorithms (Modeling)
• Gradient Descent (DL Theory)