Optimization Algorithms

Definition

Iterative algorithms that update model parameters to minimize a training loss. They differ in how the gradient signal is used: how many examples contribute per step, whether past gradients are accumulated, and whether per-parameter learning rates are adapted.

Intuition

Imagine navigating a hilly loss surface. At each step you estimate the local slope and take a step downhill. The key questions are: how accurate is that slope estimate (batch size), how fast do you move (learning rate), and do you build up momentum to glide through valleys (adaptive methods)?

Momentum smooths out oscillations by accumulating a velocity vector — frequent gradient directions build speed; noisy orthogonal fluctuations cancel out.

RMSProp equalizes update sizes across parameters by dividing by the recent RMS of each parameter’s gradient — parameters with large recent gradients get smaller steps and vice versa.

Adam combines both: a momentum term for direction, an RMSProp term for scale normalization, and bias correction for the cold-start problem of zero-initialized moment estimates.

Formal Description

Batch Variants of Gradient Descent

Let $J(\theta )$ be the loss. The general update rule is $\theta \leftarrow \theta -\alpha {\nabla }_{\theta }J$.

Variant	Batch size $B$	Notes
Full-batch GD	$B=m$	Exact gradient; slow, impractical for large data
Stochastic GD (SGD)	$B=1$	Very noisy; many updates per epoch; no GPU parallelism benefit
Mini-batch GD	$1<B<m$	Standard; typical $B\in \{32,64,128,256\}$ (powers of 2 for GPU efficiency)

Smaller batches act as a regularizer (noise); larger batches converge to sharper minima that may generalize worse.

---

Learning Rate Schedules

Decaying $\alpha$ over training improves final convergence:

• Step decay: ${\alpha }_t={\alpha }_0\cdot {\gamma }^{\lfloor t/k\rfloor }$
• Exponential decay: ${\alpha }_t={\alpha }_0\cdot e^{-\lambda t}$
• Cosine annealing: ${\alpha }_t={\alpha }_{\min }+\frac{1}{2}({\alpha }_{\max }-{\alpha }_{\min })\left(1+\cos \frac{\pi t}{T}\right)$

Linear warmup (ramp $\alpha$ from 0 over the first few thousand steps) is standard before large-scale training to avoid early instability.

---

Momentum

$$v_t\leftarrow {\beta }_1{v}_{t-1}+(1-{\beta }_1)g_t,\theta \leftarrow \theta -\alpha v_t$$

Typical ${\beta }_1=0.9$. Initialise $v_0=0$.

---

RMSProp

$$s_t\leftarrow {\beta }_2{s}_{t-1}+(1-{\beta }_2)g_t^2,\theta \leftarrow \theta -\alpha \frac{g_t}{\sqrt{s_t+ϵ}}$$

Typical ${\beta }_2=0.999$, $ϵ=10^{-8}$.

---

Adam (Adaptive Moment Estimation)

Maintain 1st (momentum) and 2nd (RMSProp) moment estimates with bias correction:

$$v_t\leftarrow {\beta }_1{v}_{t-1}+(1-{\beta }_1)g_t$$ $$s_t\leftarrow {\beta }_2{s}_{t-1}+(1-{\beta }_2)g_t^2$$ $${\hat{v}}_t=\frac{v_t}{1-{\beta }_1^t},{\hat{s}}_t=\frac{s_t}{1-{\beta }_2^t}$$ $$\theta \leftarrow \theta -\alpha \frac{{\hat{v}}_t}{\sqrt{{\hat{s}}_t}+ϵ}$$

Default hyperparameters: ${\beta }_1=0.9$, ${\beta }_2=0.999$, $ϵ=10^{-8}$, $\alpha =10^{-3}$.

AdamW is a variant that correctly decouples weight decay from the adaptive update (standard in transformer training).

Applications

• Adam/AdamW: default for transformers, NLP, most research
• SGD + momentum: still competitive in computer vision; can reach lower test error with careful LR schedule tuning
• RMSProp: historically popular for RNNs; largely superseded by Adam

Trade-offs

• Generalization gap: Adam can generalize worse than SGD+momentum on some image benchmarks due to convergence to sharper minima
• Memory: Adam requires two extra moment vectors per parameter — significant for very large models
• Epsilon sensitivity: too small $ϵ$ causes instability when ${\hat{s}}_t\approx 0$; too large dampens adaptation
• LR schedules still matter even with Adam; the adaptive rate does not eliminate the need for a good base LR

Links

• Hyperparameter Tuning
• Regularization
• Modeling
• Gradient Descent and Variants — SGD, momentum, schedules
• Adaptive Optimizers — Adam, AdamW, RMSProp
• Convex Optimization — loss landscape theory