Gradient Checking

Definition

Numerically approximates partial derivatives using the centered difference formula and compares them against analytical gradients produced by backpropagation. Used exclusively as a debugging tool to verify a backprop implementation is correct.

Intuition

Backpropagation is easy to implement incorrectly — sign errors, missing terms, wrong tensor shapes, and off-by-one errors all produce gradients that look plausible but are subtly wrong. Gradient checking acts as a unit test: if the numerical and analytical gradients agree to high precision, the implementation is almost certainly correct.

Formal Description

Centered difference approximation for parameter ${\theta }_i$:

$$\frac{\partial J}{\partial {\theta }_i}\approx \frac{J(\theta +ϵe_i)-J(\theta -ϵe_i)}{2ϵ}$$

where $e_i$ is the $i$-th standard basis vector and $ϵ\approx 10^{-7}$.

The centered difference has $O({ϵ}^2)$ error vs. $O(ϵ)$ for the one-sided difference, making it significantly more accurate.

Relative error metric:

$$\text{err}=\frac{\Vert {\nabla }_{\text{approx}}-{\nabla }_{\text{backprop}}\Vert }{\Vert {\nabla }_{\text{approx}}\Vert +\Vert {\nabla }_{\text{backprop}}\Vert }$$

Interpretation:

Relative error	Assessment
$<10^{-7}$	Great — implementation very likely correct
$10^{-7}$ – $10^{-5}$	Acceptable, borderline
$>10^{-5}$	Likely a bug in backprop

Applications

• Debugging new backpropagation implementations before training
• Catching sign errors, missing regularization terms, wrong matrix transpositions
• Verifying custom layers or loss functions in deep learning frameworks

Trade-offs

• Computational cost: requires $O(d)$ forward passes where $d$ is the number of parameters — completely impractical for training; use only on small models or a parameter subset
• Incompatible with dropout: stochastic layers produce different values on each forward pass, making the numerical approximation meaningless; disable dropout during checks
• Must include regularization: $J$ in the numerical approximation must match $J$ used in backprop exactly, including any $L_2$ penalty
• Numerical precision: very small $ϵ$ causes floating-point cancellation; very large $ϵ$ increases approximation error; $ϵ=10^{-7}$ is the standard choice

Links

• backpropagation
• gradient_descent