spectral_theorem

Spectral Theorem and Symmetric Matrices

Definition

A real symmetric matrix $A=A^{\top }\in {\mathbb{R}}^{n\times n}$ is always orthogonally diagonalisable:

$$A=Q\Lambda Q^{\top }$$

where $Q$ is orthogonal ($Q^{\top }Q=I$, columns are orthonormal eigenvectors) and $\Lambda =\text{diag}({\lambda }_1,\dots ,{\lambda }_n)$ contains real eigenvalues. This is the spectral theorem for real symmetric matrices.

Intuition

Symmetric matrices arise whenever a quantity is “self-adjoint” — e.g., covariance matrices, Hessians, graph Laplacians. The spectral theorem says these matrices are always nicely diagonalisable: their eigenvectors are mutually perpendicular and span the whole space, and all eigenvalues are real. There’s no rotation; the matrix simply stretches along orthogonal axes.

Formal Description

Spectral decomposition:

$$A=\sum _{i=1}^n{\lambda }_i{q}_i{q}_i^{\top }$$

The outer products $q_i{q}_i^{\top }$ are rank-1 orthogonal projection matrices. $A$ is a weighted sum of projections onto its eigendirections, with eigenvalues as weights.

Positive semidefinite (PSD) matrices:

$A$ is PSD if ${\mathbf{x}}^{\top }A\mathbf{x}\ge 0$ for all $\mathbf{x}\in {\mathbb{R}}^n$. Equivalent conditions:

• All eigenvalues ${\lambda }_i\ge 0$
• $A=B^{\top }B$ for some matrix $B$ (Cholesky-like factorisation)
• All principal minors are non-negative

Positive definite (PD): ${\mathbf{x}}^{\top }A\mathbf{x}>0$ for all $\mathbf{x}\ne 0$; all eigenvalues $>0$; $A$ is invertible.

Covariance matrices $\Sigma =\frac{1}{n}{X}^{\top }X$ (centred data) are always PSD: for any $v$, $v^{\top }\Sigma v=\frac{1}{n}\Vert Xv{\Vert }^2\ge 0$.

Cholesky decomposition: every PD matrix $A$ factors as $A=L{L}^{\top }$ where $L$ is lower-triangular with positive diagonal. More numerically stable than eigendecomposition for solving linear systems.

Functions of symmetric matrices: via spectral decomposition, matrix functions are well-defined:

$$f(A)=Q\text{diag}(f({\lambda }_1),\dots ,f({\lambda }_n))Q^{\top }$$

Examples: $A^{1/2}=Q{\Lambda }^{1/2}{Q}^{\top }$ (matrix square root), $e^A=Q{e}^{\Lambda }{Q}^{\top }$.

Rayleigh quotient:

$$R(A,\mathbf{x})=\frac{{\mathbf{x}}^{\top }A\mathbf{x}}{{\mathbf{x}}^{\top }\mathbf{x}}$$

${\lambda }_{\min }\le R(A,\mathbf{x})\le {\lambda }_{\max }$; the maximum is attained at the leading eigenvector (used in power iteration and PCA).

Applications

Concept	Role of spectral theorem
PCA	Eigendecomposition of covariance matrix; eigenvectors = principal directions
Kernel methods	Kernel matrix $K_{ij}=k(x_i,x_j)$ is PSD; Mercer’s theorem guarantees spectral expansion
Quadratic forms in optimization	Hessian $H={\nabla }^{2}f$ is symmetric; PD Hessian ↔ strictly convex, unique minimum
Graph Laplacian	$L=D-W$ symmetric PSD; eigenvalues encode connectivity
Gaussian distributions	Covariance $\Sigma$ must be PSD for $\mathcal{N}(\mu ,\Sigma )$ to be valid

Trade-offs

• The spectral theorem only applies to symmetric (or normal) matrices; general square matrices require Jordan normal form, not a clean eigendecomposition.
• Numerical eigendecomposition of symmetric matrices is well-conditioned and reliable; use numpy.linalg.eigh (not eig) for symmetric matrices — it’s faster and guarantees real eigenvalues.
• PSD testing in practice: add small $ϵI$ (Tikhonov regularisation) to handle numerical near-singularity.

Links

• Eigenvalues and Eigenvectors
• Singular Value Decomposition — SVD of symmetric PSD $A$ coincides with eigendecomposition
• Orthogonal Projections
• Convex Optimization — PD Hessian ↔ strictly convex function
• Multivariate Gaussian (covariance must be PSD)