Outer Product

Definition

The outer product of two column vectors produces a matrix; it is the reverse of the inner product in the sense that it expands rather than contracts dimensionality.

Intuition

Where the inner product asks “how aligned are these two vectors?” and returns a number, the outer product asks “what matrix captures every pairwise interaction between the components of these two vectors?” The result is always a rank-1 matrix — a single “direction” stretched across both vector spaces simultaneously. Sums of outer products build up higher-rank matrices and underlie SVD and low-rank approximations.

Formal Description

For $3\times 1$ column vectors $\mathbf{u},\mathbf{v}$:

$$\mathbf{u}{\mathbf{v}}^{\top }=\left(\begin{array}{c}{u}_1\\ u_2\\ u_3\end{array}\right)\left(\begin{array}{ccc}{v}_1& v_2& v_3\end{array}\right)=\left(\begin{array}{ccc}{u}_1{v}_1& u_1{v}_2& u_1{v}_3\\ u_2{v}_1& u_2{v}_2& u_2{v}_3\\ u_3{v}_1& u_3{v}_2& u_3{v}_3\end{array}\right)$$

• Every column of $\mathbf{u}{\mathbf{v}}^{\top }$ is a scalar multiple of $\mathbf{u}$; every row is a scalar multiple of ${\mathbf{v}}^{\top }$.
• The resulting matrix has rank at most 1.
• For vectors $\mathbf{u}\in {\mathbb{R}}^m$ and $\mathbf{v}\in {\mathbb{R}}^n$, the outer product $\mathbf{u}{\mathbf{v}}^{\top }\in {\mathbb{R}}^{m\times n}$.

Applications

• SVD expresses any matrix as a weighted sum of rank-1 outer products: $\mathbf{A}={\sum }_i{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^{\top }$.
• Low-rank approximations truncate this sum to compress matrices (image compression, collaborative filtering).
• Covariance matrices are sums of outer products of mean-centred data vectors: $\mathbf{\Sigma }=\frac{1}{n}{\sum }_i{\mathbf{x}}_i{\mathbf{x}}_i^{\top }$.

Trade-offs

• The outer product always yields a rank-1 matrix, so it cannot represent general linear maps on its own; a full-rank matrix requires at least $\min (m,n)$ outer products summed together.
• Unlike the inner product, the outer product is not commutative: $\mathbf{u}{\mathbf{v}}^{\top }\ne \mathbf{v}{\mathbf{u}}^{\top }$ in general (the two results have transposed shapes unless $m=n$).
• Memory cost is $O(mn)$ — for large vectors this is expensive compared to storing the two vectors separately ($O(m+n)$).

Links

• Inner Product
• Matrix Operations