Matrix Operations

Definition

Matrix operations include addition, scalar multiplication, matrix-matrix multiplication, matrix-vector multiplication, and the transpose. They form the algebraic foundation for linear algebra.

Intuition

Matrix–matrix multiplication computes all dot products between rows of the left matrix and columns of the right matrix — think of it as composing two linear transformations. The transpose geometrically “reflects” a matrix across its diagonal. The Hadamard product is purely element-wise and has no linear-map interpretation on its own.

Formal Description

Matrix Addition

Matrices can be added only if they have the same dimensions; addition is element-wise:

(a c b d) + (e g f h) = (a + e c + g b + f d + h)

i.e. $C_{i, j} = A_{i, j} + B_{i, j}$ .

Scalar Multiplication

Every element of the matrix is multiplied by the scalar $k$ :

k (a c b d) = (ka k c kb k d)

Matrix–Matrix Multiplication

An $m \times n$ matrix $A$ can be multiplied by an $n \times p$ matrix $B$ ; the result $C = AB$ is $m \times p$ :

C_{i, j} = k = 1 \sum n A_{i, k} B_{k, j}

Example ( $2 \times 2$ ):

(a c b d) (e g f h) = (a e + b g ce + d g a f + bh c f + d h)

The Hadamard product (element-wise product) is denoted $A ⊙ B$ and is distinct from standard matrix multiplication.

Matrix–Vector Multiplication

For $A \in R^{m \times n}$ and $x \in R^{n}$ , the product $Ax = b$ with $b \in R^{m}$ represents a system of linear equations.

The dot product of two vectors $x, y \in R^{n}$ is the matrix product $x^{⊤} y$ .

Transpose

The transpose $A^{⊤}$ switches rows and columns: $(A^{⊤})_{i, j} = A_{j, i}$ . For an $m \times n$ matrix, $A^{⊤}$ is $n \times m$ .

A = a_{11} a_{21} ⋮ a_{m 1} a_{12} a_{22} ⋮ a_{m 2} \dots \dots ⋱ \dots a_{1 n} a_{2 n} ⋮ a_{mn} ⟹ A^{⊤} = a_{11} a_{12} ⋮ a_{1 n} a_{21} a_{22} ⋮ a_{2 n} \dots \dots ⋱ \dots a_{m 1} a_{m 2} ⋮ a_{mn}

Matrix multiplication properties:

Distributive: $A (B + C) = AB + AC$
Associative: $A (BC) = (AB) C$
Not commutative in general: $AB \neq = BA$

Transpose properties:

$(A^{⊤})^{⊤} = A$
$(A + B)^{⊤} = A^{⊤} + B^{⊤}$
$(AB)^{⊤} = B^{⊤} A^{⊤}$
Dot product commutativity: $x^{⊤} y = y^{⊤} x$

Symmetry:

$A$ is symmetric if $A^{⊤} = A$ .
$A$ is skew-symmetric if $A^{⊤} = - A$ (diagonal elements must be zero).

Examples:

symmetric: a b c b d e c e f, skew-symmetric: 0 - b - c b 0 - e c e 0

Applications

Expressing systems of linear equations as $Ax = b$ , where $A \in R^{m \times n}$ , $b \in R^{m}$ , $x \in R^{n}$ .
Broadcasting in deep learning: $C_{i, j} = A_{i, j} + b_{j}$ adds a vector to every row of a matrix.

Trade-offs

Matrix multiplication is not commutative: $AB \neq = BA$ in general, even when both products are defined. Reversing operand order is a common source of bugs.
Dimensions must be compatible for multiplication ( $m \times n$ by $n \times p$ ); addition requires identical shapes.
The Hadamard product $A ⊙ B$ is commutative but does not correspond to function composition — mixing it with standard multiplication requires care.

Notes

Explorer

matrix_operations