Projections

The closest point in a subspace to a given vector, found by dropping a perpendicular.

Projecting onto a line

The projection of a vector b onto the line through a nonzero vector a is the point on that line closest to b. It equals (a . b)/(a . a) times a. The difference b minus its projection is orthogonal to a; this residual is what the projection removes. Projecting onto a unit vector u simplifies to (u . b) u.

Projecting onto a subspace

To project b onto the column space of a matrix A with independent columns, solve the normal equations A^T A x = A^T b for the coefficients x; the projection is A x. The projection matrix is P = A (A^T A)^{-1} A^T, which maps any vector to its closest point in the column space and satisfies P^2 = P and P^T = P.

Properties of projection matrices

• idempotent: applying twice equals applying once, P^2 = P
• symmetric for orthogonal projections: P^T = P
• eigenvalues are only 0 and 1
• I - P projects onto the orthogonal complement

The link to least squares

When Ax = b has no exact solution, the best approximate solution makes Ax as close to b as possible, which means Ax must be the projection of b onto the column space. This is exactly the least-squares solution, so projection and least-squares fitting are two views of the same computation.

import numpy as np
A = np.array([[1.0, 0.0], [1.0, 1.0], [1.0, 2.0]])
b = np.array([1.0, 2.0, 2.0])
P = A @ np.linalg.inv(A.T @ A) @ A.T
print(P @ b)   # projection of b onto column space of A

Projection underlies model reduction: projecting the full state of a physics simulation onto a low-dimensional subspace of dominant modes yields a smaller system that captures the essential dynamics.