Principal Component Analysis on Toy Data

Find the directions of maximum variance by taking the eigenvectors of the covariance matrix, and project to fewer dimensions.

The goal

PCA finds an orthogonal set of directions - principal components - ordered by how much variance the data has along each. Projecting onto the first few gives the best linear low-dimensional summary of the data in a least-squares sense.

The recipe

• Center the data by subtracting the mean of each feature.
• Form the covariance matrix (or use the SVD of the centered data).
• The eigenvectors are the principal directions; eigenvalues are the variances along them.
• Project the data onto the top-k eigenvectors.

import numpy as np
rng=np.random.default_rng(3)
t=rng.normal(0,1,300)
X=np.c_[t*2+rng.normal(0,0.2,300), t+rng.normal(0,0.2,300)]
Xc=X-X.mean(0)
cov=np.cov(Xc.T)
vals,vecs=np.linalg.eigh(cov)
order=np.argsort(vals)[::-1]
print('variance explained:',np.round(vals[order]/vals.sum(),3))
pc1=Xc@vecs[:,order[0]]        # projection onto top component
print('pc1 std:',round(pc1.std(),3))

Connection to the SVD

PCA is the SVD of the centered data matrix: the right singular vectors are the principal directions and the squared singular values are proportional to the variances. Computing PCA via SVD is more numerically stable than forming the covariance matrix explicitly, especially with many features.

Uses and cautions

PCA compresses data, denoises by dropping small-variance directions, and enables visualization in two dimensions. But it is linear and variance-driven: it can miss structure that is nonlinear or low-variance-but-important, and it requires standardizing features so that units do not dominate. Interpret components with care - they are combinations of original features, not always meaningful on their own.