Principal Component Analysis

PCA finds orthogonal directions of maximum variance, giving a linear projection that compresses data with minimal loss.

Directions of variance

Principal component analysis (PCA) rotates the feature space to a new set of orthogonal axes, the principal components, ordered by how much variance they capture. Keeping the first few components projects high-dimensional data into a low-dimensional space that preserves most of its spread, a linear form of dimensionality reduction.

How it is computed

• Center the data by subtracting each feature's mean.
• Form the covariance matrix (or use the SVD of the data directly).
• Eigenvectors give the component directions; eigenvalues give the variance along each.
• Project data onto the top-k eigenvectors.

from sklearn.decomposition import PCA
p = PCA(n_components=0.95)      # keep 95% of variance
Z = p.fit_transform(X_scaled)  # standardize X first
print(p.explained_variance_ratio_)

Reading the result

The explained-variance ratio tells you how much information each component retains; a scree plot of these values shows where returns diminish. Because PCA relies on variance and distances, standardize features first, or components will chase whichever feature has the largest units.

Uses and limits

PCA denoises, visualizes, decorrelates features, and speeds downstream models by shrinking dimensionality, easing the curse of dimensionality. Its limits: it captures only linear structure and its components, being combinations of all features, can be hard to interpret. For nonlinear structure prefer t-SNE or UMAP for visualization, or kernel PCA.

In engineering, PCA on a bank of simulation outputs reveals the few dominant modes that drive most of the variation across a design sweep.