Principal Component Analysis (PCA)

Squeeze dozens of correlated features into a handful of ones that capture almost all the story.

Key Points

• Reduces the number of features while preserving as much of the original variance (information) as possible.
• New components are linear combinations of the original features, and are uncorrelated with each other.
• Ordered by importance — the first principal component captures the most variance, the second captures the next most, and so on.
• Requires features to be scaled first, since PCA is sensitive to the magnitude of each feature.
• The components themselves usually lose direct real-world interpretability.

What it is

PCA is a dimensionality reduction technique that transforms a dataset with many, possibly correlated features into a smaller set of new, uncorrelated features called principal components, ordered by how much variance (information) they capture from the original data.

Why it exists

Datasets with dozens or hundreds of features are slow to train on, prone to overfitting, and hard to visualize. Many of those features are also correlated and carry redundant information. PCA exists to compress that redundancy into fewer, information-dense dimensions without throwing away much of the signal.

Intuition

How it works

1. Standardize the data

   Scale all features to have mean 0 and unit variance so no feature dominates due to scale alone.

2. Compute the covariance matrix

   Measure how every pair of features varies together.

3. Find eigenvectors and eigenvalues

   Eigenvectors of the covariance matrix become the new principal component directions; eigenvalues tell you how much variance each captures.

4. Project the data

   Transform the original data onto the top-k principal components, keeping the ones that capture the most variance.

When to use

• Use before modeling when you have many correlated features and want to reduce dimensionality and training time.
• Use for visualizing high-dimensional data by projecting down to 2 or 3 components.
• Use to reduce noise in data before clustering or distance-based algorithms.

When not to use

• Do not use when interpretability of individual features is required — principal components are combinations, not the original features.
• Do not use on data that hasn't been scaled — results will be skewed by whichever feature has the largest raw magnitude.
• Do not use it as a blind first step for every project — check first whether feature reduction is actually needed.

Common mistakes

• Applying PCA before scaling features, letting large-magnitude features dominate the principal components.
• Assuming components are directly interpretable as real-world variables — they rarely are.
• Reducing to too few components and losing meaningful signal, without checking the explained variance ratio.

In practice

Used for visualization of high-dimensional data, noise reduction, speeding up downstream models, and as a preprocessing step before clustering when the raw feature count is very high.

Related concepts