Principal component analysis (PCA)

Rotates correlated numeric variables into uncorrelated components ordered by the variance they capture, to summarise or reduce many variables to a few.

\mathbf{R}\,\mathbf{v}_k = \lambda_k \mathbf{v}_k, \qquad \text{proportion}_k = \frac{\lambda_k}{\sum_j \lambda_j}, \qquad \text{score}_{ik} = \mathbf{z}_i^{\top}\mathbf{v}_k

Eigen-decomposition of the correlation (or covariance) matrix

$\mathbf{R}$: correlation matrix of the variables (covariance matrix when variables are only centred, not scaled)
$\lambda_k$: eigenvalue: variance of component $k$ (in decreasing order)
$\mathbf{v}_k$: unit-length loading vector of component $k$
$\mathbf{z}_i$: standardised values of observation $i$ (each variable minus its mean, divided by its $n - 1$ sd)

When to use

Many correlated numeric variables that you want to summarise, visualise in two dimensions, or reduce before another analysis.
Use the correlation matrix (scaled PCA) when variables are in different units; otherwise the variable with the largest variance dominates.

Assumptions

Numeric variables; linear relationships. Only complete cases are used (the engine reports how many rows were dropped).
PCA is sensitive to outliers and to scaling choices. It describes variance, not causes.

How to interpret

Proportion of variance: how much of the total variation each component captures; a scree plot helps choose how many to keep.
Loadings show which variables drive a component: similar-sign loadings = an overall "size" axis; opposite signs = a contrast.
Signs are arbitrary: software may flip a whole component (the engine makes each component's largest loading positive).

Common mistakes

Running PCA on unscaled variables with different units.
Naming components with more certainty than the loadings support.
Assuming components with small variance are unimportant for predicting an outcome.

Example

Six observations of three variables, a: 2, 3, 4, 5, 6, 7; b: 4, 5, 7, 8, 11, 12; c: 1, 2, 2, 4, 3, 5 (correlation-matrix PCA). Eigenvalues 2.829, 0.168, 0.002: PC1 explains 94.3% of the variance, PC1 + PC2 99.9%. PC1 loadings 0.59, 0.58, 0.56 — an overall "size" component on which all three variables move together.