Principal directions of a transformation — the heart of PCA
PCA — the most common dimensionality reduction technique — is entirely based on eigenvalues and eigenvectors of the covariance matrix, and knowing how it works mathematically matters as much as knowing what it does. Eigenvalues and eigenvectors also appear in graph neural networks, spectral clustering, and stability analysis of training dynamics.
Most vectors change direction when multiplied by a matrix — they get rotated and stretched. But certain special vectors don't change direction — they only get stretched or shrunk. These are eigenvectors. The amount of stretching is the eigenvalue.
For a square matrix A, a non-zero vector v is an eigenvector if multiplying A by v gives back v scaled by a constant λ (lambda):
v is the eigenvector. λ is the eigenvalue. A large λ means the matrix strongly stretches data in that direction. λ = 0 means the matrix collapses data in that direction — information is lost.
Interpretation: this matrix stretches x by 2× and y by 3×. The y-direction has more variance.
Step 1: Compute the covariance matrix C = XᵀX / n. This captures how each pair of features varies together.
Step 2: Find eigenvectors of C. Each eigenvector is a direction in feature space. Its eigenvalue = variance of data in that direction.
Step 3: Sort eigenvectors by eigenvalue (largest first). Take the top-k — these are your principal components.
Step 4: Project data onto these k eigenvectors.