NexForay

SVD — Singular Value Decomposition

Generalized decomposition — compression, PCA, and LoRA

Why

SVD is how modern systems compress information. Image compression, recommendation systems, and — critically for LLMs — LoRA fine-tuning all use SVD or its logic. It generalizes eigendecomposition to non-square matrices, which covers most ML weight matrices.

Intuition

SVD says: any transformation, no matter how complex, breaks into three clean steps — rotate, stretch, rotate again. The stretch in the middle tells you which directions carry real signal and which are just noise.

A 100×100 image looks like it needs 100 directions. SVD disagrees — only 5 actually matter:

100×100 ≈ 100×5 · 5×5 · 5×100

10,000 numbers down to 1,025. Ten times smaller, barely any loss — because you kept the loud directions and dropped the whispers.

Like a song with 100 instruments where only 5 carry the melody. Mute the rest. The song still sounds the same.

Explanation
The decomposition

Any matrix A of shape (m×n) factors as:

A = U · Σ · Vᵀ U: (m×m) — left singular vectors (orthogonal, directions in output space) Σ: (m×n) — diagonal matrix of singular values σ₁ ≥ σ₂ ≥ ... ≥ 0 Vᵀ: (n×n) — right singular vectors transposed (directions in input space)

When you compute y = Av, three things happen: Vᵀ rotates input, Σ stretches along each axis, U rotates into output space.

Truncated SVD — low-rank approximation

Keep only the top-k singular values to get the best rank-k approximation:

A_k = U_k · Σ_k · V_kᵀ where U_k is (m×k), Σ_k is (k×k), V_kᵀ is (k×n)

You've compressed the original m×n matrix into m·k + k + k·n numbers. The Eckart-Young theorem proves this is the best possible rank-k approximation — no other rank-k matrix is closer in Frobenius norm.

SVD vs eigendecomposition
  • Eigendecomposition only works on square matrices. SVD works on any matrix
  • For square symmetric matrices: singular values = absolute eigenvalues; U = V = eigenvectors
  • PCA on data matrix X = truncated SVD of X (equivalent to eigendecomposition of covariance matrix)
  • SVD is numerically more stable — used in all practical implementations of PCA
Where it shows up in ML
  • PCA: sklearn's PCA uses SVD internally, not eigendecomposition
  • Recommendation systems: matrix factorization — user-item rating matrix decomposed into low-rank factors
  • LoRA fine-tuning: represents weight updates as A·B (rank-r approximation). Instead of updating full W (d×d), you update A (d×r) and B (r×d) where r ≪ d
  • Image compression: SVD of pixel matrix, keep top-k singular values
← Previous Next →