The rank is the number of linearly independent rows (or columns) - the dimensionality of the column space. A rank-deficient matrix cannot be inverted and compresses information.

Rank equals the number of non-zero singular values. Full rank: all columns are independent, the matrix spans the full output space. Rank-deficient: some columns are linear combinations of others - information is lost in the transformation. In LoRA (Low-Rank Adaptation) for fine-tuning LLMs, weight updates are constrained to low-rank matrices - the assumption is that meaningful fine-tuning changes live in a low-dimensional subspace of the full parameter space.