Datasets, weight tables, and linear transformations
Your entire training dataset is a matrix. Every weight layer in a neural network is a matrix. Every batch of data you pass through a model gets multiplied by weight matrices. Understanding matrices — what they represent, how they transform data — is non-negotiable for understanding how models work.
A matrix is a function that transforms vectors. Multiply a vector by a matrix and you get a new vector — possibly in a different dimensional space, stretched, rotated, or projected.
A 2D array of numbers with m rows and n columns — an m×n matrix. A dataset X of 1000 samples with 20 features each is a (1000, 20) matrix. A linear layer mapping 20 inputs to 512 outputs has a weight matrix W of shape (512, 20).
Flips a matrix over its diagonal — rows become columns. An m×n matrix becomes n×m.
In attention, Q · Kᵀ requires transposing K so dimensions align. In PCA, you compute XᵀX to get the covariance matrix.
Multiplying matrix A (m×n) by vector x (n×1) gives a new vector (m×1). Each row of A computes one output value as a dot product with x.
A neural layer: output = W·input + bias. Stack multiple layers and each learns a progressively more abstract transformation.
The inverse A⁻¹ (only for square matrices) satisfies A·A⁻¹ = I. If A transforms a vector, A⁻¹ undoes it. The closed-form linear regression solution is θ = (XᵀX)⁻¹Xᵀy — in practice you use solvers, not direct inversion.
The determinant measures how a matrix scales space. det = 0 means the matrix collapses space to a lower dimension — singular, not invertible, information is lost.