Why attention, PCA, and weight initialization depend on perpendicularity
Orthogonality is the reason SVD's components don't interfere with each other, why PCA axes are independent, why attention heads can specialize, and why orthogonal weight initialization leads to better gradient flow. It ties the entire linear algebra section together — the singular vectors in SVD are orthogonal, eigenvectors of symmetric matrices are orthogonal, and PCA components are orthogonal by construction. Understanding projections lets you explain what it means to "project a query onto a key" in attention.
Two vectors are orthogonal when their dot product is zero — they share zero information. A projection is the shadow one vector casts onto another. When you project a high-dimensional point onto a lower-dimensional subspace, you're asking: "what's the closest point in that subspace?"
An orthogonal matrix Q preserves lengths and angles: ‖Qx‖ = ‖x‖ for all x. Multiplying by Q is a pure rotation — no stretching. This is why U and V in SVD are called "rotation matrices."
In attention: the dot product q·k measures how much of q "projects onto" the direction k — larger values mean higher similarity and more attention weight.
This is exactly what PCA does: project each data point onto the subspace spanned by the top-k principal components. The projected coordinates are uncorrelated because the columns of U_k are orthogonal.