Measuring the size of vectors and matrices
Any time you need to measure how "big" a vector or matrix is — how far it is from zero, how different two vectors are — you need a norm. Different norms measure differently, and choosing the right one changes what gets penalized and what doesn't.
A norm is a way to measure the "size" of a vector or matrix — how far it is from zero. L2 is straight-line distance: square each component, sum, take the root. L1 is taxicab distance: just sum the absolute values. Frobenius is L2 applied to every element of a matrix — flatten it, then take L2.
Sum of absolute values. Geometrically: the distance you'd travel on a grid — each step must be horizontal or vertical, no diagonals. All dimensions are treated equally regardless of magnitude.
Straight-line distance from the origin. Large components are penalized more heavily than small ones because of the squaring. Dividing a vector by its L2 norm gives a unit vector — direction preserved, magnitude set to 1.
The matrix equivalent of L2 — square every entry, sum them all, take the root. Equivalent to flattening the matrix into one long vector and computing its L2 norm. The third form connects to SVD: ‖A‖_F² equals the sum of squared singular values.
The two norms treat large vs small values differently: