The engine of similarity, attention, and linear layers
The dot product is the single most used operation in ML. Every linear layer in a neural network is a dot product. Every attention score in a Transformer is a dot product. Every cosine similarity computation involves a dot product. If you understand nothing else from linear algebra, understand this.
The dot product measures how much two vectors agree. Think of two people each holding opinions on two topics — dimension by dimension, you multiply their stances and sum. Mostly agree → large positive. Completely disagree → large negative. Totally unrelated → zero.
Two people voting on X and Y. Person A: [+4, +1]. Person B: [+3, +3]. Dot product: (4×3) + (1×3) = 15 — they mostly agree. Flip B to [−3, −3]: (4×−3) + (1×−3) = −15 — they completely disagree. Now A = [0, 5] and B = [5, 0]: (0×5) + (5×0) = 0 — they care about entirely different things, zero overlap.
Multiply corresponding elements and sum:
Example: a = [1, 2, 3], b = [4, 5, 6]
Second formula: a · b = ||a|| × ||b|| × cos(θ), where θ is the angle between the vectors. Large when aligned (θ→0°, cos→1), zero when perpendicular (θ=90°), negative when opposite (θ=180°).
Normalizes the dot product to remove magnitude effects:
Result is always in [−1, 1]. Used everywhere in NLP — comparing sentence embeddings, finding similar documents, semantic search. This is how vector databases (Pinecone, Weaviate) do retrieval.
Every neuron computes output = w · x + b — a dot product of the weight vector w with input x. A layer with 512 neurons computes 512 dot products simultaneously, organized as a matrix multiply.
In Transformer attention: score = Q · Kᵀ — every query vector dot-producted with every key vector. High score = "this query should attend to this key."