How are word embeddings actually trained, and why do they end up meaningful?medium
Embeddings start as random vectors. As the model trains on a prediction task, wrong predictions produce a loss, backpropagation updates every weight including the embedding vectors, and this repeats across millions or billions of examples. Words that keep appearing in similar contexts keep getting nudged in similar directions, so their vectors converge to similar positions — meaning emerges from distributional patterns, not manual labeling.
The flow is: input text → convert tokens to embeddings → model predicts something → compare to the correct answer → compute loss → backpropagate → update embeddings and model weights together. Nothing about "doctor is similar to physician" is hand-coded anywhere in this loop. It emerges because sentences like "the doctor treated the patient," "the nurse treated the patient," and "the physician treated the patient" repeatedly place doctor, nurse, and physician in the same distributional slot, so gradient updates keep pushing their vectors in similar directions across enough examples. This is the practical form of the distributional hypothesis — "you shall know a word by the company it keeps" — turned into an actual training signal.