Embedding Training

NLP interview question · Medium difficulty

How are word embeddings actually trained, and why do they end up meaningful?medium

Status
✓
Type
conceptual
Topic
embedding-training
Frequency
common
Tags
embeddings, distributional-hypothesis, backpropagation, training
Answer

Embeddings start as random vectors. As the model trains on a prediction task, wrong predictions produce a loss, backpropagation updates every weight including the embedding vectors, and this repeats across millions or billions of examples. Words that keep appearing in similar contexts keep getting nudged in similar directions, so their vectors converge to similar positions — meaning emerges from distributional patterns, not manual labeling.

Explanation

The flow is: input text → convert tokens to embeddings → model predicts something → compare to the correct answer → compute loss → backpropagate → update embeddings and model weights together. Nothing about "doctor is similar to physician" is hand-coded anywhere in this loop. It emerges because sentences like "the doctor treated the patient," "the nurse treated the patient," and "the physician treated the patient" repeatedly place doctor, nurse, and physician in the same distributional slot, so gradient updates keep pushing their vectors in similar directions across enough examples. This is the practical form of the distributional hypothesis — "you shall know a word by the company it keeps" — turned into an actual training signal.

Follow-upWhy does this process require millions or billions of examples rather than a handful?
Follow-upWhat would happen to an embedding for a word that appears in training data only once or twice?
Follow-upHow is this the same underlying idea as Word2Vec, just described at the training-loop level?