Knowledge Storage

LLMs interview question · Medium difficulty

Where is knowledge stored inside an LLM, and why is it hard to edit?medium

Status
✓
Type
conceptual
Topic
knowledge-storage
Frequency
common
Tags
parameters, distributed-representations, interpretability
Answer

Knowledge lives in the model's parameters, but as distributed representations spread across many weights and layers, not as explicit records. You can't open the model and find a row that says "capital of India = New Delhi" — the same fact may be encoded across many weights simultaneously.

Explanation

The model learns patterns from training data and encodes them as high-dimensional representations rather than key-value entries. This is exactly what enables generalization — the model learns relationships, concepts, grammar, style, and reasoning patterns, not just isolated facts — but it's also why inspecting, editing, or deleting one specific fact is hard: correcting "Person X works at Company Y" may require changing weights that also encode many unrelated things, with no guarantee the edit won't have side effects elsewhere.

Follow-upWhy does distributed representation help generalization but hurt editability?
Follow-upWhat would happen if you tried to zero out the weights responsible for one fact?