Where is knowledge stored inside an LLM, and why is it hard to edit?medium
Knowledge lives in the model's parameters, but as distributed representations spread across many weights and layers, not as explicit records. You can't open the model and find a row that says "capital of India = New Delhi" — the same fact may be encoded across many weights simultaneously.
The model learns patterns from training data and encodes them as high-dimensional representations rather than key-value entries. This is exactly what enables generalization — the model learns relationships, concepts, grammar, style, and reasoning patterns, not just isolated facts — but it's also why inspecting, editing, or deleting one specific fact is hard: correcting "Person X works at Company Y" may require changing weights that also encode many unrelated things, with no guarantee the edit won't have side effects elsewhere.