Offline vs online evaluation for LLM systems?medium

Type
conceptual
Topic
offline-vs-online-evaluation-for-llm-systems
Frequency
common
Tags
mlops, offline, online, evaluation, for, llm
Answer

Offline: evaluate on static held-out dataset before deployment — controlled, reproducible, cheap.

Explanation

Offline: evaluate on static held-out dataset before deployment — controlled, reproducible, cheap. Online: evaluate in production — shadow mode (run new model without serving output), A/B testing (split traffic), monitoring (track user feedback, error rates). Both needed: offline gates deployment, online catches distribution shift and real-world edge cases.

Follow-upWhen would you choose one approach over the other?