Offline vs online evaluation for LLM systems?medium
Answer
Offline: evaluate on static held-out dataset before deployment — controlled, reproducible, cheap.
Explanation
Offline: evaluate on static held-out dataset before deployment — controlled, reproducible, cheap. Online: evaluate in production — shadow mode (run new model without serving output), A/B testing (split traffic), monitoring (track user feedback, error rates). Both needed: offline gates deployment, online catches distribution shift and real-world edge cases.
Follow-upWhen would you choose one approach over the other?