Evaluation & Observability

Study offline evaluation (golden datasets, regression test suites), LLM-as-judge scoring, online evaluation (A/B tests, user feedback signals), and observability for multi-step LLM/agent systems — tracing a request across every model call, tool call, and retrieval step it touches.

Golden/labeled datasets as a fixed yardstick across model or prompt changes, LLM-as-judge for scoring outputs that are too open-ended for exact-match metrics, human-in-the-loop review for edge cases, spans/traces that link every step of a multi-hop chain back to one request, and alerting on regressions rather than only on hard failures.

An agent or RAG pipeline can fail in a step nobody's watching — a bad retrieval, a tool call that silently returns the wrong result, a prompt change that quietly regresses one use case while improving another. Without evaluation and tracing, "it seems fine" is the only signal you have, and it's the wrong one.

Expect questions on how you'd build a regression test suite for a prompt or model change, when LLM-as-judge is trustworthy versus when you need human review, and how you'd trace a bad output in a multi-step agent call back to the step that caused it.