System Design AI Interview Questions

Combine ingestion, retrieval, generation, evaluation, and monitoring.

A typical design includes document processing, embeddings, a vector store, reranking, prompt assembly, an LLM, citations, caching, and feedback loops.

Optimize retrieval, prompt size, model choice, caching, and streaming.

Latency comes from network calls, retrieval, reranking, model generation, and tool use. Smaller models and cached context can help.

Use one when semantic search over many embeddings must be fast and scalable.

Vector databases support approximate nearest neighbor search, metadata filtering, indexing, and operational management for retrieval systems.

Track token usage, model choice, caching, batching, and routing.

Cost-aware systems use cheaper models for simple tasks, reserve stronger models for hard cases, cache repeated outputs, and limit unnecessary context.

Use offline tests, online monitoring, user feedback, and safety checks.

Evaluation should measure answer quality, grounding, retrieval quality, latency, cost, safety, and task completion across realistic examples.

Isolate tenant data in ingestion, storage, retrieval filters, permissions, logging, and evaluation.

The main risks are cross-tenant data leakage and permission mistakes. Use tenant-scoped indexes or strict metadata filters, access checks, audit logs, and regression tests.

Use queues, backoff, caching, request shaping, fallback models, and clear user-facing degradation.

AI systems often depend on external model APIs with quota and latency limits. Good design avoids cascading failures and keeps critical paths responsive.

Prompt changes can alter behavior, so versions make releases traceable, testable, and reversible.

Store prompt templates, model settings, retrieval settings, and evaluation results together. Rollouts should compare quality, safety, latency, and cost.