Feature Stores

One definition of a feature, shared by training and serving

Why

The single most common silent failure in production ML is training-serving skew: the model was trained on a feature computed one way and served on the same feature computed a slightly different way. Accuracy in the notebook doesn't show up in production, and nobody can tell why.

Knowing what a feature store is and why you'd use one is a strong signal of hands-on production experience — most people learn it by getting burned first.

Intuition

Without a feature store, every team writes its own version of "average order value in the last 30 days" — once in a training notebook (batch SQL), once in the serving path (a live API call). They drift apart. A feature store is a shared library of feature definitions with two output modes: a bulk historical table for training, and a low-latency lookup for serving — computed from the same definition.

Explanation

Problems a feature store solves

Training-serving skew: training: batch job computes feature from full history serving: API computes "similar" feature from live data → subtly different numbers → model sees inputs it never trained on Redundant feature engineering: team A builds "user_avg_spend_30d" for churn model team B rebuilds the same feature for fraud model, slightly differently → wasted work, inconsistent definitions across the company

A feature store centralizes the definition once, computes it once, and serves it both ways — eliminating both problems at the source.

Offline vs online stores

Offline store Online store ------------------------------ ------------------------------ Backed by: data warehouse/lake Backed by: key-value store Access pattern: bulk, batch Access pattern: single-key lookup Used for: training datasets Used for: real-time inference Latency: seconds-minutes OK Latency: single-digit ms required Example: point-in-time joins Example: "features for user_id=42"

The feature store's job is keeping these two views consistent — same feature definition, same values, different access pattern for different consumers.

Examples

Feast — open-source, cloud-agnostic, plugs into existing warehouses and key-value stores. Tecton — managed, enterprise-focused, strong streaming feature support. Hopsworks — open-source with a built-in feature store UI and model registry. The choice matters less than the guarantee: one definition, two serving paths, point-in-time correctness for training data.

← Previous Next →