ML Pipeline & Versioning

Data → features → training → evaluation → deployment → monitoring, with everything reproducible

Why

"It worked on my laptop" is not an answer production wants. If you can't reproduce a training run — same data, same code, same hyperparameters — you can't debug a regression, satisfy an audit, or safely roll back a bad model.

This is what separates people who trained a model once from people who've kept one alive: knowing which model is in production and how it was made requires understanding pipelines, versioning, and a registry.

Intuition

Treat a trained model like a compiled binary: it has source inputs (data version, code commit, hyperparameters) and it must be traceable back to them. The pipeline is the build process; experiment tracking is the build log; the model registry is the artifact store with a promotion gate (staging → production) instead of a plain file share.

Explanation
The ML pipeline stages

Every stage should be automated, versioned, and reproducible — a schema change upstream should trigger retraining and evaluation before anything reaches deployment.

raw data → feature engineering (transform, aggregate, join) → training (fit model on features + labels) → evaluation (metrics vs. holdout / thresholds) → deployment (push to serving infra) → monitoring (drift, latency, business metrics) ↓ triggers retraining when data/behavior shifts

Each arrow is a place teams get burned: unversioned features cause training-serving skew, unautomated evaluation lets a worse model slip into production, missing monitoring means nobody notices until users complain.

Experiment tracking

Every training run should record: hyperparameters, metrics, dataset version, and the code commit that produced it. Without this, "run #47 was better, but why?" is unanswerable.

run_id: 47 code_commit: a1b2c3d dataset_version: v2026-03-train hyperparams: {lr: 0.001, batch_size: 64, epochs: 10} metrics: {auc: 0.912, precision: 0.87, recall: 0.81}

Common tools: MLflow (open-source, self-hosted friendly), Weights & Biases (rich dashboards, team collaboration), Neptune (metadata-focused). The tool matters less than the discipline of logging every run.

Model registry: staging → production

The registry is version control for trained models — it stores artifacts and metadata, and promotes models through named stages instead of leaving "which model is live?" as a Slack question.

Registered Model: fraud-detector v3 stage=Archived (replaced, kept for rollback) v4 stage=Production (currently serving traffic) v5 stage=Staging (passed eval, awaiting canary) v6 stage=None (just trained, not yet evaluated)

Promotion from Staging to Production should be gated on evaluation thresholds, not a manual copy-paste — that gate is where CI/CD for ML picks up.