Hypothesis Testing & p-values

Deciding whether an observed pattern is real or just noise

Why

Every time you deploy a new model and ask "is this better than the old one?", you are running a hypothesis test. A/B testing — the standard way to evaluate model changes in production — is applied hypothesis testing. Data science roles test this heavily because it separates engineers who make decisions from evidence from those who go on intuition.

The classic trap: "our p-value is 0.03, so our new model is significantly better" — which ignores practical significance, multiple comparisons, and power.

Intuition

Hypothesis Testing starts from scepticism. You assume the boring explanation is true — "there's no effect, any difference I see is just random chance" (null hypothesis). Then you measure how surprising your data would be if that boring explanation were true.

The p-value answers: "if the null were actually true, how often would I see a result at least this extreme just by luck?" Small p = this would be very unlikely by luck = evidence against the null. It does NOT tell you the probability that the null is true.

Explanation
The framework
H₀ (null): "no effect" — new model accuracy = old model accuracy H₁ (alternative): "there is an effect" — new model accuracy > old Choose α (usually 0.05) — the false positive rate you'll accept. Compute p-value: p = P(seeing a result this extreme | H₀ is true) Decision: p < α → reject H₀ (statistically significant) p ≥ α → fail to reject H₀ (insufficient evidence)
Type I and Type II errors
Type I (α) — false positive
You say "new model is better" when it isn't. α=0.05 means you accept 5% false positive rate. Lowering α reduces this but makes real effects harder to detect.
Type II (β) — false negative
You say "no difference" when new model actually is better. Power = 1−β = probability of correctly detecting a real effect. Increases with sample size.
Confidence intervals
95% CI for a mean: x̄ ± 1.96 · (σ / √n) Correct interpretation: If we repeated the experiment 100 times, ~95 of the intervals would contain the true parameter value. WRONG interpretation: "There is a 95% probability the true value is in this interval." (The true value is fixed — it either is or isn't in the interval.)

CIs are more informative than p-values alone — they show both statistical significance AND the magnitude and precision of the effect.

Statistical vs practical significance
Example: Old model accuracy = 87.00% New model accuracy = 87.01% n = 10,000,000 → p-value = 0.001 Statistically significant? YES (p < 0.05) Practically significant? NO (0.01% is noise in any real system) Always ask alongside p-value: What is the effect size? (relative improvement %) Does this difference actually matter for the business?
Multiple comparisons problem

If you run 20 tests at α=0.05 and nothing is actually different, you'd expect 1 false positive by chance (0.05 × 20 = 1). Running many tests inflates the effective false positive rate.

  • Bonferroni: divide α by the number of tests. 20 tests → use α = 0.0025 per test. Conservative but simple.
  • FDR (Benjamini-Hochberg): controls the expected fraction of false positives among all rejections. Less conservative than Bonferroni.