Deciding whether an observed pattern is real or just noise
Every time you deploy a new model and ask "is this better than the old one?", you are running a hypothesis test. A/B testing — the standard way to evaluate model changes in production — is applied hypothesis testing. Data science roles test this heavily because it separates engineers who make decisions from evidence from those who go on intuition.
The classic trap: "our p-value is 0.03, so our new model is significantly better" — which ignores practical significance, multiple comparisons, and power.
Hypothesis Testing starts from scepticism. You assume the boring explanation is true — "there's no effect, any difference I see is just random chance" (null hypothesis). Then you measure how surprising your data would be if that boring explanation were true.
The p-value answers: "if the null were actually true, how often would I see a result at least this extreme just by luck?" Small p = this would be very unlikely by luck = evidence against the null. It does NOT tell you the probability that the null is true.
CIs are more informative than p-values alone — they show both statistical significance AND the magnitude and precision of the effect.
If you run 20 tests at α=0.05 and nothing is actually different, you'd expect 1 false positive by chance (0.05 × 20 = 1). Running many tests inflates the effective false positive rate.