Descriptive statistics

The first thing you reach for when you see a new dataset

Why

Before building any model, you need to understand your data. Given any new dataset, the first step is always exploratory data analysis, and descriptive statistics are the tools of EDA — get these wrong and every downstream decision rests on a shaky foundation.

Beyond EDA, these concepts appear in technical depth too: PCA requires the covariance matrix, normalisation decisions require understanding mean and variance, and model evaluation often involves comparing distributions of predictions vs actuals.

Intuition

Descriptive statistics are summaries. Instead of looking at 1 million data points, you reduce them to a handful of numbers that capture the most important properties: where is the data centred (mean/median), how spread out is it (variance/std), how is it shaped (skewness/kurtosis), and how do features relate to each other (covariance)?

Think of them as a health check for your data — you run these before any modelling to catch problems: skewed distributions, outliers, correlated features, missing patterns.

Explanation

Mean vs median

Mean: μ = (1/n) · Σ xᵢ — sum divided by count Median: middle value when sorted — 50th percentile

The mean minimises the sum of squared deviations. The median minimises the sum of absolute deviations. Key difference: outlier sensitivity.

Mean — use when

Data is roughly symmetric, no extreme outliers. One billionaire raises the "average" income of a room dramatically — mean becomes misleading.

Median — use when

Data is skewed or has outliers. House prices, income distributions. Far more representative than mean when there's extreme inequality.

Variance & standard deviation

Variance: σ² = (1/n) · Σ (xᵢ - μ)² = E[(X - μ)²] = E[X²] - (E[X])² — useful alternative form Std dev: σ = √σ² — same units as the data

Variance measures how spread out values are around the mean. Standard deviation brings it back to the original units. In ML: feature normalisation (subtract mean, divide by std) puts all features on the same scale — essential for gradient-based and distance-based models (KNN, SVM).

Adding a constant to all values doesn't change variance
Multiplying by c scales variance by c²
Variance of a constant = 0

Covariance & the covariance matrix

Cov(X, Y) = E[(X - μₓ)(Y - μᵧ)] = E[XY] - E[X]·E[Y] Cov > 0 → X and Y increase together Cov < 0 → X increases when Y decreases Cov = 0 → no linear relationship (may still be non-linear) Covariance matrix Σ (d×d, for d features): Σ = (1/n) · XᵀX (X is mean-centred) Diagonal: Var(Xᵢ) — variance of each feature Off-diagonal: Cov(Xᵢ, Xⱼ) — how features co-vary

The covariance matrix is central to PCA — eigenvectors of Σ are the principal components, eigenvalues are the variance explained along each. Large off-diagonal values mean features are correlated and potentially redundant.

Skewness & kurtosis

Skewness — asymmetry

0 = symmetric. >0 = right-skewed (long tail right, mean > median — income, house prices). <0 = left-skewed. Highly skewed features often benefit from log-transform before linear models.

Kurtosis — tail heaviness

Gaussian kurtosis = 3. High kurtosis = heavy tails, more extreme outliers (common in financial data). Low kurtosis = fewer extremes. Heavy-tailed data means outliers are more frequent than Gaussian predicts.

← Previous Next →