Bias and Variance of Estimators

An estimator's error splits into bias, variance, and irreducible noise, and reducing one often raises another.

What an estimator is

An estimator is a rule that maps data to a guess of an unknown quantity. Because data are random, the estimate is random too. Two properties describe its quality: how far off it is on average (bias) and how much it wobbles from sample to sample (variance).

Bias and variance defined

The bias is E[θ̂] − θ, the systematic offset of the average estimate from the truth. The variance is Var(θ̂), the spread of the estimate. An unbiased estimator is centered on the truth but may still be noisy; a low-variance estimator is stable but may be systematically off.

The decomposition

Mean squared error decomposes cleanly: MSE = bias² + variance. This identity is the basis of the bias-variance tradeoff in modeling. A flexible model has low bias but high variance; a rigid model has high bias but low variance. The best model minimizes their sum, not either alone.

Example: sample variance

Dividing the sum of squared deviations by n gives a biased variance estimate; dividing by n − 1 (Bessel's correction) makes it unbiased, because one degree of freedom is spent estimating the mean. The corrected version is the standard default.

Consistency

An estimator is consistent if it converges to the truth as data grows — its bias and variance both vanish in the limit. Consistency is a minimal requirement; among consistent estimators, one prefers the one with the smallest variance, the efficient estimator.