Law of Large Numbers

The law of large numbers says the sample average converges to the true mean as the number of observations grows.

The statement

For independent draws X1, X2, … with common mean μ, the sample average X̄_n = (1/n) Σ Xi converges to μ as n grows. The weak law says the probability that X̄_n departs from μ by more than any fixed amount tends to zero; the strong law says X̄_n converges to μ with probability one.

Why it holds

The variance of the sample mean is σ²/n, which shrinks to zero. As the average becomes ever more concentrated around μ, large deviations become vanishingly unlikely. The finite-variance assumption is what makes this argument work.

What it does and does not promise

It guarantees convergence of the average, not of individual outcomes. After a run of heads, a fair coin is not 'due' for tails — that is the gambler's fallacy. The average is pulled toward 0.5 by accumulating new trials that dilute the past, not by any correction.

Foundation of Monte Carlo

Every Monte Carlo estimate relies on this law: run a stochastic model n times, average the quantity of interest, and trust that the average approaches the true expectation. Without the law of large numbers, simulation would have no justification.

A limit case

Distributions without a finite mean, such as the Cauchy, violate the law: their sample average does not settle down no matter how many samples you take. This is a reminder that convergence is a property of the distribution, not a universal guarantee.