Maximum Likelihood Estimation

Maximum likelihood picks the parameter values that make the observed data most probable under the model.

The likelihood function

Given data and a model with parameters θ, the likelihood L(θ) is the probability (or density) of the observed data as a function of θ. Maximum likelihood estimation (MLE) chooses θ̂ to maximize L(θ). Intuitively, it selects the parameters under which the data you actually saw were most expected.

Log-likelihood

Kronos motion — parameter scan

Because data points are usually treated as independent, the likelihood is a product, so it is easier to maximize its logarithm, which turns the product into a sum: ℓ(θ) = Σ log f(xi; θ). The maximizer is the same, and the sum is numerically stable and differentiable.

Worked example: Bernoulli

For n coin flips with k heads, the log-likelihood is k log p + (n − k) log(1 − p). Setting its derivative to zero gives p̂ = k/n — the observed frequency. Many MLEs recover the natural sample estimate, which is one reason the method is trusted.

python

import math
def neg_loglik_bernoulli(p, k, n):
    return -(k*math.log(p) + (n-k)*math.log(1-p))
# minimized at p = k/n; check numerically
print(round(30/100, 3))  # 0.3

Properties

Consistent: converges to the true parameter as data grows.
Asymptotically normal, enabling standard errors from the curvature of ℓ.
Asymptotically efficient: attains the smallest possible variance in the limit.

Cautions

MLE can overfit with few data or many parameters, and it can be biased in small samples (the MLE of a variance divides by n, not n − 1). Regularization or a Bayesian prior addresses these, effectively pulling the estimate toward a sensible default.