Loss Functions

A loss function scores how wrong a prediction is; minimizing it over the data is what training actually optimizes.

What training minimizes

A loss function maps a prediction and its true target to a single number measuring the error. Averaged over the training set it becomes the objective an optimizer minimizes. The choice of loss encodes what you care about, whether large errors should be punished heavily, whether outliers should dominate, and what the model's outputs mean.

Regression losses

• Mean squared error (MSE): squares residuals; smooth, but very sensitive to outliers.
• Mean absolute error (MAE): absolute residuals; robust to outliers, less smooth.
• Huber loss: squared for small errors, absolute for large ones; a robust compromise.
• Quantile loss: for predicting a chosen percentile rather than the mean.

Classification losses

• Cross-entropy (log loss): penalizes confident wrong predictions harshly; the standard for probabilistic classifiers.
• Hinge loss: the margin-based loss behind support vector machines.
• Focal loss: down-weights easy examples to focus on hard, rare ones in imbalanced data.

import numpy as np
def mse(y, p):  return np.mean((y - p)**2)
def bce(y, p):  # binary cross-entropy
    p = np.clip(p, 1e-9, 1-1e-9)
    return -np.mean(y*np.log(p) + (1-y)*np.log(1-p))

Loss versus metric

The loss is what the model optimizes; it must be differentiable for gradient methods. The evaluation metric is what you actually care about (accuracy, F1), which may be non-differentiable. They often differ: you might train with cross-entropy but judge with F1. Choosing a loss that reflects the real cost of each kind of error, and matching it to the output layer, is a core modeling decision.