Softmax
A function that turns a vector of scores into a probability distribution over classes.
Definition
The softmax function exponentiates each element of a vector and divides by the sum of all exponentials, producing values that are positive and sum to one. It converts raw model scores (logits) into a probability distribution over classes.
A temperature parameter divides the logits before exponentiation: high temperature flattens the distribution toward uniform, low temperature sharpens it toward the top choice. This single knob controls the randomness of sampling from a model's output.
Numerically, the exponentials in softmax can overflow, so implementations subtract the maximum logit before exponentiating, a stability trick that changes nothing mathematically but everything in finite precision. Paired with cross-entropy loss, softmax yields clean gradients that make multi-class training well-behaved. The same function appears inside attention, where it converts raw compatibility scores into the weights that combine values.
Softmax(z_i) = exp(z_i) / sum_j exp(z_j). The largest input receives the largest probability, with the sharpness controllable by a temperature.
Uses
- Output layer of multi-class classifiers.
- Attention weights in transformers.
- Sampling temperature control in language generation.
Why it matters
Softmax paired with cross-entropy loss is the standard formulation for multi-class classification, producing differentiable, interpretable probabilities that gradient methods can optimize.
Fusion connection
A softmax head lets a Kronos classifier report calibrated probabilities across candidate plasma regimes rather than a single hard label, conveying its confidence to engineers.