Multilayer Perceptron

Stacking layers of neurons with nonlinear activations turns a linear classifier into a universal function approximator.

Structure

A multilayer perceptron (MLP) is a feedforward network with an input layer, one or more hidden layers, and an output layer. Each layer applies an affine map followed by a nonlinear activation: h = phi(W·x + b). Because the activation is nonlinear, composing layers produces boundaries that no single linear unit could express. An MLP with no activation collapses to one linear map, however many layers it has.

Why nonlinearity matters

Kronos motion — market layers

The composition of linear functions is linear. Only by inserting a nonlinear function between layers does depth buy expressive power. This is why the choice of activation is not cosmetic: it is the source of the network's ability to bend decision surfaces and fit complex data.

Universal approximation

The universal approximation theorem states that an MLP with a single hidden layer and enough units can approximate any continuous function on a bounded domain to arbitrary accuracy. The theorem is an existence result: it promises such a network exists but says nothing about how many units are needed or whether training will find the weights. In practice, depth often achieves with far fewer parameters what a single wide layer would need exponentially many units to match.

Forward and backward passes

Prediction runs a forward pass, layer by layer. Training runs backpropagation, which applies the chain rule to compute the gradient of a loss with respect to every weight, then adjusts weights by gradient descent. The same matrix multiplications used forward are transposed and reused backward, which is what makes training efficient on modern hardware.

python

import numpy as np
def mlp_forward(x, W1, b1, W2, b2):
    h = np.maximum(0, W1 @ x + b1)   # ReLU hidden layer
    return W2 @ h + b2                # linear output

Where MLPs are used

MLPs handle tabular data, act as the final classification head on top of convolutional or transformer features, and form the position-wise feedforward blocks inside transformers. In scientific modeling, small MLPs serve as fast surrogate models: a network trained on expensive plasma or materials simulations can approximate an output in microseconds, letting engineers scan large parameter spaces before committing to a full calculation.

Layers of affine maps plus nonlinear activations.
Depth adds expressive power that width alone cannot cheaply match.
Universal approximation guarantees existence, not trainability.
Backbone of classification heads and transformer feedforward blocks.