Backpropagation Through a Tiny MLP

Compute the forward pass and every gradient of a two-layer network on one example, entirely by hand.

Problem

Backpropagation computes the gradient of a loss with respect to every weight by applying the chain rule backward through the network. We take a minimal network, one input, one hidden unit with a sigmoid, one output, and a squared-error loss, and compute all gradients explicitly.

Forward pass

With input x, hidden pre-activation z1 = w1 x + b1, hidden activation h = sigmoid(z1), output y = w2 h + b2, and loss L = (y - t)^2 / 2 for target t. Each intermediate value is stored for the backward pass.

import numpy as np
x,t=1.0,0.5
w1,b1,w2,b2=0.8,0.0,0.5,0.1
z1=w1*x+b1; h=1/(1+np.exp(-z1)); y=w2*h+b2; L=0.5*(y-t)**2
dL=y-t                       # dL/dy
dw2=dL*h; db2=dL              # output layer grads
dh=dL*w2; dz1=dh*h*(1-h)      # sigmoid derivative
dw1=dz1*x; db1=dz1
print('L',round(L,4))
print('grads',[round(g,4) for g in (dw1,db1,dw2,db2)])

Result

The backward pass reuses the stored forward values: the output gradient dL/dy flows into the output weights directly, then multiplies through w2 and the sigmoid derivative h(1-h) to reach the hidden weights. Each gradient tells how the loss would change if that weight moved. A gradient-descent step subtracts a small multiple of each gradient, and repeating the whole cycle trains the network.

• Backprop is just the chain rule applied efficiently, caching forward values to avoid recomputation.
• The sigmoid derivative h(1-h) is near zero when h saturates, the source of vanishing gradients in deep nets.
• Every deep-learning framework Kronos uses automates exactly this computation over millions of parameters.