The Perceptron

The perceptron is the simplest trainable neuron: a weighted sum passed through a threshold, and the historical seed of neural networks.

Definition

A perceptron computes a linear combination of its inputs and applies a step threshold. For inputs x1..xn with weights w1..wn and bias b, it outputs 1 if w·x + b > 0 and 0 otherwise. Geometrically the weights define a hyperplane, and the perceptron reports which side of that hyperplane a point falls on. This makes it a linear binary classifier.

The learning rule

Rosenblatt's 1958 rule updates weights only on misclassified examples. For a training pair (x, y) with y in {0,1} and prediction yhat, the update is w <- w + eta·(y - yhat)·x, where eta is the learning rate. If the prediction is correct the term (y - yhat) is zero and nothing changes. If wrong, the weight vector shifts toward or away from x to reduce the error.

import numpy as np
def perceptron(X, y, eta=0.1, epochs=20):
    w = np.zeros(X.shape[1]); b = 0.0
    for _ in range(epochs):
        for xi, yi in zip(X, y):
            pred = 1 if (w @ xi + b) > 0 else 0
            w += eta * (yi - pred) * xi
            b += eta * (yi - pred)
    return w, b

The convergence theorem

If the training data is linearly separable, the perceptron rule is guaranteed to find a separating hyperplane in a finite number of updates. The bound depends on the margin between classes and the radius of the data, not on the dimension. This was one of the first provable guarantees in machine learning.

The XOR limitation

Minsky and Papert showed in 1969 that a single perceptron cannot represent XOR, because XOR is not linearly separable: no straight line separates the classes. This result cooled interest in neural networks for years. The resolution was to stack perceptrons into layers, giving the multilayer perceptron, which composes many linear boundaries into nonlinear regions.

• Inputs and weights define a separating hyperplane.
• Learning nudges weights only on mistakes.
• Guaranteed convergence when data is separable.
• Single layer cannot solve XOR or other nonlinear problems.