ModelRefs / Perceptrons & Feedforward Networks — Tutorial
Perceptrons & Feedforward Networks — Tutorial
The original artificial neuron — how a single unit learns, and why stacking them changes everything. Covers The perceptron: one artificial neuron.
Overview
The original artificial neuron — how a single unit learns, and why stacking them changes everything
Level: Intermediate. Estimated reading time: 25 minutes.
The perceptron: one artificial neuron
The perceptron (Rosenblatt, 1958) is the simplest neural unit. It takes a vector of inputs, multiplies each by a weight, sums the results, adds a bias, and passes through a step function:
output = 1 if (w·x + b) > 0 else 0
Training: compare the output to the true label y. If correct, do nothing. If wrong: - If output=0 but y=1: increase weights (w ← w + α·x) - If output=1 but y=0: decrease weights (w ← w − α·x)
The perceptron convergence theorem guarantees it finds a perfect separator if one exists. Minsky and Papert's 1969 book showed it cannot solve XOR — a linearly non-separable problem. This triggered the first AI winter for neural networks.
The fix: use a differentiable activation function (sigmoid, ReLU) instead of the step function, and add hidden layers. A network with one hidden layer can approximate any continuous function (Universal Approximation Theorem).
Feedforward networks: layers of perceptrons
A feedforward (fully-connected) network stacks layers: input → hidden₁ → hidden₂ → output. Each layer is: h = activation(W·h_prev + b).
The activation function is critical. Step function: not differentiable (cannot train with backprop). Sigmoid: smooth, outputs 0–1, but saturates at extremes (vanishing gradient). ReLU (max(0, x)): the modern default — fast, doesn't saturate for positive values.
Information flows forward (forward pass) to produce a prediction, then backward (backward pass) to compute gradients via the chain rule. This is backpropagation.
Sizes: input layer = number of features. Output layer = number of classes (softmax for multi-class) or 1 (sigmoid for binary). Hidden layers: wider = more capacity per layer, deeper = more compositionality. Start with 1–2 hidden layers and 64–256 units each.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Perceptrons & Feedforward Networks — Tutorial.