cs.CLNov 1, 2026

Detailed Derivation of The Backpropagation Algorithm

Authors: Faris Alasmary

Abstract

Backpropagation is the standard method for computing the gradients that train neural networks, but it is usually presented in a compact vector form that skips the intermediate steps. This tutorial derives backpropagation step by step for a fully connected feedforward network with sigmoid activations and a squared-error loss. Using a running example with three layers, we first differentiate the error with respect to a single weight by following its path through the network with the chain rule. We then show how summing over all paths leads to the error terms δδ and to the four general backpropagation equations in matrix form. Every term is annotated with its dimensions, so each equation can be checked for consistency. The derivation is then extended to mm training examples, where the weight gradients become a single matrix product and the bias term corresponds to broadcasting. It is also extended to mini-batch gradient descent, including why the gradient of a mini-batch estimates the full gradient. Batch and mini-batch training are both given as algorithms. Finally, each equation is linked to the corresponding line of a NumPy implementation trained on the AHDBase dataset of handwritten Arabic digits, and the complete code is included.

Figures & tables

Explore similar work

CardsList
  1. Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle

    Apr 20, 2026Abdeladhim TahimiBackpropagationOrdinary Differential Equations

  2. Backpropagation as a Nilpotent Linear System

    Jul 13, 2026Ahmed BoughammouraBackpropagationNeural Network Training Dynamics

  3. Approaching Backpropagation Accuracy with Weight Perturbation on MNIST

    Oct 15, 2026Patrick InoueGradient-Free OptimizationNeural Network Optimization