Backpropagation is the chain rule with bookkeeping

Backpropagation computes how a scalar loss changes when each intermediate value or parameter changes. It applies the chain rule in reverse order through the operations used in the forward pass. The algorithm needs the graph and enough forward values to evaluate local derivatives. Automatic differentiation libraries build and traverse this structure, but the mathematics is the same as a carefully organized hand calculation.

Consider z = wx + b, h = tanh(z), y_hat = vh, and L = 0.5 times (y_hat-y) squared. This graph has four parameters or inputs of interest: w, b, v, and x. The forward pass stores z, h, y_hat, and the error. The backward pass starts with the derivative of the scalar loss and moves toward the earlier quantities.

The derivative of L with respect to y_hat is error = y_hat-y. The derivative with respect to v is error times h. The derivative with respect to h is error times v. Since the derivative of tanh(z) is 1-h squared, the derivative with respect to z is error times v times (1-h squared). Finally, dL/dw multiplies that value by x, and dL/db leaves it unchanged.

If one parameter influences the loss through two paths, add both contributions. For example, if u = w*x and the loss uses u+u, the derivative through u is doubled. Overwriting a gradient when visiting the second path loses the first contribution. This is why many differentiation systems accumulate gradients, and why training loops must clear old accumulated gradients when accumulation is not intended.

The backward pass calculates sensitivities at the parameter values used for the forward pass. An optimizer then uses those sensitivities to update the parameters. If you update v before using it to calculate dL/dh, you mix two different parameter states inside one derivative calculation. The resulting values are generally not the gradient of the original loss.

For a batch mean loss, each example's contribution is divided by the batch size. Summing gradients without adjusting the learning rate changes the effective update scale when batch size changes. Gradient accumulation over smaller batches can reproduce a larger batch under appropriate conditions, but normalization and stochastic layers can complicate exact equivalence. Always state whether a loss is summed or averaged before comparing gradients.

For a scalar parameter w, central differences estimate its derivative as [L(w+epsilon)-L(w-epsilon)] divided by 2 epsilon. This requires two forward evaluations for each parameter and becomes expensive for a large model. For a tiny hand-built network, it is an excellent debugging tool because it checks the loss numerically without reusing the same chain-rule implementation.

Choose epsilon large enough to avoid catastrophic cancellation and small enough to approximate local behavior. Around 1e-6 often works for small double-precision examples, but it is not universal. Avoid points where the function is nondifferentiable, such as a ReLU exactly at zero, when comparing to a single analytic derivative. Inspect relative and absolute error; a tiny denominator can make relative error look alarming even when absolute error is negligible.

A matching gradient check supports the derivative implementation at the checked point. It does not prove the objective is appropriate, the data is labeled correctly, or training will generalize. You can perfectly differentiate a loss that accidentally compares each prediction with the wrong target. Shape checks, example-level traces, and held-out evaluation remain necessary.

The code calculates three parameter derivatives for a one-hidden-unit network and compares them to central differences. It never calls a machine-learning library. The mini project expands this into a small trainer with a gradient check before optimization. This order is deliberate: first establish that the update uses the derivative of the function you intended, then investigate whether optimization improves the desired behavior.

The input is one scalar and the parameters are an input weight, hidden bias, and output weight. The analytic chain-rule derivatives are independently checked by perturbing each parameter. Change the target or initial parameters, keeping the finite-difference step fixed initially.