Workspace/Mini projects
Loading progress
All mini projects
MODULE 03 · 6 HOUR BUILD

A tiny network you can audit

Build a small neural-network trainer whose forward pass, derivatives, and loss trajectory can all be inspected without a machine-learning framework.

Build evidence Record your actual checks, results, and limitations.

Build it in stages

  1. Run the seed and inspect its four-parameter tanh network.
  2. Check every parameter gradient with central differences at several non-saturated points.
  3. Add held-out examples and report training versus evaluation loss.
  4. Compare two learning rates, preserving identical initialization and data.
  5. Record per-step loss and parameter values in a portable report.
  6. Port the same calculation to a tensor framework as an optional second implementation and compare outputs.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

  • All four analytic gradients match central differences within a documented tolerance.
  • The seed reduces mean training loss on its deterministic fixture.
  • A too-large learning rate is demonstrated through measured loss, without claiming universal divergence.
  • Changing the loss normalization produces the expected gradient scale change.
  • The report clearly distinguishes a teaching network from a competitive predictive model.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py
python
import math

DATA = [(-1.0, -0.8), (0.0, 0.0), (1.0, 0.8)]

def objective(parameters):
    w, b, v, c = parameters
    total = 0.0
    grads = [0.0] * 4
    for x, target in DATA:
        h = math.tanh(w * x + b)
        error = v * h + c - target
        total += 0.5 * error * error
        dz = error * v * (1 - h * h)
        for i, value in enumerate((dz * x, dz, error * h, error)):
            grads[i] += value / len(DATA)
    return total / len(DATA), grads

def check(parameters):
    _, analytic = objective(parameters)
    errors = []
    for i in range(len(parameters)):
        plus, minus = parameters.copy(), parameters.copy()
        plus[i] += 1e-6
        minus[i] -= 1e-6
        numeric = (objective(plus)[0] - objective(minus)[0]) / 2e-6
        errors.append(abs(numeric - analytic[i]))
    return max(errors)

def train(parameters, steps=100, rate=0.2):
    parameters = parameters.copy()
    for _ in range(steps):
        _, gradient = objective(parameters)
        parameters = [p - rate * g for p, g in zip(parameters, gradient)]
    return parameters

if __name__ == "__main__":
    initial = [0.4, 0.0, 0.7, 0.0]
    final = train(initial)
    print("gradient check:", check(initial) < 1e-8)
    print(f"initial loss: {objective(initial)[0]:.6f}")
    print(f"final loss: {objective(final)[0]:.6f}")
    print("loss decreased:", objective(final)[0] < objective(initial)[0])

Push it further

Implement a small scalar automatic differentiation engine with addition, multiplication, tanh, reverse topological traversal, and accumulated gradients; compare it against the manual derivatives.