Backpropagation and updates

Choose your learning level

BeginnerIntermediateAdvanced

Start with plain-language explanations. Switch levels at any time.

Beginner explanation

Training adjusts weights to reduce a chosen loss. Backpropagation calculates how changes in parameters affect that loss. An optimizer uses those calculations to update parameters; these are related but distinct steps.

Intermediate explanation

Gradients are local rates of change. Gradient descent updates a parameter using θ ← θ − η∇L, where η is the learning rate. Too large a step can overshoot; a very small step can make progress slow.

Advanced explanation

Backpropagation applies the chain rule efficiently through a computational graph. It computes gradients, while optimizers such as SGD or Adam choose updates. Stochastic batches introduce gradient noise, and nonconvex objectives do not promise a global optimum.

PUT IT IN CONTEXT

A worked example

For L(w) = (w − 3)² at w = 1, the gradient is 2(w − 3) = −4. With learning rate 0.1, the next value is 1 − 0.1(−4) = 1.4, moving toward the minimum at 3.

TRY IT YOURSELF

Practice the idea

Using the same loss, start at w = 2 and take one step with learning rate 0.25.

Your working notes (not saved or sent)

Reveal the answer guide

The gradient is −2. The update gives w = 2 − 0.25(−2) = 2.5. The loss falls from 1 to 0.25 in this one-dimensional example.

Go further: advanced challenge

For this quadratic, examine learning rates 0.5, 1 and 1.5. Explain reaching the minimum, oscillation and divergence rather than generalizing those rates to all networks.

An open-ended discussion task. Explain your assumptions and evidence; there may be more than one defensible answer.

Continue the conversation

What part of this idea would you like to test or question? Compare your reasoning with the community.

Find related forum discussions ↗

Opens a topic search in the existing AstraCruz forum. Results depend on available posts; signing in may be required. No discussion or activity is fabricated.

Browse all forums