An early error carries through to the output
With a single weight we could examine its effect directly. In a network, a whole series of further calculations lies between an early weight and the final output. Even so, the final error depends on that weight.
We deliberately take just two multiplications in a row. On their own they are not yet a powerful deep network, but they show the learning path especially clearly. The input is 2. The first weight is 1, the second weight 2. The desired final value is 6.
First compute the full forward pass
| Step | Calculation | Result |
|---|---|---|
| Intermediate value | Input × first weight | 2 × 1 = 2 |
| Prediction | Intermediate value × second weight | 2 × 2 = 4 |
| Difference | Prediction − target | 4 − 6 = −2 |
| Squared error | Difference × difference | (−2) × (−2) = 4 |
We call the input x, the first weight w, the intermediate value h and the second weight v. The prediction is again called y-hat. So the forward path is h=w×x and y-hat=v×h. The letters are just short names for the numbers in the table.
Now ask about the influences, starting from the back
Start at the prediction 4. For the squared error, its derivative with respect to the prediction is twice the difference: 2×(−2)=−4. This means a small increase in the prediction would locally reduce the error.
Now we ask about the second weight v. The prediction is v×h. The intermediate value h is 2. So a small increase in v increases the prediction by twice that change. Taken together, the derivative of the error with respect to v is −4×2=−8.
The first number, −4, describes "prediction affects error." The second number, 2, describes "second weight affects prediction." We connect the two influences by multiplying.
One more step back
The intermediate value h is multiplied by v=2 in the next calculation. Its local effect on the error is therefore also −4×2=−8.
This intermediate value in turn came from w×x. Because x=2, a small change in w affects the intermediate value by a factor of 2. The derivative of the error with respect to the first weight w is therefore −8×2=−16.
| Derivative we want | Combined influence | Result |
|---|---|---|
| With respect to second weight v | −4 × intermediate value 2 | −8 |
| With respect to intermediate value h | −4 × second weight 2 | −8 |
| With respect to first weight w | −8 × input 2 | −16 |
Now both weights know in which direction they should be changed locally. The method didn't guess any meaning and didn't invent new targets for inner neurons. It examined the known calculation steps backwards for their effect on change.
That is the idea of the chain rule
Put simply, the chain rule says: when a change passes through several consecutive calculation steps, we multiply their local factors of change. For our first weight the short form is:
Read from left to right: We want to know how w affects the error L. To get it, we combine the effect of the prediction on the error, the effect of the intermediate value on the prediction and the effect of the weight on the intermediate value.
The concrete calculation is −4×2×2=−16. The formula is the shorthand for exactly this path. Here there is a single continuous path of influence; in larger networks several paths can come together.
What happens at branches?
If an intermediate value affects several later calculations, it can change the error along several paths. The contributions of these paths are added. Within a path we multiply local factors of change; across different paths we add their contributions.
The method becomes efficient because each intermediate derivative computed on the way back, such as −8 for h, is computed only once and then reused for all the weights before it. The paths are not listed one by one. This backward calculation in a network is called backpropagation. A computational graph is the representation of the individual calculation steps and their dependencies. "Graph" here means connections between operations, not a statistical plot of a curve.
Collect derivatives first, then change weights
We computed the derivatives −16 for w and −8 for v at the old values. With learning rate 0.01 we get w=1−0.01×(−16)=1.16 and v=2−0.01×(−8)=2.08.
Only then do we run the next forward pass: intermediate value 1.16×2=2.32; prediction 2.08×2.32=4.8256. The error is about 1.3792, so smaller than the previous 4.
If you changed one weight in the middle of the backward pass and then computed other derivatives with the new values, that would be a different, inconsistent procedure. A normal step computes all gradients at the same current state and changes the weights afterwards.
Activation functions fit into this path too
For ReLU, the local factor of change on the positive side is 1: a small change in the input is passed on unchanged. On the negative side it is 0: small changes stay within the region that is set to zero. Exactly at zero there is a kink; programs use a fixed convention there.
That's why an activation function affects not only the forward value but also the backward pass. You don't need to memorize the derivative rule for every function. But you should recognize that both directions of calculation belong together.
What the library takes over later
PyTorch can combine these derivative rules automatically. Autograd is the name for this automatic differentiation. You define the forward path; the library records the dependencies it needs and computes the gradients when you call backward.
In the next chapter we set up this library. You already know what work it takes off your hands, and which decisions — architecture, data and learning objective — remain yours.