Pre-Notes: Learning Objectives & Prerequisites

Learning Objectives

By the end of this note you will be able to:

• Define backpropagation and explain why we need it (efficient gradient computation for deep networks).
• Describe the chain rule intuition and how gradients flow backward from the loss to every parameter.
• Explain forward pass vs backward pass: what is computed when, and why order matters.
• Use computational graphs to reason about dependencies and gradient flow.
• Compare manual backprop vs automatic differentiation (autograd) and when each is used.
• Answer common interview questions and avoid wrong vs correct traps (e.g., “backprop is the learning algorithm,” “gradients are computed in the forward pass”).

Prerequisites

Before starting, you should know:

• Forward pass: A neural network takes input x and produces output ŷ by applying layers (linear + activation) in sequence.
• Loss function: L(ŷ, y) measures prediction error; we want to minimize it.
• Basic calculus: Partial derivatives, chain rule (e.g., ∂L/∂a = (∂L/∂b)(∂b/∂a)), and that the gradient is a vector of partials.

If you don’t, review Calculus for Deep Learning, Artificial Neuron & Perceptron, and Loss Functions first.

Where This Fits

• Builds on: Forward pass (activations at each layer), Loss (L and ∂L/∂ŷ), Calculus (chain rule, gradients).
• Needed for: Optimizers (they use ∂L/∂θ to update weights), training loops, and understanding why some architectures (e.g., very deep nets) have gradient issues.

This topic is part of Phase 2: Core Neural Network Internals and sits between Loss Functions and Optimizers.

1. What is Backpropagation?

Backpropagation (or backprop) is the algorithm used to compute the gradient of the loss with respect to every parameter in a neural network. It does this by applying the chain rule layer by layer, starting from the loss at the output and moving backward through the network.

In simple words: after we compute the loss L on a batch, we need to know “how much did each weight contribute to this error?” Backpropagation is the efficient way to compute those contributions (the gradients ∂L/∂θ) so the optimizer can update the weights.

Simple Intuition

Imagine a factory assembly line: the forward pass is the product moving forward through stations (layers), and each station adds something (weights, activations). When the final product (prediction) is wrong, we need to assign blame to each station: “How much did this station’s settings cause the error?” Backpropagation is like walking backward along the line, station by station, passing back a “blame signal” (gradient) so each station knows how to adjust. The chain rule tells us how to split that blame between connected stations.

Formal Definition (Interview-Ready)

Backpropagation is an algorithm that computes the gradient of a scalar loss L with respect to all trainable parameters of a neural network by recursively applying the chain rule from the output layer to the input layer. It requires a forward pass (to compute activations and the loss) and then a backward pass (to compute ∂L/∂θ for every parameter θ), enabling gradient-based optimization (e.g., SGD, Adam).

In a Nutshell

Backpropagation = the backward pass that computes ∂L/∂θ for every weight and bias using the chain rule. It is the “engine” that makes training deep networks possible by giving the optimizer the gradients it needs.

2. Why Do We Need Backpropagation?

Without backpropagation, we would have to compute gradients for millions (or billions) of parameters separately—e.g., by finite differences—which would be prohibitively slow. Backpropagation reuses the structure of the network and the chain rule so that one backward pass gives us all the gradients we need in time proportional to the forward pass.

Old vs New Paradigm

Key Reasons

1. Efficiency: One backward pass computes all ∂L/∂θ in time comparable to the forward pass (roughly 2–3× cost), not “number of parameters” times the forward cost.
2. Scalability: Deep networks have too many parameters for naive gradient computation; backprop is the only practical way to train them with gradient descent.
3. Exact gradients: Unlike finite differences, backprop gives exact (analytical) gradients up to floating-point precision, so optimization is stable and correct.
4. Composability: Any differentiable building block (layer, activation) can be plugged in; we only need its local derivative (∂output/∂input) for the chain rule.

Real-World Relevance

3. Core Building Block: The Chain Rule and Gradient Flow

The Chain Rule (Intuition)

If L depends on z, and z depends on y, and y depends on x, then:

∂L/∂x = (∂L/∂z)(∂z/∂y)(∂y/∂x)

Each layer only needs to know: (1) the upstream gradient (∂L/∂output of this layer) and (2) its local derivative (∂output/∂input). Multiply them to get the gradient to pass further backward (∂L/∂input of this layer).

One Neuron Example

For a single neuron: z = w·x + b, then a = σ(z) (activation), and the loss L depends on a.

• ∂L/∂a comes from the layer above (or from the loss).
• ∂L/∂z = (∂L/∂a)(∂a/∂z) = (∂L/∂a) σ′(z) — chain rule.
• ∂L/∂w = (∂L/∂z) x, ∂L/∂b = ∂L/∂z, ∂L/∂x = (∂L/∂z) w — pass ∂L/∂x to the previous layer.

So each layer: receives ∂L/∂(its output), computes ∂L/∂(its parameters) and ∂L/∂(its input), sends ∂L/∂(its input) backward.

Computational Graph View

We can think of the network as a directed graph: nodes are operations (matmul, add, ReLU, loss); edges are tensors (activations, weights). The forward pass computes and stores each node’s output. The backward pass visits nodes in reverse order, and at each node applies the chain rule using stored forward values and the incoming gradient from downstream.

Interview-ready: The core building block is the chain rule: ∂L/∂θ = (∂L/∂y)(∂y/∂θ) where y is this layer’s output. Backprop is layer-wise application of this: start from ∂L/∂ŷ (from the loss) and propagate backward, multiplying by local Jacobians. Each layer must implement forward (compute output) and backward (given ∂L/∂output, compute ∂L/∂params and ∂L/∂input).

Diagram: Forward and Backward Pass

Caption: Forward: input x → layers → ŷ → L. Backward: ∂L/∂ŷ → layer gradients → ∂L/∂θ and ∂L/∂x. Gradients flow in reverse order of the forward pass.

In a Nutshell

Backprop = chain rule from loss backward. Each layer gets ∂L/∂(output), multiplies by ∂(output)/∂(input) and ∂(output)/∂(params) to get ∂L/∂(params) (for the optimizer) and ∂L/∂(input) (for the previous layer).

Think about it: Why must we do the forward pass before the backward pass? What would we be missing if we tried to compute gradients without it?

4. Process: Forward Pass, Loss, Backward Pass, Update

Step-by-Step (One Training Iteration)

1. Forward pass: Given input x, compute activations at each layer and finally ŷ = model(x). Store any values needed for the backward pass (e.g., pre-activation z, for ReLU: which inputs were > 0).
2. Loss: Compute L = L(ŷ, y) (e.g., cross-entropy, MSE). This is a scalar.
3. Backward pass (backprop): Start with ∂L/∂ŷ (gradient of loss w.r.t. model output). Then layer by layer backward, compute:
   • ∂L/∂θ for this layer’s parameters (weights, biases),
   • ∂L/∂input for this layer’s input (which becomes the “upstream gradient” for the previous layer).
4. Optimizer step: Use ∂L/∂θ for all parameters to update θ (e.g., θ ← θ − η ∇L). This is not part of backprop—backprop only computes gradients; the optimizer uses them.

What Gets Stored and When

Interview tip: The forward pass must store whatever the backward pass needs to compute local derivatives (e.g., z for σ′(z), or the mask for ReLU). That’s why training uses more memory than inference—we keep activations for the backward pass.

In a Nutshell

One iteration: Forward (x → ŷ, store activations) → Loss (L) → Backward (∂L/∂ŷ → … → ∂L/∂θ) → Update (optimizer uses ∂L/∂θ). Backprop is only the backward part; it does not update parameters.

5. Key Sub-Topics: Computational Graphs, Gradient Accumulation, Automatic Differentiation

Computational Graphs

• Nodes: Operations (e.g., matmul, add, ReLU, loss).
• Edges: Data (tensors) flowing between nodes.
• Forward: Execute nodes in topological order; store outputs on edges (and any values needed for backward).
• Backward: Start from the loss node; for each node in reverse topological order, compute gradient of loss w.r.t. its inputs using the gradient w.r.t. its output(s) and the stored forward values.

This is exactly what frameworks like PyTorch and TensorFlow do: they build a dynamic (PyTorch) or static (older TF) graph and run backprop on it.

Gradient Accumulation

When the batch is too large to fit in memory, we can split it into micro-batches, run forward + backward on each micro-batch without updating parameters, accumulate the gradients (add ∂L/∂θ from each micro-batch), then do one optimizer step with the accumulated gradient. So: backprop runs multiple times; the optimizer step runs once per “logical” batch.

Interview tip: Gradient accumulation = sum gradients over several forward/backward passes, then update once. Effective batch size = micro-batch size × number of micro-batches. Used when you want a large batch size but have limited GPU memory.

Automatic Differentiation (Autograd)

Automatic differentiation is the technique that implements backpropagation automatically: given a function defined as a composition of differentiable operations, the system (PyTorch, JAX, TensorFlow) records the operations during the forward pass and then replays them in reverse to compute gradients. You don’t write ∂L/∂θ by hand—you just write the forward code; autograd gives you ∂L/∂θ.

• Forward-mode AD: Computes derivatives along with the forward pass (one input direction at a time); less common in DL.
• Backward-mode AD: Computes gradients of a scalar output w.r.t. all inputs in one backward pass—this is backpropagation. This is what PyTorch and TensorFlow use.

In a Nutshell

Computational graph = representation of the computation (nodes = ops, edges = tensors); backprop = traverse it in reverse with the chain rule. Gradient accumulation = add gradients over several small batches, then update. Autograd = backprop done automatically by the framework from your forward code.

6. Comparison: Forward Pass vs Backward Pass, Backprop vs Optimizer

Forward vs Backward Pass

Backpropagation vs Optimizer

Interview tip: Backprop does not change parameters; it only computes gradients. The optimizer changes parameters. Saying “backprop updates the weights” is wrong—the optimizer updates the weights using the gradients that backprop computed.

7. Common Types and Variants

1. Standard Backpropagation (Batch)

• What: One forward pass on a batch, then one backward pass; gradients are averaged (or summed) over the batch. Optimizer then updates θ.
• Use case: Default in most training loops (e.g., loss.backward() then optimizer.step()).
• Example: Training a CNN on batches of 32 images; one backprop per batch.

2. Backpropagation Through Time (BPTT)

• What: Backprop in recurrent networks where the same weights are used at each time step. The gradient is backpropagated through time (unrolling the RNN).
• Use case: Training RNNs, LSTMs, GRUs on sequences.
• Example: Language model: loss at time T; gradients flow back through t = T−1, T−2, … to t = 1.

3. Gradient Accumulation (Multiple Backward, One Update)

• What: Run forward + backward on micro-batches; add gradients (don’t zero them between micro-batches); then one optimizer step. Effective batch size = micro-batch size × number of steps.
• Use case: Large effective batch when GPU memory is limited.
• Example: Effective batch 64 with 4 micro-batches of 16; 4× forward/backward, 1× optimizer step.

4. Automatic Differentiation (Autograd)

• What: Framework automatically computes ∂L/∂θ by recording the forward ops and applying the chain rule in reverse. User writes only forward code.
• Use case: All modern DL (PyTorch, TensorFlow, JAX). Hand-written backprop is rare except in teaching or custom ops.
• Example: loss.backward() in PyTorch triggers autograd to fill .grad on all parameters.

5. Truncated BPTT

• What: In RNNs, only backprop through a limited number of time steps (e.g., last K steps) to save memory and compute; older steps get no gradient.
• Use case: Very long sequences where full BPTT is too expensive.
• Example: Training on long text with a 100-step truncation window.

8. FAQs & Common Student Struggles

Q1. Is backpropagation the same as the learning algorithm?

No. Backpropagation only computes the gradients ∂L/∂θ. The learning (updating parameters) is done by the optimizer (e.g., SGD, Adam) using those gradients. So: backprop = gradient computation; optimizer = parameter update.

Q2. Why do we need the forward pass before the backward pass?

The backward pass needs intermediate values from the forward pass to compute local derivatives (e.g., σ′(z), or which inputs were positive for ReLU). So we must run the forward pass first and store those values; then the backward pass uses them with the chain rule.

Q3. Why is it called “back” propagation?

Because we propagate the gradient (sensitivity of the loss to each quantity) backward through the network—from the output/loss toward the input—opposite to the direction of the forward computation.

Q4. What is the chain rule in one sentence?

The chain rule says: if L depends on y and y depends on x, then ∂L/∂x = (∂L/∂y)(∂y/∂x). Backprop applies this repeatedly from the loss back through each layer.

Q5. Why does training use more memory than inference?

During training, we store activations (and sometimes intermediate tensors) from the forward pass so the backward pass can use them to compute gradients. During inference, we don’t run backprop, so we don’t need to keep those activations and can free memory after each layer. So training memory ≈ forward activations + gradients + optimizer state.

Q6. What is gradient accumulation and when do we use it?

Gradient accumulation means running forward + backward on several small batches, adding the gradients (instead of zeroing them), and then doing one optimizer step. We use it when we want a large effective batch size but cannot fit a large batch in memory (e.g., big model or long sequences).

Q7. What is automatic differentiation (autograd)?

Automatic differentiation is the machinery that automatically computes derivatives (gradients) of a function defined by a program. In DL, we use backward-mode autograd: the framework records operations during the forward pass and then runs a backward pass applying the chain rule, giving ∂L/∂θ for all parameters. We don’t derive or code gradients by hand.

Q8. Can we have non-differentiable operations in the network?

We can have them in the forward pass, but no gradient will flow through them (gradient = 0 or undefined). So training by gradient descent will not update parameters that only affect the loss through that operation. For things like discrete sampling (e.g., reinforcement learning), people use tricks (straight-through estimator, REINFORCE) to get a usable gradient signal. For standard supervised learning, we keep the forward path differentiable.

9. Applications (With How They Are Achieved)

1. Training Any Deep Network (Vision, NLP, etc.)

Applications: Image classification (CNN), machine translation (Seq2Seq, Transformers), speech recognition, recommendation.

How backprop achieves this:

• After each forward pass and loss computation, backprop computes ∂L/∂θ for every layer.
• The optimizer uses these gradients to update θ so the loss decreases. Without backprop, we could not train deep networks with gradient-based methods.

Example: ResNet for ImageNet: forward (conv layers → FC → Softmax → loss), then backprop through all layers to get gradients for every conv and FC weight, then SGD/Adam update.

2. Recurrent Networks (RNN, LSTM, GRU)

Applications: Language modeling, sequence-to-sequence, time-series prediction.

How backprop achieves this:

• BPTT (backpropagation through time): the same weights are reused at each time step; the gradient is computed and summed (or averaged) over time steps from the loss at the last (or each) step backward.
• This allows the RNN to learn long-range dependencies (within the limits of vanishing/exploding gradients).

Example: LSTM for next-word prediction: loss at final time step; backprop through the LSTM cells backward in time to get ∂L/∂(LSTM weights).

3. Custom Layers and Research

Applications: New architectures, custom loss terms, differentiable relaxations.

How backprop achieves this:

• Autograd lets you define new differentiable operations; as long as you implement forward and (if needed) backward, gradients flow through your layer. So researchers can try new layers without hand-deriving the full backprop for the whole net.
• Understanding backprop helps debug vanishing/exploding gradients and design better architectures (e.g., skip connections to improve gradient flow).

Example: Differentiable attention, custom normalization layers, or new activation functions—all rely on backprop (via autograd) to train.

4. Transfer Learning and Fine-Tuning

Applications: Fine-tuning pretrained models (e.g., BERT, ResNet) on a new task.

How backprop achieves this:

• We freeze some layers (no gradient computed or no update) and backprop only through the unfrozen layers (and the loss). So gradients are computed only for the parameters we want to update; the rest stay fixed.
• Backprop is the same algorithm; we just choose which θ receive gradients and get updated.

Example: BERT base, freeze all but the last two layers; backprop computes gradients only for those layers; optimizer updates only those parameters.

10. Advantages and Limitations (With Examples)

Advantages

1. Efficient gradient computation
One backward pass gives all ∂L/∂θ in time proportional to the forward pass (roughly 2–3×), instead of O(parameters) forward passes (finite differences).

Example: A network with 1M parameters: finite differences would need ~1M extra forward passes per step; backprop needs one backward pass.

2. Exact gradients
Gradients are analytically exact (up to floating-point), so optimization is correct and stable, unlike approximate methods.

Example: SGD with exact gradients converges reliably; noisy or approximate gradients can require more tuning or fail to converge.

3. Composable and automatic
Any differentiable module can be plugged in; autograd implements backprop automatically from the forward code, so we can build complex models without hand-coding gradients.

Example: Adding a new layer in PyTorch: implement forward; if you use standard ops, backward is free; if you add a custom op, you implement its backward once.

4. Scales to very deep networks
With care (good initialization, skip connections, stable activations), backprop can train 100+ layer networks. Without backprop, deep training would be infeasible.

Example: ResNet-152, Vision Transformers—all trained with backprop (and optimizers) on top of the same chain-rule idea.

Limitations

1. Vanishing gradients
In very deep or recurrent nets, gradients can shrink exponentially as they propagate backward, so early layers get almost no update (vanishing gradient). This limits how deep we can train or how long the dependency we can learn.

Example: Plain RNNs on long sequences: gradient from time T to time 1 can vanish; LSTMs/GRUs and better architectures mitigate this.

2. Exploding gradients
Gradients can also grow exponentially backward, causing numerical overflow and unstable updates. We address this with gradient clipping, better initialization, and architecture choices.

Example: Training a deep transformer without gradient clipping can lead to NaNs; clipping (e.g., max norm 1.0) keeps updates bounded.

3. Memory cost
Training requires storing activations (and sometimes other intermediates) for the backward pass, so training uses significantly more memory than inference. Very large batches or very deep nets can hit memory limits.

Example: Large language model training: we use gradient checkpointing (recompute some activations in backward instead of storing) to trade compute for memory.

4. Local minima and saddle points
Backprop gives local gradient information. In non-convex loss landscapes, we can get stuck in poor local minima or saddle points. This is a limitation of gradient-based optimization in general, not only of backprop; backprop just supplies the gradients.

Example: Training from a bad initialization can converge to a suboptimal solution; multiple restarts or better initialization help.

11. Interview-Oriented Key Takeaways

• Backpropagation = algorithm to compute ∂L/∂θ for all parameters by applying the chain rule from the loss backward through the network. It does not update parameters—the optimizer does.
• Forward pass computes activations and loss and stores values needed for backward; backward pass computes gradients layer by layer in reverse order.
• Chain rule: ∂L/∂x = (∂L/∂y)(∂y/∂x); each layer receives ∂L/∂(output), multiplies by local ∂(output)/∂(input) and ∂(output)/∂(params), and passes ∂L/∂(input) backward.
• Computational graph = graph of operations; backprop = reverse traversal with chain rule. Autograd = backprop done automatically by the framework.
• Gradient accumulation = add gradients over several micro-batches, then one optimizer step; used for large effective batch size with limited memory.
• Vanishing/exploding gradients are limitations of backprop in deep/recurrent nets; addressed by architecture (skip connections, LSTMs), initialization, and gradient clipping.

12. Common Interview Traps

Trap 1: “Backpropagation updates the weights.”

Wrong: Backprop is the step that changes the weights.

Correct: Backpropagation only computes the gradients ∂L/∂θ. The optimizer (SGD, Adam, etc.) updates the weights using those gradients (e.g., θ ← θ − η∇L). Backprop = gradient computation; optimizer = parameter update.

Trap 2: “Gradients are computed during the forward pass.”

Wrong: We get gradients as we go forward.

Correct: Gradients are computed in the backward pass. The forward pass computes activations and the loss and stores what the backward pass needs. The backward pass then runs after the loss is computed and propagates ∂L/∂θ from the output toward the input.

Trap 3: “Backpropagation is the same as gradient descent.”

Wrong: Backprop and gradient descent are the same thing.

Correct: Gradient descent (or SGD, Adam) is the optimization algorithm that updates parameters (e.g., θ ← θ − η∇L). Backpropagation is the algorithm that computes ∇L (the gradients). We use backprop to get the gradients, then gradient descent (or another optimizer) to update the parameters.

Trap 4: “We can skip the forward pass if we only want gradients.”

Wrong: We can compute gradients without running the forward pass.

Correct: The backward pass needs intermediate values (activations, pre-activation values, masks) from the forward pass to compute local derivatives. So we must run the forward pass first and store those values; only then can we run backprop.

Trap 5: “Backprop only works for fully connected networks.”

Wrong: Backprop is only for MLPs.

Correct: Backprop works for any differentiable composition of operations: CNNs (conv, pool, etc.), RNNs (BPTT), Transformers, custom layers. As long as each operation has a well-defined derivative, the chain rule applies and backprop gives ∂L/∂θ for all parameters.

Trap 6: “Autograd is a different algorithm from backpropagation.”

Wrong: Autograd is something other than backprop.

Correct: Autograd (automatic differentiation) is the implementation of backpropagation by the framework. It records the forward computation and replays it in reverse with the chain rule. So “autograd” is backprop done automatically from your forward code—same mathematical algorithm.

13. Simple Real-Life Analogy

Backpropagation is like blame assignment in a long chain: the final product (prediction) is wrong, and we ask each station (layer) in reverse order: “Given how much the loss is sensitive to your output, how much are you responsible, and how much should you pass back to the previous station?” The chain rule is the rule that splits that responsibility. The optimizer is the manager that actually changes each station’s settings (weights) based on that blame (gradient)—backprop only computes the blame; it doesn’t change anything by itself.

14. Backpropagation in System Design – Interview Traps (If Applicable)

Trap 1: Ignoring memory vs speed when scaling batch size

Wrong thinking: We should always use the largest batch that fits in memory for speed.

Correct thinking: Larger batches mean more activations to store for backprop, so memory can be the bottleneck. Use gradient accumulation if you need a large effective batch but can’t fit it in memory: same effective batch, lower peak memory. Trade-off: more forward/backward passes per optimizer step, so slightly more compute per step.

Example: Training a large transformer: batch size 8 fits, but we want effective batch 32; run 4 micro-batches with gradient accumulation, then one optimizer step.

Trap 2: Assuming gradients are always correct in production training

Wrong thinking: The framework always computes correct gradients; we don’t need to check.

Correct thinking: Numerical issues (overflow, underflow, NaNs), non-differentiable ops, or wrong loss can produce wrong or NaN gradients. In production, use gradient clipping to avoid explosions, monitor gradient norms or loss for NaNs, and validate that the loss decreases in early steps. For custom layers, consider gradient checking (compare autograd with finite differences) during development.

Example: NaNs in loss after a few steps—often exploding gradients or log(0); add gradient clipping and numerical stability (e.g., eps in log).

Trap 3: Forgetting that inference doesn’t need backprop

Wrong thinking: We need to keep the same code path for training and inference.

Correct thinking: Inference only needs the forward pass; no loss, no backward, no gradient storage. So we can disable gradient computation (torch.no_grad()), drop stored activations, and often use half precision or quantization to save memory and speed. Don’t carry training-only overhead (backprop, optimizer state) into inference.

Example: Serving a model in production: run only forward pass, no .backward(), and use inference mode to reduce memory and latency.

15. Interview Gold Line

Backpropagation is the algorithm that computes the gradient of the loss with respect to every parameter by applying the chain rule backward through the network; it does not update the weights—the optimizer does. It is the heart of deep learning because without it we could not efficiently train deep networks with gradient-based optimization.

16. Code Snippets (Python)

Minimal backprop by hand (single neuron, MSE)

import numpy as np

def forward(x, w, b):
    z = np.dot(x, w) + b
    a = 1 / (1 + np.exp(-z))  # sigmoid
    return z, a

def backward_mse(x, w, z, a, y_true):
    # L = (a - y)^2  =>  dL/da = 2(a - y)
    dL_da = 2 * (a - y_true)
    # a = sigmoid(z)  =>  da/dz = a * (1 - a)
    dL_dz = dL_da * (a * (1 - a))
    # z = xw + b  =>  dL/dw = dL_dz * x,  dL/db = dL_dz
    dL_dw = np.outer(x, dL_dz)
    dL_db = dL_dz
    return dL_dw, dL_db

# Usage (single sample):
x = np.array([1.0, 0.5])
w = np.array([0.3, -0.2])
b = 0.1
y_true = 1.0
z, a = forward(x, w, b)
dL_dw, dL_db = backward_mse(x, w, z, a, y_true)
# Optimizer would do: w -= lr * dL_dw, b -= lr * dL_db

PyTorch: backprop is automatic

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 32),
    nn.ReLU(),
    nn.Linear(32, 1),
    nn.Sigmoid()
)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCELoss()

# One step
x = torch.randn(4, 10, requires_grad=False)
y = torch.randint(0, 2, (4, 1)).float()
optimizer.zero_grad()
out = model(x)
loss = criterion(out, y)
loss.backward()   # backprop: fills .grad on all parameters
optimizer.step()  # update parameters using .grad

Interview tip: In PyTorch, loss.backward() runs backprop (autograd) and fills .grad on every parameter. optimizer.step() uses those gradients to update parameters. Always call optimizer.zero_grad() before a new backward pass so gradients are accumulated intentionally (e.g., for gradient accumulation) rather than by mistake.

17. Self-Check and “Think About It” Prompts

Self-check 1: What is the difference between backpropagation and the optimizer?

Self-check 2: Why do we need to store activations during the forward pass?

Self-check 3: In one sentence, what does the chain rule say for backprop?

Think about it: If a layer’s activation function were non-differentiable at some points (e.g., a hard threshold), what would happen to the gradient flowing through that layer?

Self-check answers (concise):
- 1: Backprop computes ∂L/∂θ; the optimizer updates θ using those gradients.
- 2: The backward pass needs those values to compute local derivatives (e.g., σ′(z), ReLU mask).
- 3: ∂L/∂x = (∂L/∂y)(∂y/∂x); we multiply the upstream gradient by the local derivative.
- 4: The gradient would be undefined or zero at those points, so parameters before that layer would not receive a useful gradient and might not train properly (or we’d need a surrogate gradient like in straight-through estimators).

18. Likely Interview Questions

• What is backpropagation and why do we need it?
• What is the difference between the forward pass and the backward pass?
• Explain the chain rule in the context of backpropagation.
• Does backpropagation update the weights? If not, what does?
• Why does training use more memory than inference?
• What is gradient accumulation and when would you use it?
• What is automatic differentiation (autograd)? How does it relate to backprop?
• Why can’t we compute gradients without the forward pass?
• What are vanishing and exploding gradients? How do we mitigate them?
• How does backprop work for CNNs? For RNNs (BPTT)?

19. Elevator Pitch

30 seconds:
Backpropagation is the algorithm that computes the gradient of the loss with respect to every parameter in the network. We do a forward pass to get predictions and the loss, then a backward pass applying the chain rule from the loss back through each layer. Each layer gets the gradient of the loss with respect to its output and multiplies by its local derivative to get gradients for its parameters and to pass backward to the previous layer. Backprop only computes gradients; the optimizer (SGD, Adam) updates the weights. It’s efficient—one backward pass gives all gradients—and it’s what makes training deep networks possible.

2 minutes:
Backpropagation is how we compute ∂L/∂θ for every weight and bias in a neural network. We first run a forward pass: input goes through each layer, we get the prediction and the loss, and we store activations and any values we’ll need for derivatives. Then we run the backward pass: we start with the gradient of the loss with respect to the output (∂L/∂ŷ), and we go layer by layer backward. At each layer we use the chain rule: we have ∂L/∂(this layer’s output), we multiply by the local derivative (∂output/∂input and ∂output/∂params) to get ∂L/∂(params) for this layer—which we give to the optimizer—and ∂L/∂(input)—which we pass to the previous layer. So backprop is just the chain rule applied in reverse order. It doesn’t update the weights; the optimizer does that using these gradients. Frameworks like PyTorch do this automatically (autograd): you write the forward code, they record it and run the backward pass. Without backprop we couldn’t train deep networks efficiently, because we’d need something like finite differences which would be way too slow. Limitations include vanishing and exploding gradients in very deep or recurrent nets, which we address with better architectures (skip connections, LSTMs), initialization, and gradient clipping.

20. One-Page Cheat Sheet (Quick Revision)

21. Formula Card

22. What’s Next and Revision Checklist

What’s Next

• Optimizers: They use the gradients ∂L/∂θ produced by backprop to update parameters (SGD, Adam, learning rate schedules).
• Activation Functions: Their derivatives (e.g., σ′(z), ReLU′) are the local derivatives in the chain rule; saturation (small σ′) contributes to vanishing gradients.
• Loss Functions: The loss gives the starting gradient ∂L/∂ŷ for the backward pass; different losses have different gradient forms.
• Vanishing/Exploding Gradients: Deeper dive on why they happen and fixes (ResNet, LayerNorm, clipping).
• Regularization and Generalization: Dropout, BatchNorm—their role in training and how backprop flows through them.

Revision Checklist

Before an interview, ensure you can:

1. Define backpropagation in one sentence (algorithm that computes ∂L/∂θ by chain rule backward).
2. State why we need it (efficient gradient computation for deep networks; one backward pass for all θ).
3. Distinguish forward pass vs backward pass (what is computed, order, what is stored).
4. Explain chain rule in one line and for one layer (∂L/∂input = upstream × local derivative).
5. Correct the trap: “Backprop updates the weights” (no—optimizer updates; backprop only computes gradients).
6. Correct the trap: “Gradients are computed in the forward pass” (no—backward pass).
7. Explain gradient accumulation (add gradients over micro-batches, one update) and when to use it.
8. Explain autograd (automatic backprop from forward code).
9. Mention vanishing/exploding gradients and one mitigation each (e.g., skip connections, gradient clipping).

Related Topics

• Calculus for Deep Learning (chain rule, gradients)
• Loss Functions (L and ∂L/∂ŷ — starting point for backprop)
• Activation Functions (local derivatives σ′, ReLU′)
• Optimizers (use ∂L/∂θ to update θ)
• Optimization Fundamentals (gradient descent, what we minimize)

End of Backpropagation study notes.