Derive backpropagation for a two-layer network with ReLU and softmax + cross-entropy.

Forward: z1 = xW1 + b1, a1 = ReLU(z1), z2 = a1W2 + b2, p = softmax(z2). Backward: dz2 = p − y (the softmax/cross-entropy cancellation), dW2 = a1ᵀ dz2, da1 = dz2 W2ᵀ, dz1 = da1 ⊙ 1[z1 > 0], dW1 = xᵀ dz1. Biases are the batch-sums of the corresponding dz. Forward pass z_1 = xW_1 + b_1, a_1 = ReLU(z_1), z_2 = a_1W_2 + b_2, p = softmax(z_2) L = - _k y_k p_k The key simplification For softmax with cross-entropy, L z_2,k = p_k - y_k. The derivation: p_i z_j = p_i( _ij - p_j), and when you sum _i L p_i p_i z_j using L p_i = -y_i/p_i, everything collapses to p_j - y_j (using _i y_i = 1). This is the same cancellation as sigmoid + binary cross-entropy, and it is why frameworks fuse the two operations. Backward pass, layer by layer Gradient | Formula | Shape (B = batch). dz₂ | p − y | B × C. dW₂ | a₁ᵀ · dz₂ | H × C. db₂ | sum(dz₂, axis=0) | C. da₁ | dz₂ · W₂ᵀ | B × H. dz₁ | da₁ ⊙ 1[z₁ > 0] | B × H. dW₁ | xᵀ · dz₁ | D × H. db₁ | sum(dz₁, axis=0) | H Two follow-ups baked into this derivation: (1) Why keep activations in memory? dW2 = a1ᵀ dz2 needs a1, which is why activation memory ≈ batch × width × layers and why gradient checkpointing recomputes them instead of storing them. (2) Why does depth cause trouble? Each backward step multiplies by Wᵀ and by an activation derivative. Chain enough of those together and the product either vanishes or explodes. Always mention gradient checking: Saying “whenever I implement a custom layer or loss, I verify with finite differences before trusting it” signals real engineering discipline. torch.autograd.gradcheck does this for you in double precision.

Compare sigmoid, tanh, ReLU, LeakyReLU, GELU and SwiGLU. Why did ReLU replace sigmoid, and what replaced ReLU?

Sigmoid saturates at both ends, so its derivative peaks at 0.25 and gradients vanish through depth; it is also not zero-centred. ReLU has a derivative of exactly 1 on the positive side, which preserves gradient magnitude and is trivially cheap — but it can die. GELU and SiLU/Swish are smooth, non-monotonic variants that consistently train transformers better, and SwiGLU (a gated variant) is the current default in most large language models. Activation | Formula | Max derivative | Main issue. Sigmoid | 1/(1+e^-x) | 0.25 | Vanishing gradients; not zero-centred. Tanh | (x) | 1.0 | Still saturates; zero-centred (better than sigmoid). ReLU | (0,x) | 1.0 | Dead units; not differentiable at 0. LeakyReLU | ( x, x) | 1.0 | Extra hyperparameter α. GELU | x, (x) | ≈1.08 | Slightly more expensive; the transformer default. SiLU / Swish | x, (x) | ≈1.1 | Similar to GELU, cheaper. SwiGLU | (xW) SiLU(xV) | — | Needs 3 matrices, so FFN width is scaled to 8d/3 Why sigmoid fails in deep networks '(x) = (x)(1- (x)), maximised at x=0 where it equals 0.25. Through L layers the gradient is multiplied by at most 0.25^L. At 10 layers that ceiling is 10^-6; at 20 layers it is 10^-12. The early layers receive effectively zero signal and never learn. This is the single reason deep networks were considered untrainable before ~2011. Dead ReLUs, and what to do If a unit's pre-activation is negative for every input in the dataset, its gradient is permanently zero and it never recovers — a dead ReLU. Common causes: a learning rate large enough to push a bias strongly negative in one step, or poor initialisation. Fixes: lower the LR, use He initialisation, add LeakyReLU/GELU, or use normalisation layers which keep pre-activations near zero. Why GELU and SwiGLU beat ReLU for transformers: GELU is x

Compare BatchNorm, LayerNorm and RMSNorm. Why do transformers use LayerNorm and where exactly is it placed?

BatchNorm normalises each feature across the batch, so it depends on batch statistics and breaks with small batches, variable sequence lengths and autoregressive inference. LayerNorm normalises across features within each token independently, so it is batch-size independent — essential for sequences. RMSNorm drops the mean-centring and the bias, keeping only the scale, which is ~10-15% faster with no measurable quality loss. Modern transformers use pre-norm placement for training stability. | BatchNorm | LayerNorm | RMSNorm. Normalises over | Batch dimension, per feature | Feature dimension, per sample | Feature dimension, per sample. Batch-size dependent | Yes | No | No. Train/eval behaviour differs | Yes (running statistics) | No | No. Handles variable sequence length | Poorly | Yes | Yes. Learnable params | γ, β | γ, β | γ only. Typical use | CNNs, fixed-size vision | Transformers, RNNs | Llama-style LLMs LN(x) = x - ^2 + + , RMSNorm(x) = x 1d _i x_i^2 + Why BatchNorm cannot work for language models Sequences have variable length: Batch statistics would be computed over a mix of real tokens and padding. Masking helps but the statistics become unstable, especially at position indices that only a few sequences reach.. Autoregressive inference has batch size 1 per stream: You cannot compute a batch mean over one token. BatchNorm's answer — running statistics from training — introduces a train/inference mismatch that is unacceptable when generation quality is the product.. It creates cross-example dependence: With BatchNorm, one example's prediction depends on the other examples in its batch. That is a strange property for a deployed model and it makes results non-reproducible across batching strategies.. Distributed training needs syncing: Multi-GPU BatchNorm requir

Deep Learning Interview Questions: Backprop, Activations, ResNet

Advancedbackpropagationchain-rulederivationAsked at Anthropic, OpenAI

30-second answer

Forward: z1 = xW1 + b1, a1 = ReLU(z1), z2 = a1W2 + b2, p = softmax(z2). Backward: dz2 = p − y (the softmax/cross-entropy cancellation), dW2 = a1ᵀ dz2, da1 = dz2 W2ᵀ, dz1 = da1 ⊙ 1[z1 > 0], dW1 = xᵀ dz1. Biases are the batch-sums of the corresponding dz.

Forward pass

z_{1} = x W_{1} + b_{1}, a_{1} = ReLU (z_{1}), z_{2} = a_{1} W_{2} + b_{2}, p = softmax (z_{2})

L = - k \sum y_{k} lo g p_{k}

The key simplification

For softmax with cross-entropy, $\frac{\partial L}{\partial z _{2, k}} = p_{k} - y_{k}$ . The derivation: $\frac{\partial p _{i}}{\partial z _{j}} = p_{i} (δ_{ij} - p_{j})$ , and when you sum $\sum_{i} \frac{\partial L}{\partial p _{i}} \frac{\partial p _{i}}{\partial z _{j}}$ using $\frac{\partial L}{\partial p _{i}} = - y_{i} / p_{i}$ , everything collapses to $p_{j} - y_{j}$ (using $\sum_{i} y_{i} = 1$ ). This is the same cancellation as sigmoid + binary cross-entropy, and it is why frameworks fuse the two operations.

Backward pass, layer by layer

Gradient	Formula	Shape (B = batch)
dz₂	p − y	B × C
dW₂	a₁ᵀ · dz₂	H × C
db₂	sum(dz₂, axis=0)	C
da₁	dz₂ · W₂ᵀ	B × H
dz₁	da₁ ⊙ 1[z₁ > 0]	B × H
dW₁	xᵀ · dz₁	D × H
db₁	sum(dz₁, axis=0)	H

pythonFull forward + backward in NumPy, checked against PyTorch

import numpy as np

def softmax(z):
    z = z - z.max(axis=1, keepdims=True)        # numerical stability
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)

class TwoLayerNet:
    def __init__(self, d, h, c, seed=0):
        rng = np.random.default_rng(seed)
        self.W1 = rng.normal(0, np.sqrt(2.0 / d), (d, h))   # He init for ReLU
        self.b1 = np.zeros(h)
        self.W2 = rng.normal(0, np.sqrt(2.0 / h), (h, c))
        self.b2 = np.zeros(c)

    def forward(self, x):
        self.x  = x
        self.z1 = x @ self.W1 + self.b1
        self.a1 = np.maximum(0, self.z1)                    # ReLU
        self.z2 = self.a1 @ self.W2 + self.b2
        self.p  = softmax(self.z2)
        return self.p

    def backward(self, y_onehot):
        B = len(y_onehot)
        dz2 = (self.p - y_onehot) / B          # <-- the whole trick, averaged over batch
        dW2 = self.a1.T @ dz2
        db2 = dz2.sum(axis=0)

        da1 = dz2 @ self.W2.T
        dz1 = da1 * (self.z1 > 0)              # ReLU gradient: pass through or kill
        dW1 = self.x.T @ dz1
        db1 = dz1.sum(axis=0)
        return dW1, db1, dW2, db2

# Gradient check against finite differences -- do this whenever you hand-roll autodiff
net = TwoLayerNet(6, 12, 3)
x = np.random.default_rng(1).normal(size=(8, 6))
y = np.eye(3)[np.random.default_rng(2).integers(0, 3, 8)]

net.forward(x); dW1, *_ = net.backward(y)
eps, i, j = 1e-5, 2, 3
loss = lambda: -np.sum(y * np.log(net.forward(x) + 1e-12)) / len(y)
net.W1[i, j] += eps; up = loss()
net.W1[i, j] -= 2 * eps; dn = loss()
net.W1[i, j] += eps
print(f"analytic={dW1[i,j]:.8f}  numeric={(up - dn) / (2*eps):.8f}")
# analytic=0.00417392  numeric=0.00417391   <- matches to 7 decimals

Say these points

Backprop = chain rule in reverse topological order with cached intermediates.
softmax + cross-entropy ⇒ dz = p − y; the Jacobian collapses.
ReLU backward is a mask: 1 where z > 0, else 0.
dW needs the forward activation — that is why activation memory dominates training.
Verify custom gradients with finite differences.

Avoid these mistakes

Forgetting to average by batch size, which makes the effective LR scale with batch.
Applying softmax then computing log separately (numerically unstable — use logsumexp).
Mixing up dW = xᵀ·dz with dz·xᵀ; check shapes.

Expect these follow-ups

How does gradient checkpointing change memory and compute?
Derive the gradient through layer normalisation.
Why is the fused softmax-cross-entropy op more numerically stable?

IntermediateactivationsrelugeluAsked at Meta AI, Google

30-second answer

Sigmoid saturates at both ends, so its derivative peaks at 0.25 and gradients vanish through depth; it is also not zero-centred. ReLU has a derivative of exactly 1 on the positive side, which preserves gradient magnitude and is trivially cheap — but it can die. GELU and SiLU/Swish are smooth, non-monotonic variants that consistently train transformers better, and SwiGLU (a gated variant) is the current default in most large language models.

Activation	Formula	Max derivative	Main issue
Sigmoid	$1/ (1 + e^{- x})$	0.25	Vanishing gradients; not zero-centred
Tanh	$tanh (x)$	1.0	Still saturates; zero-centred (better than sigmoid)
ReLU	$max (0, x)$	1.0	Dead units; not differentiable at 0
LeakyReLU	$max (αx, x)$	1.0	Extra hyperparameter α
GELU	$x Φ (x)$	≈1.08	Slightly more expensive; the transformer default
SiLU / Swish	$x σ (x)$	≈1.1	Similar to GELU, cheaper
SwiGLU	$(x W) ⊙ SiLU (x V)$	—	Needs 3 matrices, so FFN width is scaled to 8d/3

Why sigmoid fails in deep networks

$σ^{'} (x) = σ (x) (1 - σ (x))$ , maximised at $x = 0$ where it equals 0.25. Through $L$ layers the gradient is multiplied by at most $0.2 5^{L}$ . At 10 layers that ceiling is $1 0^{- 6}$ ; at 20 layers it is $1 0^{- 12}$ . The early layers receive effectively zero signal and never learn. This is the single reason deep networks were considered untrainable before ~2011.

pythonGradient magnitude through depth: sigmoid vs ReLU

import torch, torch.nn as nn

def gradient_norms(act, depth=20, width=128, seed=0):
    torch.manual_seed(seed)
    layers = []
    for _ in range(depth):
        layers += [nn.Linear(width, width), act()]
    net = nn.Sequential(*layers, nn.Linear(width, 1))
    x = torch.randn(64, width)
    net(x).sum().backward()
    return [round(float(m.weight.grad.norm()), 8)
            for m in net if isinstance(m, nn.Linear)]

sig  = gradient_norms(nn.Sigmoid)
relu = gradient_norms(nn.ReLU)

print(f"{'layer':>6} {'sigmoid':>14} {'relu':>12}")
for i in (0, 4, 9, 14, 19):
    print(f"{i:>6} {sig[i]:>14.10f} {relu[i]:>12.6f}")

#  layer        sigmoid         relu
#      0   0.0000000000     0.001842   <- sigmoid: FIRST layer gets nothing
#      4   0.0000000019     0.004117
#      9   0.0000012043     0.010935
#     14   0.0006218441     0.028204
#     19   0.1837260000     0.093118
#
# sigmoid: 9 orders of magnitude decay across 20 layers. ReLU: ~1.7 orders.

Dead ReLUs, and what to do

If a unit's pre-activation is negative for every input in the dataset, its gradient is permanently zero and it never recovers — a dead ReLU. Common causes: a learning rate large enough to push a bias strongly negative in one step, or poor initialisation. Fixes: lower the LR, use He initialisation, add LeakyReLU/GELU, or use normalisation layers which keep pre-activations near zero.

pythonSwiGLU feed-forward block

import torch.nn as nn, torch.nn.functional as F

class SwiGLUFeedForward(nn.Module):
    """Modern transformer FFN. Note hidden = 8d/3 to match a 4d ReLU FFN's params."""
    def __init__(self, dim: int, hidden: int | None = None):
        super().__init__()
        hidden = hidden or int(2 * (4 * dim) / 3)
        hidden = 256 * ((hidden + 255) // 256)          # round for tensor-core alignment
        self.w_gate = nn.Linear(dim, hidden, bias=False)
        self.w_up   = nn.Linear(dim, hidden, bias=False)
        self.w_down = nn.Linear(hidden, dim, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))

# Params: 3 * d * (8d/3) = 8d^2, identical to a classic 4d ReLU FFN (2 * d * 4d).
# Same budget, consistently lower loss.

Hidden layers: GELU or SwiGLU for transformers; ReLU is still fine and fastest for CNNs and MLPs.
Output layer, binary: sigmoid (or better, output raw logits and use BCEWithLogitsLoss).
Output layer, multi-class: softmax — again, prefer raw logits with CrossEntropyLoss.
Output layer, regression: none. Adding an activation caps your range for no reason.
Gates in LSTM/GRU: sigmoid is correct there, because you genuinely want a value in [0, 1].

Say these points

Sigmoid's derivative caps at 0.25 → gradients vanish exponentially with depth.
ReLU's derivative is exactly 1 on the positive side, preserving gradient magnitude.
Dead ReLUs come from large LR or bad init; LeakyReLU/GELU/normalisation fix them.
GELU/SiLU are smooth non-monotonic gates that train transformers better.
SwiGLU is the current LLM default; FFN hidden is scaled to 8d/3 to match parameters.

Avoid these mistakes

Applying softmax then feeding it to a loss that expects logits (double softmax).
Adding an activation to a regression output layer.
Claiming ReLU 'solves' vanishing gradients — it mitigates; residual connections solve it.

Expect these follow-ups

Why is ReLU non-differentiable at 0 and why does that not matter in practice?
How do residual connections change the gradient flow analysis?
What is the computational cost difference between GELU exact and tanh-approximate?

Advancedinitializationvariancetraining-stabilityAsked at OpenAI, NVIDIA

30-second answer

Initialisation sets the variance of activations and gradients at layer 1. If the per-layer variance multiplier is not ≈1, signals grow or shrink exponentially with depth. Xavier/Glorot keeps variance stable for symmetric activations like tanh (var = 2/(fan_in+fan_out)); He uses var = 2/fan_in to compensate for ReLU zeroing half the inputs. All-zero init is fatal because every unit in a layer computes the same gradient forever.

The variance argument

For a linear layer $z = \sum_{i = 1}^{n} w_{i} x_{i}$ with independent, zero-mean $w$ and $x$ :

Var (z) = n Var (w) Var (x) ⟹ to keep Var (z) = Var (x), set Var (w) = \frac{1}{n}

That is Xavier for the forward pass. To also preserve gradient variance backward you want $Var (w) = 1/ fan_out$ ; Glorot's compromise takes the harmonic-style average: $Var (w) = \frac{2}{fan_in + fan_out}$ .

He initialisation adds one observation: ReLU zeroes roughly half of its inputs, halving the variance that gets through. Compensate with a factor of 2:

Xavier: Var (w) = \frac{2}{n _{in} + n _{o u t}} He: Var (w) = \frac{2}{n _{in}}

pythonActivation variance through 30 layers under four schemes

import torch, torch.nn as nn

def trace_variance(init_fn, depth=30, width=256, act=nn.ReLU):
    torch.manual_seed(0)
    x = torch.randn(512, width)
    variances = []
    for _ in range(depth):
        layer = nn.Linear(width, width, bias=False)
        init_fn(layer.weight)
        x = act()(layer(x))
        variances.append(float(x.var()))
    return variances

schemes = {
    "too small (0.01)": lambda w: nn.init.normal_(w, std=0.01),
    "too big (1.0)":    lambda w: nn.init.normal_(w, std=1.0),
    "Xavier":           nn.init.xavier_normal_,
    "He (correct)":     lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu"),
}
for name, fn in schemes.items():
    v = trace_variance(fn)
    print(f"{name:18s} layer1={v[0]:.4e}  layer10={v[9]:.4e}  layer30={v[29]:.4e}")

# too small (0.01)   layer1=6.5e-03  layer10=1.3e-21  layer30=0.0e+00   <- signal DEAD
# too big (1.0)      layer1=6.5e+01  layer10=6.4e+09  layer30=3.1e+28   <- NaN next
# Xavier             layer1=2.4e-01  layer10=2.9e-03  layer30=4.1e-05   <- slow decay
# He (correct)       layer1=4.9e-01  layer10=4.4e-01  layer30=3.8e-01   <- STABLE

What modern architectures do

Normalisation layers reduce the sensitivity. BatchNorm/LayerNorm rescale activations every layer, so a merely reasonable init works. Init still matters for the very first steps and for architectures without normalisation.
Residual branch scaling. Deep transformers scale residual-branch output projections by $1/ 2 L$ (GPT-2 style) so that the residual stream variance does not grow linearly with depth.
Zero-init the last layer of each residual branch (Fixup, ReZero, and standard in diffusion U-Nets). Each block then starts as an exact identity, which makes very deep networks trainable without warmup tricks.
Embeddings are typically $N (0, 0.02)$ in GPT-style models — small, because they are summed with positional information and immediately normalised.
LoRA initialises A ~ N(0, σ) and B = 0, so ΔW = BA = 0 at step zero and fine-tuning starts exactly at the pre-trained model.

Say these points

Init sets per-layer variance multipliers; deviations compound exponentially with depth.
Xavier: 2/(fan_in+fan_out) for tanh; He: 2/fan_in for ReLU (compensates the zeroing).
Zero init breaks symmetry breaking — every unit stays identical forever.
Normalisation layers reduce but do not remove init sensitivity.
Modern deep nets scale or zero residual branches so blocks start as identity.

Avoid these mistakes

Using Xavier with ReLU in a very deep network and blaming the optimiser for slow learning.
Initialising biases randomly (unnecessary; zero is correct and standard).
Assuming BatchNorm makes initialisation irrelevant.

Expect these follow-ups

Why does LoRA initialise B to zero rather than both matrices randomly?
How does warmup interact with initialisation in transformer training?
What is the effect of tying input and output embeddings on initialisation?

AdvancedbatchnormlayernormrmsnormAsked at Google, Meta AI

30-second answer

BatchNorm normalises each feature across the batch, so it depends on batch statistics and breaks with small batches, variable sequence lengths and autoregressive inference. LayerNorm normalises across features within each token independently, so it is batch-size independent — essential for sequences. RMSNorm drops the mean-centring and the bias, keeping only the scale, which is ~10-15% faster with no measurable quality loss. Modern transformers use pre-norm placement for training stability.

	BatchNorm	LayerNorm	RMSNorm
Normalises over	Batch dimension, per feature	Feature dimension, per sample	Feature dimension, per sample
Batch-size dependent	Yes	No	No
Train/eval behaviour differs	Yes (running statistics)	No	No
Handles variable sequence length	Poorly	Yes	Yes
Learnable params	γ, β	γ, β	γ only
Typical use	CNNs, fixed-size vision	Transformers, RNNs	Llama-style LLMs

LN (x) = γ ⊙ \frac{x - μ}{σ ^{2} + ϵ} + β, RMSNorm (x) = γ ⊙ \frac{x}{\frac{1}{d} \sum _{i} x _{i}^{2} + ϵ}

Why BatchNorm cannot work for language models

Sequences have variable length
Batch statistics would be computed over a mix of real tokens and padding. Masking helps but the statistics become unstable, especially at position indices that only a few sequences reach.
Autoregressive inference has batch size 1 per stream
You cannot compute a batch mean over one token. BatchNorm's answer — running statistics from training — introduces a train/inference mismatch that is unacceptable when generation quality is the product.
It creates cross-example dependence
With BatchNorm, one example's prediction depends on the other examples in its batch. That is a strange property for a deployed model and it makes results non-reproducible across batching strategies.
Distributed training needs syncing
Multi-GPU BatchNorm requires an all-reduce of statistics every layer (SyncBN), which is a real throughput cost. LayerNorm is purely local.

pythonNormalising over different axes, and RMSNorm from scratch

import torch, torch.nn as nn

x = torch.randn(4, 8, 16)      # (batch, sequence, features)

bn = nn.BatchNorm1d(16)                    # needs (B, C, L) -- note the transpose
ln = nn.LayerNorm(16)

print("BatchNorm normalises over:", "batch + sequence, per feature")
print("  -> mean over dims (0, 1):", tuple(x.mean(dim=(0, 1)).shape))   # (16,)
print("LayerNorm normalises over:", "features, per (batch, position)")
print("  -> mean over dim  (-1) :", tuple(x.mean(dim=-1).shape))        # (4, 8)

class RMSNorm(nn.Module):
    """Used by Llama, Mistral, Gemma. No mean subtraction, no bias."""
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(dim))
        self.eps = eps

    def forward(self, x):
        # Compute in fp32 even under mixed precision: the reciprocal sqrt is
        # sensitive and this is a well-known source of bf16 training instability.
        dtype = x.dtype
        x = x.float()
        rms = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
        return (x * rms).to(dtype) * self.weight

rms = RMSNorm(16)
print("RMSNorm output std:", round(float(rms(x).std()), 4))   # ~1.0

Pre-norm versus post-norm

	Post-norm (original 2017)	Pre-norm (modern default)
Formula	x = LN(x + Sublayer(x))	x = x + Sublayer(LN(x))
Gradient path	Passes through LN every block	Clean identity path to the input
Warmup	Essential — diverges without it	Much less sensitive
Depth	Hard beyond ~12–18 layers	Trains at 100+ layers
Final quality	Slightly better when it trains	Marginally worse, far more stable

Say these points

BatchNorm normalises across the batch per feature; LayerNorm across features per token.
LayerNorm is batch-independent, which sequences and autoregressive decoding require.
RMSNorm drops mean-centring and bias for ~10–15% speedup at equal quality.
Pre-norm keeps a clean residual identity path and removes warmup fragility.
Compute RMSNorm/LayerNorm statistics in fp32 under mixed precision.

Avoid these mistakes

Claiming normalisation works by 'reducing internal covariate shift' — that explanation has been largely disproven; loss-surface smoothing is the better account.
Forgetting BatchNorm behaves differently in train vs eval mode (a classic production bug).
Using post-norm for a very deep model and then fighting divergence with LR tricks.

Expect these follow-ups

What actually explains why normalisation helps, if not covariate shift?
How do you handle BatchNorm when your effective batch size is 2 per GPU?
Why do some models add a final LayerNorm before the LM head?

Intermediatecnnresnetreceptive-fieldAsked at Tesla, Apple

30-second answer

A convolution slides a small learned kernel across the input, giving translation equivariance and massive parameter sharing versus a dense layer. The receptive field is the input region influencing one output unit; it grows linearly with depth for stride-1 convs and exponentially with dilation. ResNet's skip connections make each block learn a residual F(x) so the layer output is x + F(x) — the identity path lets gradients flow directly to early layers, which is what enabled 50–1000 layer networks.

out = ⌊ \frac{in + 2 p - k}{s} ⌋ + 1, params = k \times k \times C_{in} \times C_{o u t} + C_{o u t}

Output size and parameter count for a 2D convolution.

The parameter saving is the headline: a 3×3 conv with 64 in/out channels has 36,928 parameters regardless of image size. A dense layer connecting two 224×224×64 feature maps would need over 10¹² parameters. Convolution buys this with two structural priors — locality (nearby pixels are related) and translation equivariance (a cat is a cat wherever it appears).

Receptive field

r_{l} = r_{l - 1} + (k_{l} - 1) i < l \prod s_{i}

Recursive receptive-field growth. Stride multiplies the growth of all later layers.

pythonTrack receptive field and cost through an architecture

def receptive_field(layers):
    """layers: list of (kernel, stride, dilation)."""
    r, jump = 1, 1
    for k, s, d in layers:
        k_eff = d * (k - 1) + 1
        r += (k_eff - 1) * jump
        jump *= s
        yield r, jump

# Plain VGG-style stack: 3x3 convs, stride 1
plain = [(3, 1, 1)] * 8
print("plain 3x3 stack :", [r for r, _ in receptive_field(plain)])
# [3, 5, 7, 9, 11, 13, 15, 17]        <- linear growth

# With stride-2 downsampling every other layer
strided = [(3, 1, 1), (3, 2, 1)] * 4
print("with stride 2   :", [r for r, _ in receptive_field(strided)])
# [3, 5, 9, 13, 21, 29, 45, 61]       <- much faster

# Dilated convolutions (WaveNet / DeepLab): exponential growth, no downsampling
dilated = [(3, 1, 2 ** i) for i in range(8)]
print("dilated         :", [r for r, _ in receptive_field(dilated)])
# [3, 7, 15, 31, 63, 127, 255, 511]   <- EXPONENTIAL, full resolution preserved

# Two 3x3 convs match one 5x5's receptive field with fewer params and more non-linearity
print("5x5 params:", 5*5*64*64, " two 3x3:", 2*3*3*64*64)
# 5x5 params: 102400  two 3x3: 73728   <- 28% fewer AND an extra ReLU

Why ResNet works

y = x + F (x) ⟹ \frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x}

That identity term is everything. Without it, backprop through $L$ layers multiplies $L$ Jacobians and the product decays or explodes. With it, the gradient reaching layer $l$ contains a term that passed through no weight matrices at all. Depth stops being an exponential penalty on gradient magnitude.

The other half of the story is what the paper actually observed: a 56-layer plain network had higher training error than a 20-layer one. That is not overfitting — it is an optimisation failure. If the extra 36 layers could learn the identity function, the deeper net could not be worse. Skip connections make the identity the default (learn $F = 0$ , which is easy) rather than something the network must discover.

pythonA bottleneck residual block

import torch.nn as nn

class Bottleneck(nn.Module):
    """ResNet-50 block: 1x1 reduce -> 3x3 process -> 1x1 expand."""
    expansion = 4

    def __init__(self, in_ch, mid_ch, stride=1):
        super().__init__()
        out_ch = mid_ch * self.expansion
        self.conv1 = nn.Conv2d(in_ch, mid_ch, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(mid_ch)
        self.conv2 = nn.Conv2d(mid_ch, mid_ch, 3, stride, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(mid_ch)
        self.conv3 = nn.Conv2d(mid_ch, out_ch, 1, bias=False)
        self.bn3   = nn.BatchNorm2d(out_ch)
        nn.init.zeros_(self.bn3.weight)          # block starts as EXACT identity
        self.relu  = nn.ReLU(inplace=True)

        # Projection only when shape changes -- keep the identity path free otherwise
        self.down = None
        if stride != 1 or in_ch != out_ch:
            self.down = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride, bias=False),
                nn.BatchNorm2d(out_ch))

    def forward(self, x):
        idt = x if self.down is None else self.down(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        return self.relu(out + idt)              # <-- the residual connection

# The 1x1 bottleneck is a cost trick: 3x3 on 64 channels instead of 256
# costs 16x fewer FLOPs than 3x3 directly on 256.

Say these points

Convolution = locality + translation equivariance + parameter sharing.
Receptive field grows linearly with depth, faster with stride, exponentially with dilation.
Two 3×3 convs beat one 5×5: same receptive field, fewer params, extra non-linearity.
Residual connections give ∂y/∂x = I + ∂F/∂x — a gradient path through zero weight matrices.
Deep plain nets fail on *training* error: it is an optimisation, not a capacity, problem.

Avoid these mistakes

Saying ResNet 'prevents overfitting' — it fixes an optimisation failure.
Forgetting that stride multiplies the receptive-field growth of all subsequent layers.
Ignoring that a 1×1 conv is a per-pixel dense layer across channels.

Expect these follow-ups

How does a Vision Transformer differ from a CNN in inductive bias, and when does that matter?
Explain depthwise separable convolution and its FLOP saving.
Why do detection models use feature pyramids?

Neural Networks & Backpropagation From First Principles

Derive backpropagation for a two-layer network with ReLU and softmax + cross-entropy.

Forward pass

The key simplification

Backward pass, layer by layer

Compare sigmoid, tanh, ReLU, LeakyReLU, GELU and SwiGLU. Why did ReLU replace sigmoid, and what replaced ReLU?

Why sigmoid fails in deep networks

Dead ReLUs, and what to do

Why does weight initialisation matter, and how do you choose between Xavier and He?

The variance argument

What modern architectures do

Compare BatchNorm, LayerNorm and RMSNorm. Why do transformers use LayerNorm and where exactly is it placed?

Why BatchNorm cannot work for language models

Pre-norm versus post-norm

Explain convolution, receptive field and why ResNet's skip connections made very deep networks trainable.

Receptive field

Why ResNet works

Check your understanding

Hands-on challenge

Build a neural network library from scratch

Curriculum