StackScholar

Deep Learning

Pre-Norm

x = x + Sublayer(LN(x)) - normalisation inside the residual branch, leaving the residual stream unnormalised.

Why interviewers ask about it

Creates an uninterrupted identity path to layer 0, which removed the warmup fragility of the original post-norm transformer and enabled 100+ layer models.

Related terms

Residual Connection

y = x + F(x), giving ∂y/∂x = I + ∂F/∂x - a gradient path that passes through no weight matrices.

LayerNorm

Normalisation across the feature dimension within each sample, independent of batch composition.

This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.