Deep Learning

RMSNorm

LayerNorm without mean-centring and without a bias - scale only.

Why interviewers ask about it

Ablations showed re-centring contributes almost nothing. ~10–15% faster at identical quality; used by Llama, Mistral and Gemma.

Normalisation across the feature dimension within each sample, independent of batch composition.

x = x + Sublayer(LN(x)) - normalisation inside the residual branch, leaving the residual stream unnormalised.

This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.