Deep Learning

He Initialization

Weight variance 2/fan_in, compensating for ReLU zeroing roughly half its inputs.

Why interviewers ask about it

Xavier (2/(fan_in+fan_out)) is for symmetric activations like tanh. Using Xavier with ReLU in a deep network causes slow signal decay.

Exponential decay of gradient magnitude through depth, caused by repeatedly multiplying by Jacobians with small spectral norm.

This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.