From the source
We do not want the tensors in the network—either the weights, activations or gradients—to grow too large or too small.
Very small and very large tensors cause a variety of problems not just limited to numerical underflow and overflow.
For example, weight matrices changing size during training makes it harder to design training algorithms—since the relative size of updates to weights has a significant impact on the speed of learning.
The gold standard for keeping tensors healthy is to normalize them.
Normalization is commonplace for activation vectors, where we use techniques like layer norm to put the activations on a good scale before passing them to the next layer.
It is also commonplace to normalize gradient updates, where we can interpret fast training algorithms like the Muon optimizer as spectrally normalizing the updates.
Normalization provides us with certainty about the sizes of tensors—without needing to check Wandb!—and when training large neural networks with many interacting components, having certainty about the network internals is valuable.
…
