From the source
From the source
Fixes a bug in the Transformers Trainer where gradient accumulation loss was not computed correctly for token-level tasks such as causal LM training.
Now the default loss function automatically handles the correct normalization, and a new API is exposed to allow users to pass custom loss functions.
From the source
To address this issue, we’re changing the way our models and training work in two ways: If users are using the “default” loss functions, we will automatically take into account the needed changes when using gradient accumulation, to make sure the proper loss is reported and utilized, fixing the core issue at hand.
huggingface.co
