Hugging Face· PlatformCapability change

Fixing Gradient Accumulation

Fixes a bug in the Transformers Trainer where gradient accumulation loss was not computed correctly for token-level tasks such as causal LM training. Now the default loss function automatically handles the correct normalization, and a new API is exposed to allow users to pass custom loss functions.

From the source

To address this issue, we’re changing the way our models and training work in two ways: If users are using the “default” loss functions, we will automatically take into account the needed changes when using gradient accumulation, to make sure the proper loss is reported and utilized, fixing the core issue at hand.

huggingface.co

Announced

16 Oct 2024 00:00 UTC

Detected

03 Aug 2026 21:12 UTC (+656d after)