# Hugging Face — Fixing Gradient Accumulation

- Company: Hugging Face (huggingface.co)
- Announced: 2024-10-16T00:00:00+00:00
- Category: capability-change
- Subject: Platform
- Source: https://huggingface.co/blog/gradient_accumulation
- Record: https://forck.live/items/1814-fixing-gradient-accumulation

Fixes a bug in the Transformers Trainer where gradient accumulation loss was not computed correctly for token-level tasks such as causal LM training. Now the default loss function automatically handles the correct normalization, and a new API is exposed to allow users to pass custom loss functions.

## Evidence

Verbatim from https://huggingface.co/blog/gradient_accumulation:

> To address this issue, we’re changing the way our models and training work in two ways: If users are using the “default” loss functions, we will automatically take into account the needed changes when using gradient accumulation, to make sure the proper loss is reported and utilized, fixing the core issue at hand.

---

Record: https://forck.live/items/1814-fixing-gradient-accumulation
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md