# Hugging Face — Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2

- Company: Hugging Face (huggingface.co)
- Announced: 2024-08-21T00:00:00+00:00
- Category: developer-tool-release
- Subject: Platform
- Models affected: llama2-7B, mistral-7B, granite-8B-code, Llama 2 and 3, Mistral, Mixtral, Granite, DBRX, Falcon, Gemma, OLMo, Phi 1, 2, and 3, phi3, Qwen 2 and 2 MoE, StableLM, StarCoder 2
- Source: https://huggingface.co/blog/packing-with-FA2
- Record: https://forck.live/items/1836-improving-hugging-face-training-efficiency-through-packing-with-flash

Hugging Face announces a new feature in Transformers and TRL libraries that allows packing of training examples without padding, compatible with Flash Attention 2, using a new DataCollatorWithFlattening and a padding_free flag. It reports up to 2x throughput improvement and up to 20% peak memory reduction on certain datasets, with no impact on convergence.

## Evidence

Verbatim from https://huggingface.co/blog/packing-with-FA2:

> Hugging Face Transformers now addresses this with a new feature that maintains boundary awareness during packing, alongside the introduction of a new data collator, DataCollatorWithFlattening.

---

Record: https://forck.live/items/1836-improving-hugging-face-training-efficiency-through-packing-with-flash
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md