Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2
Developer tool
Hugging Face announces a new feature in Transformers and TRL libraries that allows packing of training examples without padding, compatible with Flash Attention 2, using a new DataCollatorWithFlattening and a padding_free flag.
It reports up to 2x throughput improvement and up to 20% peak memory reduction on certain datasets, with no impact on convergence.
