# Hugging Face — GaLore: Advancing Large Model Training on Consumer-grade Hardware

- Company: Hugging Face (huggingface.co)
- Announced: 2024-03-20T00:00:00+00:00
- Category: research-paper
- Subject: Platform
- Models affected: Llama
- Source: https://huggingface.co/blog/galore
- Record: https://forck.live/items/1924-galore-advancing-large-model-training-on-consumer-grade-hardware

GaLore is a method that reduces memory footprint for training large language models by projecting gradients into low-rank subspaces, enabling training of up to 7 billion parameter models on consumer GPUs like the NVIDIA RTX 4090. It achieves over 82.5% reduction in memory for optimizer states and can be combined with 8-bit optimizers for further savings.

## Evidence

Verbatim from https://huggingface.co/blog/galore:

> “more than 82.5% reduction in memory for storing optimizer states during training”

---

Record: https://forck.live/items/1924-galore-advancing-large-model-training-on-consumer-grade-hardware
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md