# Hugging Face — Fine-tuning Llama 2 70B using PyTorch FSDP

- Company: Hugging Face (huggingface.co)
- Announced: 2023-09-13
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/ram-efficient-pytorch-fsdp
- Record: https://forck.live/items/1998-fine-tuning-llama-2-70b-using-pytorch-fsdp
- Subject: Platform
- Models affected: Llama 2 70B, meta-llama/Llama-2-70b-chat-hf

Best practices for fine-tuning Llama 2 70B using PyTorch FSDP, addressing challenges such as CPU RAM usage during model loading, checkpoint saving, and training speed optimization.

## Evidence

Verbatim from https://huggingface.co/blog/ram-efficient-pytorch-fsdp:

> In this blog post, we will look at how to fine-tune Llama 2 70B using PyTorch FSDP and related best practices.

---

Record: https://forck.live/items/1998-fine-tuning-llama-2-70b-using-pytorch-fsdp
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md