From the source
Perplexity's guide explains how to fine-tune a large language model and when to reach for prompting or retrieval-augmented generation instead.
- Fine-tuning refines a pre-trained model on a small, task-specific dataset of example input/output pairs, moving its weights toward the behavior you want, and parameter-efficient methods like LoRA make it cheap enough to run on a single GPU.
- The guide advises working up a ladder of prompting, then RAG, then fine-tuning, and says prompting is the fastest and cheapest lever, though longer prompts slow work and cost more.
- It draws the line at knowledge versus behavior: RAG changes what the model knows, while fine-tuning changes how it behaves, so RAG suits facts that change often or live in your own documents.
- Fine-tuning carries risks the guide lists, including catastrophic forgetting, overfitting on repetitive data, staleness of knowledge baked into weights, and obsolescence when a stronger model ships.
- Production systems often pair fine-tuned models with RAG, using one for consistent behavior and the other for current facts.