# Baseten — When does distillation help reinforcement learning?

- Company: Baseten (baseten.co)
- Announced: not stated by the source (first seen 2026-11-06T18:26:27.263522+00:00)
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.baseten.co/blog/when-does-distillation-help-reinforcement-learning
- Record: https://forck.live/items/32766-when-does-distillation-help-reinforcement-learning
- Subject: Baseten
- Models affected: Qwen3, GPT-5.6-Sol

Baseten studied whether supervised fine-tuning on teacher solutions improves later reinforcement learning, comparing Qwen3 models at 0.6B, 1.7B, 4B and 8B parameters under GRPO across sixteen reasoning tasks. Distillation refers to supervised training on teacher responses, which the company describes as often used to warm-start a model so that reinforcement learning has a non-zero starting reward. Students were fine-tuned on teacher responses from GPT-5.6-Sol, with the main experiment using 2M response tokens before RL and three RL repetitions per starting condition at each size. SFT improved accuracy before RL at every size, by 31.2% at 0.6B against 15.3% at 8B, but the benefit after RL fell as students grew. After 80 million response tokens of RL, SFT + RL led RL by 11.3% at 0.6B, 13.1% at 1.7B and 2.1% at 4B, while at 8B RL alone beat SFT + RL by 6.4%. Baseten concludes that higher accuracy before RL does not necessarily yield higher accuracy after it, and that distillation loses value as the student gets larger.

## Evidence

Verbatim from https://www.baseten.co/blog/when-does-distillation-help-reinforcement-learning:

> We are very interested in the interplay between distillation and reinforcement learning. Evidently, much of the advancement in capabilities in LLMs as of late has been due to scaling up RL, both in compute and number of environments. Distillation is often seen as a way to warm-start a model in preparation for RL. This is because in RL we need a non-zero starting reward with a model in order to be able to hill-climb, and if we can go from zero to non-zero by training on a few traces from a better model, then we unlock the value of the whole RL environment.
> 
> Distillation is thus seen as a crutch used by open-source model labs as a way to stay within a stone’s throw of the closed labs with stronger models. However, it is unclear how distilling from a stronger model can affect the ceiling (and floor) with respect to the capabilities of the model you are training. Here lie some interesting questions:
> 
> How much does being on-policy in distillation matter? For instance, if I am training on a completely different model family, will this later fry my model? Will it somehow decrease the entropy? How much RL do I need to repair this?
> 
> Suppose I have a fixed compute budget.

---

Record: https://forck.live/items/32766-when-does-distillation-help-reinforcement-learning
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
