# Baseten — The bias of on-policy self-distillation

- Company: Baseten (baseten.co)
- Announced: 2026-11-06
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.baseten.co/blog/the-bias-of-on-policy-self-distillation
- Record: https://forck.live/items/32898-the-bias-of-on-policy-self-distillation
- Subject: Baseten
- Models affected: Qwen3-8B, Olmo-3-7B-Instruct, Qwen3-1.7B

Baseten's Base Labs argues that on-policy self-distillation (OPSD) is not worth further research effort, saying the method trades a noisy but well-aimed reward for a clean but biased one. OPSD has a student model learn from a teacher that is the same model given extra context, such as a ground-truth answer, so the teacher never has to check, backtrack or hedge. The authors say that dense per-token supervision mostly adds bias, and that this bias is worse than having more variance, especially in heavily trained models. They point to results where OPSD gains faded: with thinking enabled, average accuracy fell for all five thinking models compared with no thinking, and on held-out coding problems SDPO lost 2.5 points on Qwen3-8B and 9.9 on Olmo-3-7B-Instruct, while GRPO gained. One study cited found OPSD guides the student toward the teacher model averaged over hints rather than toward a teacher conditioned on success. The authors still allow that OPSD may remain useful in certain contexts.

## Evidence

Verbatim from https://www.baseten.co/blog/the-bias-of-on-policy-self-distillation:

> We will start this research update by saying that on-policy self-distillation (OPSD) may remain useful in certain contexts, but we do not believe spending further research efforts on it (or related variants) will be worth it. Reading through the lens of the literature and our own work, we explain why.
> 
> What is OPSD?
> 
> In distillation overall, the goal is to get a student model q q to learn behavior from a teacher model p p . Generally we denote such types of post-training methods distillation (as our goal is to distill knowledge from a different “teacher” model). One common form of off-policy distillation samples from the teacher model and trains the student to match the teacher’s distributions along those rollouts:
> 
> However, this can create a mismatch between the rollouts the student sees during training and those it generates at inference. For example, if the student tends to make an early mistake the teacher rarely makes, this form of distillation will struggle to fix the mistake.

---

Record: https://forck.live/items/32898-the-bias-of-on-policy-self-distillation
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
