From the source
Perplexity researchers propose JOLT, a method that trains one model in two roles so a stronger teacher gives guidance matched to the student's current ability.
- Outcome-reward reinforcement learning gives a sparse signal that cannot tell which intermediate actions helped or hurt, which limits sample efficiency on long-horizon agentic tasks where successes are rare.
- On-Policy Distillation adds denser token-by-token feedback from a stronger teacher, but a model good at a task is not necessarily good at teaching it, since its feedback may sit beyond the student's reach.
- Building on On-Policy Self-Distillation, JOLT uses a single model as a teacher with privileged information such as verified solutions and as a student without it, training both jointly.
- The student follows the standard distillation objective, while the teacher's updates push its feedback to move the student in the same direction as RL on the student's own reward.
- The researchers call a teacher reward-consistent when the distillation update it induces matches the student's expected-reward policy gradient up to positive scaling, and define a teacher that satisfies this condition.