# Baseten — Understanding the compute efficiency of value functions in RL

- Company: Baseten (baseten.co)
- Announced: not stated by the source (first seen 2026-11-06T18:26:26.577197+00:00)
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.baseten.co/blog/understanding-the-compute-efficiency-of-value-functions-in-rl
- Record: https://forck.live/items/32762-understanding-the-compute-efficiency-of-value-functions-in-rl
- Subject: Baseten

Baseten's Base Labs ran toy experiments to test when a value function, or critic, is worth the extra compute it costs to train in reinforcement learning. The lab frames the question as whether a critic removes more gradient noise than it costs in opportunity cost to more rollouts, and says that is not usually the case. It reports that most papers swapping the group mean for a PPO critic or a proxy perform equally at equal steps, and that adding a critic on top of the group helps even less than hoped. A "perfect critic" removes at most 40-60% of the advantage variance on two single-turn tasks, according to the lab. The only area where a critic may help is a small value head, and even then the benefit may be marginal. The post derives explained-variance ceilings for a prefix-only residual critic and a baseline-aware predictor, showing the latter gains an extra 1/G from cancelling group-baseline noise.

## Evidence

Verbatim from https://www.baseten.co/blog/understanding-the-compute-efficiency-of-value-functions-in-rl:

> The starting of a research lab often involves revisiting the eminent literature and understanding why something worked (or why it did not). A theme of this week at Base Labs has been thoroughly understanding and internalizing the historical canon of the post-training field. One canonical example (that is very front of mind) is the introduction of GRPO [ 1 ] and correspondingly, the decline of PPO.
> 
> Before GRPO was introduced, doing RL on a language model meant doing PPO [ 2 ]. In brief, PPO involves some policy and a value model (a ‘critic’ of this policy) that attempts to predict how well the answer will score at each token [ 3 ]. The creation of GRPO meant removing the critic in place of scoring each answer against the mean reward of a group of completions sampled for the same prompt. The theoretical argument here was that instantiating and training a second network is basically the same number of parameters as the policy, and perhaps more importantly, must try to make accurate per-token predictions from a single scalar reward given at end.

---

Record: https://forck.live/items/32762-understanding-the-compute-efficiency-of-value-functions-in-rl
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
