From the source
Baseten's Base Labs ran toy experiments to test when a value function, or critic, is worth the extra compute it costs to train in reinforcement learning.
- The lab frames the question as whether a critic removes more gradient noise than it costs in opportunity cost to more rollouts, and says that is not usually the case.
- It reports that most papers swapping the group mean for a PPO critic or a proxy perform equally at equal steps, and that adding a critic on top of the group helps even less than hoped.
- A "perfect critic" removes at most 40-60% of the advantage variance on two single-turn tasks, according to the lab.
- The only area where a critic may help is a small value head, and even then the benefit may be marginal.
- The post derives explained-variance ceilings for a prefix-only residual critic and a baseline-aware predictor, showing the latter gains an extra 1/G from cancelling group-baseline noise.




