# Apple — Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps

- Company: Apple (apple.com)
- Announced: 2026-11-02
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/planning-grounding-ui-navigation
- Record: https://forck.live/items/30541-online-dense-reinforcement-learning-with-planning-grounding-agent-for-ui
- Subject: Machine Learning Research

Apple researchers propose an online reinforcement learning framework that trains UI navigation agents directly on live iOS applications, arguing that frontier vision-language models are too costly for everyday use while smaller SFT or offline RL models memorize static trajectories and fail to recover from errors. The framework splits the agent into a Planning agent and a UI Grounding model, and uses an LLM-as-a-Judge to supply dense step-level feedback alongside a terminal reward for overall task completion. The agent is optimized with Proximal Policy Optimization and step-level advantage estimation, plus GiGPPO, a neighbor-conditioned value estimation method the authors say improves critic stability by drawing reward signals from visually similar states across concurrent trajectories. Supporting this at scale required infrastructure that orchestrates hundreds of concurrent virtual iOS devices. On UINavBench, iOSWorld and AndroidWorld, the online RL agents substantially outperform SFT and terminal-only RL baselines and converge faster, with the 8B agents showing nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android and outperforming Gemini-3.5-Flash while tying Gemini-3.1-Pro on real iOS applications.

## Evidence

Verbatim from https://machinelearning.apple.com/research/planning-grounding-ui-navigation:

> Authors Peitong Duan*, Seyed Khashaiar Gatmiry*, Jeff Nichols, Emmanuel Abbe, Samy Bengio
> 
> While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation.

---

Record: https://forck.live/items/30541-online-dense-reinforcement-learning-with-planning-grounding-agent-for-ui
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md