From the source
Apple researchers propose an online reinforcement learning framework that trains UI navigation agents directly on live iOS applications, arguing that frontier vision-language models are too costly for everyday use while smaller SFT or offline RL models memorize static trajectories and fail to recover from errors.
- The framework splits the agent into a Planning agent and a UI Grounding model, and uses an LLM-as-a-Judge to supply dense step-level feedback alongside a terminal reward for overall task completion.
- The agent is optimized with Proximal Policy Optimization and step-level advantage estimation, plus GiGPPO, a neighbor-conditioned value estimation method the authors say improves critic stability by drawing reward signals from visually similar states across concurrent trajectories.
- Supporting this at scale required infrastructure that orchestrates hundreds of concurrent virtual iOS devices.
- On UINavBench, iOSWorld and AndroidWorld, the online RL agents substantially outperform SFT and terminal-only RL baselines and converge faster, with the 8B agents showing nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android and outperforming Gemini-3.5-Flash while tying Gemini-3.1-Pro on real iOS applications.
