From the source
IBM Research's ALTK-Evolve learns a playbook of behavioral rules that lifts a basic GPT-4.1 customer-service agent to 67.5% Pass^4 on τ²-bench retail, above every retail entry in its October 12, 2026 leaderboard snapshot.
- The method analyzes the agent's attempts on training tasks and distills successes and failures into short rules, such as how to authenticate a customer or handle a return, which are appended to the system prompt.
- The same model writes the guidelines and runs the agent, so no weight updates or human annotation are needed.
- The team tested the approach on τ²-bench, a Sierra benchmark where an LLM-simulated customer, domain tools and a written policy constrain each conversation, and a task counts as solved only when the final database state matches the expected outcome.
- On held-out tasks, mean accuracy rose 4.4 points in retail and 7.5 points in airline, while the stricter Pass^4 rose 10 points in each domain.
- The team compared no guidelines against a high-confidence core and a lossless set, and reports that the gains come from better-informed calls rather than longer conversations.
