From the source
Microsoft and
Hugging Face introduced ThinkingBox, a benchmark that grades AI agents on the database records they leave behind rather than the tool calls or text they generate, and runs every task 20 times to measure consistency.
- Across 507 stateful business workflows, 79,853 of 121,680 valid trials failed executable checks, and 67.24% of those failures still terminated cleanly with no tool error.
- Only three models retained most of their single-attempt score across 20 repeats: GPT-6 Astra kept 78%, while Claude Opus 5.5 and Claude Opus 5 each kept 71%.
- Kimi-K3 solved 93.89% of tasks at least once but only 13.41% on every attempt, showing breadth does not guarantee reliability.






