From the source
Lead story
Ask your AI
Top stories
Models & availability
Latest
The Agent Said It Was Done. The Database Disagreed.
ThinkingBox grades agents on database state, not tool calls, across 507 workflows.
Microsoft and Hugging Face introduced ThinkingBox, a benchmark that evaluates AI agents on the terminal backend state and side effects they leave behind rather than on tool call correctness.
Across 507 stateful business workflows run 20 times each against various LLMs, 79,853 of 121,680 trials failed executable checks, with 67.24% of failures appearing clean despite wrong field values (77.61%), unintended extra effects (43.30%), or missing required effects (25.36%).
The benchmark reveals large consistency gaps: Claude Opus 5.5 leads overall at 67.16% pass@1, while Kimi-K3 solves 93.89% of tasks at least once but only 13.41% in all 20 attempts.
From the source
The gap is substantial. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks nevertheless found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%.
huggingface.co