From the source
From the source
From the source

From the source
A traveler asks a support agent to switch a $1,450 hotel booking from a suite to a deluxe room. Company policy states that the change requires a manager’s approval first. In 15 of 20 attempts, the open-weight model Qwen3.8-27B made the change anyway.
It updated the room, refunded money, marked the support ticket solved, and told the traveler it was done. While that may sound like good customer service, it’s a violation of company policy. In our first post , the failure ran the other way.
An agent told a traveler their quiet-room request had been added, while the booking’s special requests field stayed empty. We built ThinkingBox to catch both kinds of failure. It runs an agent in an isolated, stateful environment, lets a simulated user answer its questions, and then checks the records the agent left behind rather than simply trusting what it said.
Since then, we’ve expanded the evaluation to 18 models and published our method and results in a paper . This post covers three things we learned: Average success and repeatable success rank models differently The cheapest successful attempt is not the cheapest dependable task …
commandline.microsoft.com