From the source
Ai2 released XYBench, a benchmark of 8,115 real-world questions that tests whether AI assistants recognize when a user's request rests on a misconception.
- The benchmark targets the XY problem, where someone wants to accomplish X but asks for help with an assumed approach Y, such as asking how to thicken frosting that slides off cupcakes that are simply too warm.
- Ai2 developed XYBench with researchers at Carnegie Mellon,
NVIDIA, and Johns Hopkins, drawing questions from Stack Overflow, Stack Exchange, curated examples, and WikiHow guides.
- XYBench measures three things: whether a response solves the stated and intended requests, which one it emphasizes, and whether it steers the user toward the underlying goal.
- Claude Sonnet 4.5 serves as the automated judge, and two human reviewers checked its judgments on 500 responses, with agreement ranging from 97.5–100% on whether a solution appeared to 74.2–79.8% on pragmatic emphasis.
- Ai2 says all five evaluated models were more likely to include a solution to the stated request than to the underlying goal, and on Stack Overflow they focused primarily on the underlying goal in only 35–41% of responses.






