# Ai2 — XYBench: Helping with the problem behind the question

- Company: Ai2 (allenai.org)
- Announced: 2026-11-13
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://allenai.org/blog/xybench
- Record: https://forck.live/items/35973-xybench-helping-with-the-problem-behind-the-question
- Subject: Olmo / Molmo / Asta
- Models affected: Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, DeepSeek-V4 Pro, Claude Sonnet 4.5

Ai2 released XYBench, a benchmark of 8,115 real-world questions that tests whether AI assistants recognize when a user's request rests on a misconception. The benchmark targets the XY problem, where someone wants to accomplish X but asks for help with an assumed approach Y, such as asking how to thicken frosting that slides off cupcakes that are simply too warm. Ai2 developed XYBench with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins, drawing questions from Stack Overflow, Stack Exchange, curated examples, and WikiHow guides. XYBench measures three things: whether a response solves the stated and intended requests, which one it emphasizes, and whether it steers the user toward the underlying goal. Claude Sonnet 4.5 serves as the automated judge, and two human reviewers checked its judgments on 500 responses, with agreement ranging from 97.5–100% on whether a solution appeared to 74.2–79.8% on pragmatic emphasis. Ai2 says all five evaluated models were more likely to include a solution to the stated request than to the underlying goal, and on Stack Overflow they focused primarily on the underlying goal in only 35–41% of responses.

## Evidence

Verbatim from https://allenai.org/blog/xybench:

> Truly useful AI assistants need to be able to help users who ask misguided questions—not just users who ask the right question with the right amount of detail. We’re releasing XYBench , a benchmark of 8K+ questions that tests how AI assistants respond to requests based on misconceptions, such as asking how to thicken frosting that keeps sliding off cupcakes when the real problem is that the cupcakes are still too warm to frost. XYBench won the Best Resource Paper Award at EMNLP 2026.
> 
> A mismatch between the approach someone asks about and the goal they want to achieve is known as the XY problem : a person wants to accomplish X, assumes a particular approach Y will work, and asks for help with Y. In the cupcake example, the person is so committed to making the frosting stick that they fail to consider the warmth of the cupcake as the true problem to be addressed.
> 
> Developed with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins, XYBench exposes a critical gap in LLMs' ability to troubleshoot problems: recognizing when a user's request is based on an implicit misconception and appropriately redirecting them toward a solution that addresses their actual goal.

---

Record: https://forck.live/items/35973-xybench-helping-with-the-problem-behind-the-question
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
