From the source
Elorian introduced DecoyCount, a benchmark of 1,660 real-world images built to separate perception from counting by recording the specific wrong count each scene invites.
- Each image carries a pre-registered decoy, a wrong answer the scene suggests, logged in advance with the mechanism that invites it, so a model that returns the decoy reveals which interpretation it chose.
- The benchmark spans 2,203 decoys across seven mechanisms, with three prompts per image: a neutral question, a cued version naming the intended rule, and a multi-hop scenario that implies the rule without stating it.
- Items were kept only if Claude Opus 5, GPT 5.6 Sol and Gemini 3.6 Flash all missed the neutral answer and at least one returned a registered decoy exactly; blind review retained 1,660 of 2,100 candidates, corrected 121 counts and replaced 79 images.
- Elorian reports that a one-sentence cue moves accuracy by 11.8 to 22.1 points, and that changing "are there" to "are visible" shifts Gemini 3.8 Flash by 6.69 points, repairing 246 answers while breaking 135.
- The best unaided score was 43.2% exact match, by GPT 6 Astra, and models answered the decoy five of six times more often than the truth.
