From the source
LG AI Research is presenting ATAD, an agent-centered dynamic assessment protocol for evaluating LLM reasoning, at ICLR2026 as a response to the limits of static benchmarks.
- The company argues that widely used static benchmarks suffer from performance saturation, data contamination, format overfitting and short lifespans, since the latest LLMs have already surpassed human-level performance on them.
- ATAD introduces a three-party interaction structure it calls Teacher-Orchestrator-Student, in which the Orchestrator alone handles validation of generated problems.
- That design is meant to prevent the loss of clarity and rise in ambiguity that can occur as difficulty increases, according to the post.
- The Orchestrator also identifies where the student agent fails, a process the researchers call Failure-Driven Sample Finalization, which is used to build the benchmark and expose the model's actual blind spots.
- The post singles out Text Anomaly Detection as a field where researchers face a trade-off between clarity and difficulty, because language lacks the fixed answers of mathematics or science: harder problems tend toward ambiguous answers, while clearer ones become too easy to discriminate between models.
