TutorMoments: Do AI tutors know when to help and when to hold back?

Ai2 introduced TutorMoments, a framework and dataset for evaluating whether large language models can appropriately decide when to scaffold or push for rigor in math tutoring.

Based on 462 real tutoring transcripts with over 1,500 teacher-annotated key moments, the evaluation found that models tend to over-help and rarely push students to deeper thinking, even when the trade-off is spelled out in the prompt.

Ai2 released the de-identified transcripts, replay pipeline code, and model replays for reproducibility.