LG AI Research presented a study applying mechanistic interpretability to graph transformers, specifically analyzing how TokenGT processes graph data.

The work found that models trained on degree calculation, cycle detection, and shortest-path distance tasks all begin with a shared local-structure computation, such as encoding node degree through ID-matching attention in the first transformer layer.