From the source
Apple researchers propose a round-trip protocol to measure how much tree-structured compositional content survives when language models serialize expressions into natural language.
- Evaluating 16 models, they find the channel is lossy and asymmetric, with at least 73.6% of failures originating at generation, and that fine-tuning on ∼3600 examples lifts open-weight models above an untrained frontier model.