From the source
Apple researchers propose a two-stage recipe for distilling a pretrained Transformer into a Mamba-like state space model without any Attention blocks.
- The work addresses a known problem: a naïve distillation from Transformers to Mamba fails to preserve the teacher's performance, a limitation usually worked around with hybrid Attention and SSM designs.
- Their approach first distills knowledge from a Transformer into a linearized version of Attention using an adaptation of the kernel trick, then distills that linearized model into an adapted Mamba.
- The distilled Mamba retains most of the original Pythia-1B Transformer's performance, reaching a perplexity of 14.01 against the teacher's 13.86.
- The authors report this largely carries over to downstream tasks, though gaps appear on tasks that rely more on in-context recall.
- They ran ablations at 1B scale with 10B tokens, scaling analysis across model sizes and distillation token counts, and a sensitivity analysis on how tokens are allocated between the two stages.