# Apple — Attention to Mamba: A Recipe for Cross-Architecture Distillation

- Company: Apple (apple.com)
- Announced: 2026-11-03
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/attention-mamba-distillation
- Record: https://forck.live/items/34244-attention-to-mamba-a-recipe-for-cross-architecture-distillation
- Subject: Machine Learning Research
- Models affected: Mamba, Pythia-1B

Apple researchers propose a two-stage recipe for distilling a pretrained Transformer into a Mamba-like state space model without any Attention blocks. The work addresses a known problem: a naïve distillation from Transformers to Mamba fails to preserve the teacher's performance, a limitation usually worked around with hybrid Attention and SSM designs. Their approach first distills knowledge from a Transformer into a linearized version of Attention using an adaptation of the kernel trick, then distills that linearized model into an adapted Mamba. The distilled Mamba retains most of the original Pythia-1B Transformer's performance, reaching a perplexity of 14.01 against the teacher's 13.86. The authors report this largely carries over to downstream tasks, though gaps appear on tasks that rely more on in-context recall. They ran ablations at 1B scale with 10B tokens, scaling analysis across model sizes and distillation token counts, and a sensitivity analysis on how tokens are allocated between the two stages.

## Evidence

Verbatim from https://machinelearning.apple.com/research/attention-mamba-distillation:

> Authors Abhinav Moudgil†**, Ningyuan Huang‡**, Eeshan Gunesh Dhekane, Pau Rodríguez Lopez, Luca Zappella, Federico Danieli
> 
> State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compared to their Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train Transformers, and many pretrained Transformer models are readily available. To facilitate the adoption of SSMs while leveraging existing pretrained Transformers, we aim to identify an effective recipe to distill an Attention-based model into a Mamba-like architecture. In prior work on cross-architecture distillation, however, it has been shown that a naïve distillation procedure from Transformers to Mamba fails to preserve the original teacher performance, a limitation often overcome with hybrid solutions combining Attention and SSM blocks. The key argument from our work is that, by equipping Mamba with a principled initialization, we can recover an overall better recipe for cross-architectural distillation.

---

Record: https://forck.live/items/34244-attention-to-mamba-a-recipe-for-cross-architecture-distillation
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
