# Apple — On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

- Company: Apple (apple.com)
- Announced: 2026-11-10
- Category: research-paper
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://machinelearning.apple.com/research/gating-memorization-in-context
- Record: https://forck.live/items/34761-on-the-importance-of-gating-memorization-vs-in-context-learning-in-state-space
- Subject: Machine Learning Research

Apple researchers, with Harvard co-authors, published a study showing that the gating mechanism in State Space Models explains both their strengths and failures in in-context learning and retrieval. SSMs offer constant memory and linear compute, but they lag behind Transformers on tasks that need in-context learning and precise retrieval, which has slowed their adoption for large-scale language modeling. Through theory and experiments, the authors show gating causes SSMs to first learn an in-weights memorization solution while delaying, or even preventing, convergence to a correct in-context learning solution. The paper says this happens even when the architecture and its memory capacity impose no fundamental limitation. On the other hand, the authors find gating is often beneficial for improving generalization to long sequence lengths. The work was done while the lead author was at Apple, with Harvard University as an affiliation.

## Evidence

Verbatim from https://machinelearning.apple.com/research/gating-memorization-in-context:

> Authors William L. Tong†**, Aryo Lotfi, Emmanuel Abbe, Kostas Vaggelakos, Vishnu Banna, Etai Littwin, Josh Susskind, Cengiz Pehlevan†, Eran Malach
> 
> State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent networks. Specifically, we show through theory and experiments that this gating mechanism causes SSMs to first learn an in-weights “memorization” solution, while delaying, or even preventing, convergence to a correct in-context learning solution. Importantly, this happens even in cases where there are no fundamental limitations due to the architecture or its memory capacity.

---

Record: https://forck.live/items/34761-on-the-importance-of-gating-memorization-vs-in-context-learning-in-state-space
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
