# Apple — Weighted State Souping for Efficient Long-Context Retrieval

- Company: Apple (apple.com)
- Announced: 2026-11-10
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/weighted-state-souping
- Record: https://forck.live/items/34762-weighted-state-souping-for-efficient-long-context-retrieval
- Subject: Machine Learning Research

Apple researchers, with Harvard co-authors, propose weighted state souping, a method that learns query-dependent weights for composing cached state space model encodings into a context for downstream queries. State souping and vanilla SSMs deteriorate on long, crowded contexts, which limits their use in long-horizon domains, the paper says. The resulting models improve long-context performance substantially across single- and multi-hop document question answering, needle-in-a-haystack retrieval, variable tracking, and tool use. A sparse variant yields near-constant encoding cost with respect to context length while preserving, and sometimes improving, performance. The paper also offers theoretical arguments for why souping works and when it may fail, and discusses informally the advantages of weighted souping. The work was accepted at the Workshop on Personalized, Aligned, Long-term Memory for AI Systems (PALM) at NeurIPS 2026.

## Evidence

Verbatim from https://machinelearning.apple.com/research/weighted-state-souping:

> Authors William L. Tong†**, Aryo Lotfi, Emmanuel Abbe, Josh Susskind, Cengiz Pehlevan†, Eran Malach
> 
> This paper was accepted at the Workshop on Personalized, Aligned, Long-term Memory for AI Systems (PALM) at NeurIPS 2026.
> 
> State space models (SSMs) combine fast inference and fixed state size with competitive language-modeling performance. Recently, state souping has been proposed as a procedure whereby an SSM independently encodes chunks of tokens like documents, tool definitions, or memory records. The resulting state encodings may be cached and recombined to cheaply construct a context for downstream queries. However, both state souping and vanilla SSMs deteriorate on long, crowded contexts, limiting their utility for long-horizon domains. To address this issue, we propose weighted state souping, a simple method that learns query-dependent weights for composing cached states. The resulting models improve long-context performance substantially across both single- and multi-hop document question answering, needle-in-a-haystack retrieval, variable tracking, and tool use.

---

Record: https://forck.live/items/34762-weighted-state-souping-for-efficient-long-context-retrieval
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
