From the source
Apple researchers, with Harvard co-authors, propose weighted state souping, a method that learns query-dependent weights for composing cached state space model encodings into a context for downstream queries.
- State souping and vanilla SSMs deteriorate on long, crowded contexts, which limits their use in long-horizon domains, the paper says.
- The resulting models improve long-context performance substantially across single- and multi-hop document question answering, needle-in-a-haystack retrieval, variable tracking, and tool use.
- A sparse variant yields near-constant encoding cost with respect to context length while preserving, and sometimes improving, performance.
- The paper also offers theoretical arguments for why souping works and when it may fail, and discusses informally the advantages of weighted souping.
- The work was accepted at the Workshop on Personalized, Aligned, Long-term Memory for AI Systems (PALM) at NeurIPS 2026.
