From the source
Apple researchers, with Harvard co-authors, published a study showing that the gating mechanism in State Space Models explains both their strengths and failures in in-context learning and retrieval.
- SSMs offer constant memory and linear compute, but they lag behind Transformers on tasks that need in-context learning and precise retrieval, which has slowed their adoption for large-scale language modeling.
- Through theory and experiments, the authors show gating causes SSMs to first learn an in-weights memorization solution while delaying, or even preventing, convergence to a correct in-context learning solution.
- The paper says this happens even when the architecture and its memory capacity impose no fundamental limitation.
- On the other hand, the authors find gating is often beneficial for improving generalization to long sequence lengths.
- The work was done while the lead author was at Apple, with Harvard University as an affiliation.
