From the source
From the source
Sakana AI introduces Neural Attention Memory Models (NAMMs), a learned memory system for transformers that improves performance and efficiency by selectively retaining or forgetting tokens, and is universally transferable across different foundation models and modalities without retraining.
The paper also releases a new Japanese long-context benchmark called ChouBun.
From the source
our latest breakthrough, detailed in our paper, An Evolved Universal Transformer Memory , introduces a revolutionary memory system for transformers, inspired by how human memory selectively retains and prunes information. Our learned memories not only boost both performance and efficiency of existing, pre-trained transformers, but are also universally transferable across different foundation models, even beyond language, without any re-training!
sakana.ai
