Sakana AI· Sakana models

An Evolved Universal Transformer Memory

This is the second in a series of blog posts presenting the results of Sakana AI’s research projects that were supported by the Japanese Ministry of Economy, Trade and Industry’s GENIAC supercomputing grant . Summary At Sakana AI, we draw inspiration from nature to extend the capabilities of foundation models. This was evidenced in our recent work on improving performance with evolutionary model merging , discovering qualitative and diverse agentic skills , and uncovering new ways for LLMs to contribute to AI research . Following on in this line of research, our latest breakthrough, detailed in our paper, An Evolved Universal Transformer Memory , introduces a revolutionary memory system for transformers, inspired by how human memory selectively retains and prunes information. Our learned memories not only boost both performance and efficiency of existing, pre-trained transformers, but are also universally transferable across different foundation models, even beyond language, without any re-training! Memory is a crucial component of cognition , allowing humans to selectively store and extract important information from the ceaseless noise dominating our lives. …

Announced

09 Dec 2024 15:00 UTC

Detected

28 Aug 2026 21:48 UTC (+627d after)