From the source
Google Research is proposing a sleep framework that gives large language models a periodic cycle of active interaction and offline consolidation, so they can absorb new facts without losing old ones.
- The problem it answers is that deployed models stay static behind a knowledge cutoff, while continual parameter updates often trigger catastrophic forgetting and in-context learning forgets everything once the session ends.
- The framework alternates a wake phase, where the model stores temporary information in short-term memory, with a sleep phase that redirects computation to memory consolidation and self-improvement.
- During consolidation, the model scales its capacity by adding low-rank experts into stable lower-frequency memory blocks through a Mixture of Experts and Hope architecture, then resets the high-frequency temporary parameters.
- A dreaming stage generates synthetic samples, randomly activates non-standard experts to mix domains, filters the samples by gradient-based selection, and refines the model with supervised fine-tuning.
- Google Research says the approach lets models retain long-context knowledge up to 10M tokens and improve complex reasoning, all within a fixed architecture and human-defined training rules.