DiffusionBlocks: Training Neural Networks One Block at a Time
Research paper
Sakana AI introduces DiffusionBlocks, a method to train neural networks one block at a time by reinterpreting the forward pass as a diffusion process, reducing memory requirements while matching end-to-end performance across ViT, DiT, masked diffusion, autoregressive transformers, and recurrent-depth transformers.
The paper was accepted at ICLR 2026.