From the source
From the source
Sakana AI introduces TAID, a novel knowledge distillation method for transferring knowledge from large language models to small ones.
They release two models: TAID-LLM-1.5B (English) and TinySwallow-1.5B (Japanese), with TinySwallow-1.5B achieving state-of-the-art performance among similar-sized models.
The research is accepted as a Spotlight paper at ICLR 2025.
From the source
Key highlights of this release: We introduce TAID (Temporally Adaptive Interpolated Distillation), a novel knowledge distillation method that enables efficient transfer of knowledge from large language models to smaller ones. In collaboration with the Institute of Science Tokyo, we have developed TinySwallow-1.5B, a compact Japanese language model that enables offline execution on edge devices like smartphones. Our research has been accepted as a Spotlight paper at ICLR, a top international conference in machine learning.
sakana.ai
