Making Knowledge Distillation Cheap Enough to Run at Scale

MultiverseComputingCAI researchers publish a paper on efficient knowledge distillation, introducing offline top-K logit caching and a fused chunked KL loss to reduce VRAM usage, enabling long-context training on a single GPU.

From the source
Our latest paper, Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss, tackles this with two systems changes: caching the teacher's top-K logits once so the teacher never has to sit in memory alongside the student, and a new, memory-efficient KL-divergence loss that avoids ever materializing the full vocabulary-size × sequence-length matrix, cutting VRAM use far below what the default implementations in libraries like PyTorch or NVIDIA Megatron-Bridge achieve.
huggingface.co

Visit source ↗huggingface.co