Llamba: scaling distilled recurrent models for efficient language processing

Cartesia released a technical report introducing MOHAWK, a multi-stage architecture distillation method that converts Transformer models into efficient Mamba-2 variants.

• The company released model weights for the Llamba family (Llamba-1B, Llamba-3B, Llamba-8B), distilled from corresponding Llama-3.X models, and reported throughput improvements up to 12× for Llamba-8B versus Llama-3.1-8B.
• The work targets on-device and high-throughput inference scenarios.