Accelerate StarCoder with 🤗 Optimum Intel on Xeon: Q8/Q4 and Speculative Decoding

Summary by forck

Hugging Face and Intel present optimizations for StarCoder-15B on 4th gen Xeon, achieving over 7x inference acceleration through 8-bit and 4-bit quantization combined with assisted generation (speculative decoding).

The work uses SmoothQuant for INT8 quantization and demonstrates significant speedup without accuracy loss on the HumanEval benchmark.