Faster Assisted Generation with Dynamic Speculation
Hugging Face and Intel labs announce dynamic speculative decoding, a new method for accelerating text generation up to 2.7x, which becomes the default in Transformers version 4.45.0. The method adjusts the number of draft tokens based on confidence, outperforming the heuristic approach.
