From the source
Hugging Face introduces assisted generation, a new decoding method that reduces latency for autoregressive text generation by up to 10x on commodity hardware.
Assisted Generation: a new direction toward low-latency text generation
Research paper
From the source
by using our new method, assisted generation, you can reduce latency up to 10x in commodity hardware!
huggingface.co
