From the source
Optimization of LLM performance, specifically how long prompts block other requests, and describes a vLLM update that implements parallel prefills with a limit on long prompt requests to reduce time-to-first-token for short prompts.
How Long Prompts Block Other Requests - Optimizing LLM Performance
Research paper
From the source
In one of the latest vLLM updates, an improved strategy has been implemented: it allows for parallel prefills of different requests but with a limit to the number of concurrently processed long prompt requests.
huggingface.co
