From the source
The problem of request queuing in LLM inference engines and presents a solution using fair scheduling with separate queues per user and model, and dynamic rate limiting to keep the backend queue short.
Efficient Request Queueing – Optimizing LLM Performance
From the source
Serving LLMs to many applications and users in parallel is challenging because they compete for limited GPU resources. This article is the first in a series on LLM performance, based on our experience with serving self-hosted LLMs at TNG Technology Consulting GmbH. In the first part, we focus on the impact of queuing and discuss different scheduling strategies.
huggingface.co
