From the source
Inference spend is rising as agents are deployed more widely, take on more complex tasks, and run longer autonomously.
Enterprises are starting to put model routing into production to keep that spend in check.
Among models capable of serious agent work, prices span more than fiftyfold, and a single task can take dozens or hundreds of model calls as the agent reads files, runs tools, and revises its approach.
In conversations with leaders at systemically important banks, cybersecurity companies, and major technology firms, the same architecture question keeps coming up.
A gateway receives model requests from applications and forwards them to model providers.
Every model call already passes through one, so why put model routing in the harness?
At the gateway, an agent's model call looks like any other completion request: a prompt goes out, tokens come back, and the connection closes.
Inside the harness, that call is one step in a computation that can run for hours.
…
