From the source
- Airbnb runs coding agents for thousands of engineers through an internal LLM gateway on
Amazon Bedrock, and uses cost per task to pick which models those agents default to.
- The company says eight new models arrived in about five weeks, and by mid-October roughly two thirds of the requests its engineers sent through coding agents went to those new models.
- Because each release can shift output tokens, cost, latency and capacity at once, Airbnb runs evals on its own engineering tasks before a rollout to measure each model's score and cost per task, defined as the list-price cost of all tokens an agent uses to finish one task.
- After a rollout, CloudWatch metrics in the AWS/Bedrock namespace show what changed in production, including InvocationLatency, TimeToFirstToken, InvocationThrottles and token counts for uncached input, cache reads, cache writes and output.
- Airbnb uses the eval results to set each default, the model and reasoning effort its coding tools use unless an engineer picks another, and each effort ceiling, the highest effort allowed for a model.
- The gateway records the model, client, team and user for each request, and Airbnb says it uses those records only in aggregate, for cost and capacity planning.






