From the source
Vercel's guide explains how to build evals for agents, which it describes as users of software that do work on people's behalf.
- It argues that checking whether an agent completed a task is harder than it sounds, because an agent can place a "successful" order for a hot latte when the user asked for an iced one and the logs still look fine.
- The guide groups agentic software tests into a pyramid by how many LLM calls they need: deterministic tests with none, one-turn evals with one, and end-to-end evals with two or more across a whole task.
- It says that split is a good proxy for the effort, time and cost of building and running each tier, and that cheap fast tests should make up the bulk of a suite.
- A worked example uses sfbeans, a fictional coffee-ordering API, where a request for a large iced latte under $8 by 9:00 is checked at each layer.
- The end-to-end eval is built with Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results and records their actions.






