From the source
Vercel's guide explains how to build evals for agents that use software, focusing on whether an agent completes a user's task correctly.
- It argues that a successful request can still be the wrong outcome, since an agent can place a "successful" order for a hot latte when the user asked for an iced one, and logs look fine.
- Because LLM outputs vary between runs, the guide says one good manual test cannot tell you whether a fix worked, so the same tasks must be repeated and graded before and after a change.
- It groups agentic tests into a pyramid by how many LLM calls they need: deterministic tests, one-turn evals, and end-to-end evals, and says the cheap, fast tests should make up the bulk of a suite.
- The worked example is SF Beans, a fictional coffee-ordering API, with an end-to-end eval built using Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results, and records their actions.
- The guide defines task, variant, trial, run and trace, and says goals should come from observability, customer tickets, or the roadmap rather than aiming for full coverage.






