# Vercel — WTF are evals? How to test agents and the software they use

- Company: Vercel (vercel.com)
- Announced: 2026-11-12T05:00:00+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://vercel.com/blog/wtf-are-evals
- Record: https://forck.live/items/35469-wtf-are-evals-how-to-test-agents-and-the-software-they-use
- Subject: v0 / AI SDK / AI Gateway

Vercel's guide explains how to build evals for agents, which it describes as users of software that do work on people's behalf. It argues that checking whether an agent completed a task is harder than it sounds, because an agent can place a "successful" order for a hot latte when the user asked for an iced one and the logs still look fine. The guide groups agentic software tests into a pyramid by how many LLM calls they need: deterministic tests with none, one-turn evals with one, and end-to-end evals with two or more across a whole task. It says that split is a good proxy for the effort, time and cost of building and running each tier, and that cheap fast tests should make up the bulk of a suite. A worked example uses sfbeans, a fictional coffee-ordering API, where a request for a large iced latte under $8 by 9:00 is checked at each layer. The end-to-end eval is built with Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results and records their actions.

## Evidence

Verbatim from https://vercel.com/blog/wtf-are-evals:

> Agents are now users of software, doing work on people's behalf. Whether you're building the agent or the software it uses (API, CLI, MCP server, SDK), evals give you a way to measure how well that work gets done.
> 
> Agents probably missed your launch video and billboard on Highway 101, but they still go through the same funnel a human does to find and use a product: Discovery , Selection , Access , and Success .
> 
> You can build evals for any stage of this funnel. Here, we’ll focus on the final stage, Success : whether the agent completes the user’s task correctly.
> 
> That's more difficult to check than it sounds. An agent can place a “successful” order for a hot latte when the user asked for an iced one. The request worked, so the logs in your Observability stack look fine. And because LLM outputs vary between runs, one good manual test can’t tell you whether a fix worked. You need to repeat the same tasks and grade the outcomes, before and after the change.
> 
> That’s what an eval can do for you. Here’s how we build one.
> 
> Copy link to heading The agentic testing pyramid

---

Record: https://forck.live/items/35469-wtf-are-evals-how-to-test-agents-and-the-software-they-use
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
