# Vercel — Agents are users, so your software needs evals

- Company: Vercel (vercel.com)
- Announced: 2026-11-16T05:00:00+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://vercel.com/blog/agent-evals
- Record: https://forck.live/items/36828-agents-are-users-so-your-software-needs-evals
- Subject: v0 / AI SDK / AI Gateway

Vercel's guide explains how to build evals for agents that use software, focusing on whether an agent completes a user's task correctly. It argues that a successful request can still be the wrong outcome, since an agent can place a "successful" order for a hot latte when the user asked for an iced one, and logs look fine. Because LLM outputs vary between runs, the guide says one good manual test cannot tell you whether a fix worked, so the same tasks must be repeated and graded before and after a change. It groups agentic tests into a pyramid by how many LLM calls they need: deterministic tests, one-turn evals, and end-to-end evals, and says the cheap, fast tests should make up the bulk of a suite. The worked example is SF Beans, a fictional coffee-ordering API, with an end-to-end eval built using Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results, and records their actions. The guide defines task, variant, trial, run and trace, and says goals should come from observability, customer tickets, or the roadmap rather than aiming for full coverage.

## Evidence

Verbatim from https://vercel.com/blog/agent-evals:

> Agents are now users of software, doing work on people's behalf. Whether you're building the agent or the software it uses (API, CLI, MCP server, SDK), evals give you a way to measure how well that work gets done.
> 
> Agents probably missed your launch video and billboard on Highway 101, but they still go through the same funnel a human does to find and use a product: Discovery , Selection , Access , and Success .
> 
> You can build evals for any stage of this funnel. Here, we’ll focus on the final stage, Success : whether the agent completes the user’s task correctly.
> 
> That's more difficult to check than it sounds. An agent can place a “successful” order for a hot latte when the user asked for an iced one. The request worked, so the logs in your Observability stack look fine. And because LLM outputs vary between runs, one good manual test can’t tell you whether a fix worked. You need to repeat the same tasks and grade the outcomes, before and after the change.
> 
> That’s what an eval is, a test you run many times and score. Here’s how we build one.
> 
> Copy link to heading The agentic testing pyramid
> 
> If a test asks “did it work?”, an eval asks “how often does it work, and how well?”

---

Record: https://forck.live/items/36828-agents-are-users-so-your-software-needs-evals
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
