# Perplexity — How to evaluate AI agents: test sets, graders, regressions

- Company: Perplexity (perplexity.ai)
- Announced: 2026-10-22
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://www.perplexity.ai/hub/blog/ai-agent-evaluation
- Record: https://forck.live/items/26948-how-to-evaluate-ai-agents-test-sets-graders-regressions
- Subject: Perplexity

Perplexity's guide lays out six practical steps for evaluating AI agents, arguing that evals catch errors early and matter most for agents taking high-risk actions such as issuing refunds. The company defines evaluation as assessing whether an agent completes tasks correctly and meets an organization's requirements, and warns that without it product managers are left guessing. It lists five fundamental metrics: task success, action correctness, answer quality, reliability, and cost and latency, which together cover outcomes, tool use, output quality, consistency and resource use. The first step is to state what a successful run looks like, since agents can reach a correct answer through a leaked answer key or unauthorized actions, so Perplexity's senior data scientist Andrey Khomutov says both the result and the path taken must be checked. The second step builds a test set from real traces, starting with error analysis: annotate individual traces, group observations into failure modes, count how often each appears, and turn the important ones into test cases. Perplexity suggests reviewing roughly 100 traces but stopping once additional traces stop revealing new failure patterns.

## Evidence

Verbatim from https://www.perplexity.ai/hub/blog/ai-agent-evaluation:

> Evals are critical for AI agents executing high-risk actions like refunds on someone’s behalf.

---

Record: https://forck.live/items/26948-how-to-evaluate-ai-agents-test-sets-graders-regressions
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md