From the source
Perplexity's guide lays out six practical steps for evaluating AI agents, arguing that evals catch errors early and matter most for agents taking high-risk actions such as issuing refunds.
- The company defines evaluation as assessing whether an agent completes tasks correctly and meets an organization's requirements, and warns that without it product managers are left guessing.
- It lists five fundamental metrics: task success, action correctness, answer quality, reliability, and cost and latency, which together cover outcomes, tool use, output quality, consistency and resource use.
- The first step is to state what a successful run looks like, since agents can reach a correct answer through a leaked answer key or unauthorized actions, so Perplexity's senior data scientist Andrey Khomutov says both the result and the path taken must be checked.
- The second step builds a test set from real traces, starting with error analysis: annotate individual traces, group observations into failure modes, count how often each appears, and turn the important ones into test cases.
- Perplexity suggests reviewing roughly 100 traces but stopping once additional traces stop revealing new failure patterns.