# Amazon — The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

- Company: Amazon (amazon.com)
- Announced: 2026-11-10T22:35:03+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/
- Record: https://forck.live/items/34326-the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-amazon-bedrock
- Subject: Bedrock / Nova

Amazon's playbook shows teams how to evaluate production agents on Amazon Bedrock AgentCore, arguing that a good score on one metric can hide a failure on another. It recommends starting from business outcomes rather than metric names, then breaking each outcome into observable behaviors and mapping them to evaluators at trace or session level. AgentCore draws evaluators from three sources: built-in evaluators authored and quality-tested by AWS, third-party managed evaluators from DeepEval and AutoEval, and custom evaluators that take a built-in or third-party evaluator as a base. Built-in evaluators are referenced by ID, such as Builtin.GoalSuccessRate and Builtin.Helpfulness, and the service handles the rest. In managed mode AgentCore hosts the third-party code, picks the judge model and runs inference on its own capacity, so teams already using those libraries in development can run the same metrics in production. Built-in evaluators cover response quality, task and goal completion, tool use, skills and safety, alongside three ground-truth trajectory checks.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/:

> When you deploy an agent to production, one of the hardest questions is whether it’s actually working? The bar isn’t whether it runs. It’s whether it does what the business needs, at scale. Many teams will tell you they can’t really see this. They spot-check outputs, watch latency, and collect user feedback when they can get it. None of that tells you whether the agent finished the user’s task, stayed on topic across a long session, leaked personally identifiable information (PII), or quietly skipped half the steps it was supposed to follow.
> 
> Figure 1: Teams usually watch surface signals, but business outcomes are decided by the agent behaviors below the waterline
> 
> Agent evaluation solves this problem. What makes this challenging is that a good score on one metric can hide a failure on another. A support agent can resolve every ticket and still expose personal information in its replies. In this case, the agent is failing although it has a high completion rate. To get evaluation right, measure the behaviors that matter for each agent, at the right level of detail.
> 
> This post shows how to do this on Amazon Bedrock AgentCore.

---

Record: https://forck.live/items/34326-the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-amazon-bedrock
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
