From the source
Amazon's playbook shows teams how to evaluate production agents on Amazon Bedrock AgentCore, arguing that a good score on one metric can hide a failure on another.
- It recommends starting from business outcomes rather than metric names, then breaking each outcome into observable behaviors and mapping them to evaluators at trace or session level.
- AgentCore draws evaluators from three sources: built-in evaluators authored and quality-tested by AWS, third-party managed evaluators from DeepEval and AutoEval, and custom evaluators that take a built-in or third-party evaluator as a base.
- Built-in evaluators are referenced by ID, such as Builtin.GoalSuccessRate and Builtin.Helpfulness, and the service handles the rest.
- In managed mode AgentCore hosts the third-party code, picks the judge model and runs inference on its own capacity, so teams already using those libraries in development can run the same metrics in production.
- Built-in evaluators cover response quality, task and goal completion, tool use, skills and safety, alongside three ground-truth trajectory checks.






