From the source
Amazon introduced the Universal Agent Evaluation Framework (UAEF), a framework-agnostic system that scores AI agents with one consistent set of metrics so teams can compare agent versions before release.
- Teams building agents on AWS use frameworks such as LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore, and each records execution differently, so metrics and scoring logic built against one trace format must be rebuilt when a team switches frameworks or runs agents on more than one.
- UAEF uses adapters to convert traces from each framework into a common format called the AgentTrace, then scores them with 36 built-in metrics across 7 dimensions, including tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning.
- Customers across 12 organizations in telecom, healthcare and financial services used UAEF to evaluate their agents and compare versions before release, according to the post.
- UAEF takes the agent under test and a dataset as inputs; ground truth is optional and only metrics that compare a response against a reference use it, while evaluating tool calling requires expected tool calls with their order and arguments.
- Traces can be supplied from a past run, log or observability store, or UAEF can invoke the agent itself given a runtime ARN or HTTP endpoint.
- UAEF screens traces for personally identifiable information such as email addresses, phone numbers and credit card numbers before data is stored, and persists runs as experiments in Amazon DynamoDB and Amazon S3.






