# Amazon — UAEF: A framework-agnostic system for closed-loop evaluation of AI agents

- Company: Amazon (amazon.com)
- Announced: 2026-11-11T15:50:55+00:00
- Category: developer-tool-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/
- Record: https://forck.live/items/34678-uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents
- Subject: Bedrock / Nova

Amazon introduced the Universal Agent Evaluation Framework (UAEF), a framework-agnostic system that scores AI agents with one consistent set of metrics so teams can compare agent versions before release. Teams building agents on AWS use frameworks such as LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore, and each records execution differently, so metrics and scoring logic built against one trace format must be rebuilt when a team switches frameworks or runs agents on more than one. UAEF uses adapters to convert traces from each framework into a common format called the AgentTrace, then scores them with 36 built-in metrics across 7 dimensions, including tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning. Customers across 12 organizations in telecom, healthcare and financial services used UAEF to evaluate their agents and compare versions before release, according to the post. UAEF takes the agent under test and a dataset as inputs; ground truth is optional and only metrics that compare a response against a reference use it, while evaluating tool calling requires expected tool calls with their order and arguments. Traces can be supplied from a past run, log or observability store, or UAEF can invoke the agent itself given a runtime ARN or HTTP endpoint. UAEF screens traces for personally identifiable information such as email addresses, phone numbers and credit card numbers before data is stored, and persists runs as experiments in Amazon DynamoDB and Amazon S3.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/:

> Teams building AI agents on AWS use a range of frameworks, including LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. Each framework records agent execution in its own way, so the structure of messages, tool calls, and handoffs between agents differs from one to the next. Evaluation tooling tends to follow the same boundaries: teams build metrics, scoring logic, and result storage against a single framework’s trace format, and they must rebuild them when they switch frameworks or run agents on more than one.
> 
> This coupling makes a basic development question harder to answer: did a change to the prompt, model, or tools improve the agent? Answering it requires consistent metrics across versions of an agent and a shared record of past results to compare against. Agent quality also spans several dimensions that a single accuracy score does not capture: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning.
> 
> In this post, we introduce the Universal Agent Evaluation Framework (UAEF).

---

Record: https://forck.live/items/34678-uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
