# Amazon — Use decision models with Amazon Bedrock AgentCore Evaluations

- Company: Amazon (amazon.com)
- Announced: 2026-11-04T18:46:36+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/use-decision-models-with-amazon-bedrock-agentcore-evaluations/
- Record: https://forck.live/items/31699-use-decision-models-with-amazon-bedrock-agentcore-evaluations
- Subject: Bedrock / Nova
- Models affected: Jev, Strands Decider 2B

Amazon's guide shows how to connect decision model backends to Amazon Bedrock AgentCore Evaluations through code-based evaluators, for teams that want a label and a thresholdable confidence score rather than free-form text. Decision models take a structured state and a fixed set of allowed answers and return a calibrated probability over them, which the post says suits narrow, repeatable questions such as whether a response stayed on policy or whether a tool call is safe to approve. AgentCore Evaluations scores an agent across its lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic, and supports custom evaluators defined as either an LLM-as-a-judge or a code-based evaluator. Code-based evaluators run logic in an AWS Lambda function that can call an external system, model or API, including a decision model. The walkthrough connects two backends: Jev, a hosted decision model from TypeSafe, and Strands Decider 2B, an open-source decision model the reader runs. Both support yes/no questions returning a probability between 0 and 1, choice questions over up to 255 options, and score questions over 2 to 10 ordered levels. The post frames decision models as an addition to, not a replacement for, the built-in LLM-as-a-judge evaluators, useful as a screening layer that selects turns warranting deeper assessment.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/use-decision-models-with-amazon-bedrock-agentcore-evaluations/:

> Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust.
> 
> Amazon Bedrock AgentCore Evaluations , a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock.

---

Record: https://forck.live/items/31699-use-decision-models-with-amazon-bedrock-agentcore-evaluations
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md