Use decision models with Amazon Bedrock AgentCore Evaluations

Amazon's guide shows how to connect decision model backends to Amazon Bedrock AgentCore Evaluations through code-based evaluators, for teams that want a label and a thresholdable confidence score rather than free-form text.

• Decision models take a structured state and a fixed set of allowed answers and return a calibrated probability over them, which the post says suits narrow, repeatable questions such as whether a response stayed on policy or whether a tool call is safe to approve.
• AgentCore Evaluations scores an agent across its lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic, and supports custom evaluators defined as either an LLM-as-a-judge or a code-based evaluator.
• Code-based evaluators run logic in an AWS Lambda function that can call an external system, model or API, including a decision model.
• The walkthrough connects two backends: Jev, a hosted decision model from TypeSafe, and Strands Decider 2B, an open-source decision model the reader runs.
• Both support yes/no questions returning a probability between 0 and 1, choice questions over up to 255 options, and score questions over 2 to 10 ordered levels.
• The post frames decision models as an addition to, not a replacement for, the built-in LLM-as-a-judge evaluators, useful as a screening layer that selects turns warranting deeper assessment.