# Cohere — Rethinking Confidence in Reasoning Models

- Company: Cohere (cohere.com)
- Announced: 2026-10-20
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://cohere.com/blog/rethinking-confidence-in-reasoning-models
- Record: https://forck.live/items/25236-rethinking-confidence-in-reasoning-models
- Subject: Command

Cohere researchers say standard reasoning models cannot reliably report how likely their answers are to be correct, and propose a training method called CALIBER that teaches a model to give two separate confidence estimates. The first estimate is made before reasoning begins and answers how likely the model is to solve the question at all; the second is made after an answer is produced and answers how likely that particular answer is to be correct. CALIBER trains the model to report both in one response, alongside its reasoning and answer, using different targets for each: initial confidence is trained on how often the model solves that question across several attempts, while final confidence is trained on whether the individual answer is correct. The researchers trained 7B and 30B models on 30,000 math problems and evaluated them on 1,000 held-out problems plus three question-answering benchmarks. On BigMathDigits, the 7B model's Expected Calibration Error fell from 0.059 for the strongest single-confidence baseline to 0.028 for CALIBER, a 52.5% reduction, and CALIBER also posted the best AUROC (0.806) and Brier score (0.130). The team argues that a single confidence score is not enough, because a model that reports 70% every time can be perfectly calibrated yet give a user no way to decide which answers to check first.

## Evidence

Verbatim from https://cohere.com/blog/rethinking-confidence-in-reasoning-models:

> You roll two fair six-sided dice. What is the probability that their sum is at least 10?
> 
> A model might hesitate at first. Then it counts the possibilities: three ways to roll 10, two ways to roll 11, and one way to roll 12. Six outcomes out of 36. The answer is 1/6 .
> 
> In our illustrative example (figure below, on the left), the model starts at 25% confidence and finishes at 88%. Those percentages describe its confidence in getting the answer right, not the probability of the dice outcome. Working through the problem has changed its assessment.
> 
> Confidence can move the other way, too (figure below, on the right). Reasoning through a solution that initially looks straightforward may reveal a mistaken assumption or a contradiction. A useful confidence estimate should be able to reflect that change, even when the model still gives the wrong answer. However, standard reasoning models do not necessarily come with this ability: producing a detailed reasoning process does not guarantee that a model can reliably assess and communicate its uncertainty.

---

Record: https://forck.live/items/25236-rethinking-confidence-in-reasoning-models
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md