Claude vs GPT vs Gemini Confidence Scores: The Least Confident Model in 222 Council Sessions
Three AI models answer the same question under the same prompt. One is the least confident seat 95% of the time — and it is not the one getting outvoted.
Council asks three AI models the same question and synthesises one verdict out of their answers. Every model returns the same shape: a recommendation, its reasoning, and a confidence score from 0 to 100.
That confidence number was supposed to be the cheapest useful signal in the product. It arrives free with an answer I am already paying for, it is a single integer, and it seems to mean something obvious. I used it to sort things. I used it, two days ago, to decide which advisor to put in front of people.
Then I ran the numbers across every session in the database, and the number turns out not to mean what it looks like it means.
Why these sessions are a fair comparison
The setup makes this cleaner than most model comparisons, and by accident rather than design.
Every seat on the panel gets the identical system prompt, the identical JSON schema, the identical instruction to answer in the user’s language, and the identical lowest-reasoning setting for its provider. They run concurrently on the same question — not a benchmark question, a real one somebody typed. Nothing about the arrangement favours any seat, and nothing in it varies except the model behind the seat.
Across 222 sessions since June, 98 of them had the default bench all three answering together. That is 98 paired comparisons: same input, three outputs, three self-reported confidence numbers.
Claude is the least confident seat 95% of the time
| Model | Avg confidence | Least confident seat | Sessions at ≤50 |
|---|---|---|---|
| Gemini 3.7 Flash | 77.9 | 4 of 98 | 3 |
| GPT-5.6 Terra | 75.4 | 1 of 98 | 0 |
| Claude Sonnet 5 | 55.6 | 93 of 98 | 30 |
Claude Sonnet 5 was the most confident seat in exactly one session out of 98. The average gap between it and Gemini, on the same question, is 22.3 points.
My first instinct was that this was an artefact of who was asking. One account in the database has 50 sessions and is almost certainly my own testing, and if that account happened to ask Claude-unfriendly questions it could drag the whole result. So I dropped it: 93 sessions remain, spread across 79 different people, and the averages barely move — 78.0, 75.1, 55.1, with Claude the least confident seat in 89 of 93. It is not one person’s question style. It is the model.
The obvious explanation is wrong
The tidy story would be that Claude is the panel’s contrarian — that it is less confident because it keeps landing on the minority view, and low confidence is the honest report of being the odd one out.
Two things in the data say otherwise.
No lab is the habitual dissenter. When the synthesiser writes up where the panel diverged, it usually names which lab broke ranks. Across 222 sessions it names OpenAI 163 times, Google 158, and Anthropic 154. That is about as evenly distributed as three numbers can be. Disagreement is not one seat against two; it is genuinely spread around the table.
Confidence does not rank the same way agreement does. If the score tracked “am I about to be outvoted”, the most confident model would be the one that most often backs the final verdict. It isn’t:
| Model | Avg confidence | Backs the final verdict |
|---|---|---|
| Gemini 3.7 Flash | 77.9 (highest) | 82.4% |
| GPT-5.6 Terra | 75.4 | 91.7% (highest) |
| Claude Sonnet 5 | 55.6 | 74.8% |
Gemini is the most confident seat at the table and agrees with the room less often than GPT does. The orderings come apart at the top, which is where a real signal would have to hold up.
So the number is not measuring a property of the answer. It is measuring a property of the lab — how a given model house was tuned to talk about its own certainty. Anthropic’s model hedges. Google’s does not. Put them side by side under an identical prompt and you are reading two dialects, not two measurements.
I want to be precise about the claim, because it would be easy to overstate. This is self-reported confidence, under one prompt, at each provider’s lowest reasoning setting, on advice questions. It is not a claim that Claude is worse, or better calibrated, or more honest. It is a narrower and more practical claim: these numbers are not comparable across vendors, so any code that ranks models by them is sorting on house style.
Which is exactly what I had just shipped.
The feature I broke by believing my own number
The day before I ran this query, I shipped a small thing to both apps. The advisor chat — where you can follow up 1:1 with one member of the panel — sits below every expert card, and almost nobody scrolled far enough to find it. Twelve of 222 sessions have ever carried a follow-up. So I added a card directly under the verdict that names the sharpest disagreement and pre-selects an advisor to argue with.
Which advisor? The most confident one. It felt self-evidently right when I wrote it: point people at the seat that is surest of itself.
Replaying that rule against all 138 sessions:
| Advisor the card would select | Share |
|---|---|
| Gemini 3.7 Flash | 63.0% |
| GPT-5.6 Terra | 28.3% |
| Gemini 3.1 Flash-Lite | 7.2% |
| Claude Sonnet 5 | 0.7% — one session out of 138 |
I built a feature that routes essentially every follow-up conversation to Google, on the basis of a number that only says Google’s model is chattier about its own certainty. A third of my own bench is invisible behind it. It is live right now. Nobody would ever have reported it as a bug, because from the outside it just looks like a card with a name on it.
The fix is not a better ranking function over the same number. It is to stop ranking on that number: the interesting seat to argue with is the one that dissented, and dissent is something I can observe directly in the vote array instead of inferring from a self-report.
While I was in there: the panel has never once agreed
Two other things fell out of the same query, and one of them is dead code I did not know I had written.
The nudge card has a branch that suppresses it when the council is unanimous, on the grounds that there is no split to point at. That branch has never executed. Of 138 sessions with the current schema, zero have an empty divergence list. Not one. The average session carries 3.1 explicit disagreements, and:
- In 76% of sessions, at least one of the three proposed options had the seats split their votes.
- In 38%, there was no option that all three models backed.
- Consensus scores run from 35 to 100, averaging 77.8.
I had written a guard for a case that does not exist in reality. Three frontier models given a real decision and told to commit to a recommendation will find something to disagree about, every single time.
The part that justifies the product
The last finding is the one I did not go looking for.
Career questions are the biggest category in Council by a wide margin — 29.3% of all sessions, more than double the next one. They are also the category the models agree on least:
| Category | Avg consensus |
|---|---|
| Learning | 93.9 |
| Health | 82.1 |
| Lifestyle | 81.3 |
| Tech | 76.3 |
| Money | 76.0 |
| Career | 72.4 |
| Creative | 69.6 |
The thing people most want a second opinion about is precisely the thing where one model’s answer is least representative of what the models collectively think. If the panel agreed about careers, Council would be an expensive way to do what one API call already does. It doesn’t, so it isn’t.
That is the strongest argument for the product I have, and I did not write
it — I found it in a GROUP BY.
The honest caveats
This is 222 sessions from 109 people, of which three are paying. It is small. A chunk of the early sessions are my own multilingual testing of the same example question. The category table has fewer than 20 sessions behind several of its rows, and I would not defend the gap between Health and Lifestyle for a second.
But the headline result does not need the sample to be large, only paired — and 93 out of 98, holding at 89 of 93 across 79 unrelated people, is not a sampling artefact. Neither is a feature that routes 0.7% of its traffic to a third of the bench.
What I would tell anyone building on more than one model
Do not compare self-reported confidence across vendors, and never let it drive routing, ranking, ordering, or which model gets the last word. Within a single model it may still carry information — Claude’s own 15 and Claude’s own 95 plausibly mean different things. Across models it is a unit conversion nobody has defined, and code that treats the numbers as interchangeable will quietly develop a favourite lab.
The broader lesson is older and duller than the finding. I shipped a routing rule one day because it was obviously correct, and disproved it the next with a query I could have run first. The query took four minutes. Everything it needed was already sitting in the database, untouched, the entire time the feature was being built and reviewed.