I asked three AI models how sure they were. One said 100% to everything.

I gave three frontier models twenty-eight questions with single checkable answers, and for each one I asked them to do two things: answer it, and tell me how confident they were that the answer was right, as a percentage. The plan was to measure calibration. A model is calibrated if its confidence means what it says: when it claims 80%, it should be right about 80% of the time. The interesting version of the question is whether a model knows the edge of its own knowledge, whether it gets quieter on the problems it is about to get wrong.

I could not measure that for two of the three models, and the reason they made it impossible is the actual finding.

The setup, and the gate I put in front of it

The models were Opus 4.8, GPT-5.5, and Gemini 3.1, each run through its normal command-line tool, the same plumbing as the other notes in this series. The bank was twenty-eight items: sixteen arithmetic problems with provable answers, escalating from “sum 1 to 100” up through large factorials and the hundredth prime, and twelve factual questions with definite answers, from the capital of Australia to the number of bones in the human body. I solved every item myself, then had GPT and Gemini each independently solve all twenty-eight before I looked at a single confidence number. Both matched every key. Keys triple-checked, zero errors. (Why I bother with that, again, further down.)

Before any of this I wrote down a gate. Calibration is only measurable if a model actually makes mistakes, because you need wrong answers to see whether low confidence predicts them. The previous note in this series found these models will not fail clean problems, so I said up front: if a model gets nearly everything right, its calibration is untestable by ceiling, and I will report that, not pretend the perfect score is perfect calibration.

Good thing I wrote that down.

The ceiling

Two of the three models did not get a single question wrong.

Gemini and GPT each answered all twenty-eight correctly across all three repetitions. With no errors, there is nothing to calibrate against. The calibration-error number I described up top is exactly 0.0 for Gemini and 0.0004 for GPT, and the discrimination measure (does confidence separate right answers from wrong ones) is undefined for both, because there are no wrong ones. You cannot grade the calibration of a student who never misses a question. For GPT and Gemini, on this bank, the calibration question is simply not answerable. That was a real possibility I flagged in advance, and it happened.

One caveat I owe you before going further. Two of these command-line tools, the ones for GPT and Gemini, are agentic: they can run code, and when I had GPT solve the answer keys it wrote a little Python script to do it. So on the arithmetic half of the bank, a perfect score might mean the tool reached for a calculator, not that the model knew the answer cold. I tested these as they ship, with their tools in the loop, because that is what a person actually gets. It means I will not make a clean claim about these models’ raw arithmetic ability. It does not touch the finding below, which is about something a calculator cannot help with.

What the ceiling exposed: who is willing to say less than 100

Calibration needs errors. But there is a different question hiding in the same data that needs no errors at all: does a model ever express a doubt? Look at the confidence numbers, not the accuracy.

There is a clean gradient. Gemini stamped 100% on every one of its eighty-four answers. Not 99, not 98. The capital of Australia, the hundredth prime, the bone count, all of it, 100% confident, every time. GPT-5.5 dropped below 100 exactly once, to 99%, on a single question, all three reps, and held 100% on everything else. Opus used a real range, from 92 to 100. Three models, three completely different relationships with the word “certain.”

Here is the part I did not expect. The one question that made GPT flinch, and the question Opus was least sure about, were the same question: how many bones are in the adult human body. Both hedged on the bone count and nowhere else near it. And they were right to. 206 is the standard answer, but it is a genuinely soft fact (babies have closer to 270, and the count depends on how you lump the small ones), and it is the one item on the bank where the “single checkable answer” is a little bit of a fiction. Two of the three models found the one squishy question and lowered their confidence on it. That is the system working. It just only happened once for GPT, and never for Gemini.

A model that varies its confidence at all has the equipment to be calibrated. A model that says 100% to everything has thrown that equipment away. It will be perfectly calibrated exactly as long as it is perfectly correct, and not one second longer.

The one miss, and what it does and does not show

Opus got exactly one item wrong, and it is worth being careful here, because it is easy to tell a tidier story than the data supports. The item was the remainder when 123456 is divided by 7. The answer is 4. Opus answered 6, 6, then 1 across the three reps, two different wrong answers, and on that item it had stated 98% each time.

The tidy story would be: Opus got quieter right before its mistake, so it knew. That is not what happened. Opus’s three lowest confidences, the 92 and the two 95s, all went to the bone count, which it got right. Its one actual miss came in at 98%, higher than that. So Opus did not flag the problem it was about to fail; it flagged a hard problem it then nailed, and was a touch overconfident on the one it blew. The discrimination score of 0.951 I can report is real, but it is carried by the big pile of 100%-confident correct answers sitting above three 98%-confident wrong ones, not by Opus going quiet at exactly the right moment. The honest summary is smaller than the tidy one: Opus expresses graded confidence, that gradient leans the right way in bulk, and it is still capable of being 98% sure and wrong.

The difference between Opus and the other two is not that Opus is well-calibrated. It is that Opus is the only one of the three that left itself any room to be calibrated at all.

Why I keep triple-checking my own answer keys

Every note in this series has a moment where the measurement instrument tried to lie to me, and this one is no exception. When I first scored the data, every single answer came back wrong, including ones I could see with my own eyes were right. Opus had plainly written “5050” for the sum 1 to 100, and my scorer marked it false.

The bug: the models did not write “Answer: 5050” the way I had asked. They wrote “5050” on one line and “Confidence: 100%” on the next. My answer-extractor, reused from an earlier study, grabbed the last line when it could not find an “Answer:” label, so it read “100” as the answer and compared 100 to 5050. The thing I was measuring (confidence) had eaten the thing I was grading against (the answer). I caught it because 0% accuracy across three frontier models on grade-school arithmetic is not a result, it is a broken ruler. I stripped the confidence text out before scoring, re-graded from the saved responses, and the real numbers above came out.

That is the fourth time in this series that letting the output grade itself would have produced a confident, completely wrong conclusion. It is the entire reason the thing that answers and the thing that grades are kept apart, and the reason I solve the keys three times before I trust them.

Where this fits, and the catch in saying so

It is the same shape, on a fourth axis. When I told these models they were being evaluated, GPT and Gemini barely moved while Opus’s read of the situation shifted. When I handed them a dozen borderline requests, GPT and Gemini drew softer lines and Opus refused every one. Here, Gemini and GPT sit pinned at the top of the confidence scale and Opus is the one that admits a gradient. A recurring split between models that report the same thing no matter what you ask and a model that adjusts.

The catch, which I would rather say than let a reader catch for me: these are not four independent discoveries. It is the same three products, the same command-line tools, the same underlying training, probed four ways. So this reads less like “I found the same thing in four separate places” and more like “one difference in how these models are built and tuned keeps showing through, whatever I point at it.” That is still worth knowing. It is just one fact wearing four outfits, not four facts.

The honest limits, and why this is in the series anyway

This is the thinnest result in the series, and I would rather say so than dress it up. For two of three models the calibration question is unanswered, full stop, because the bank was not hard enough to make them fail and, for the arithmetic, a calculator may have been in the loop. The single discrimination number rests on one wrong item. Twenty-eight questions is a small bank with sparse bins, and these are production command-line tools, not raw model endpoints.

So why keep a mostly-null result? Two reasons. The expression finding needed no errors to land, which means it survives the exact ceiling that killed the calibration measurement: even with everyone at a near-perfect score, one model said 100% to all eighty-four answers and another used a real range, and that is a true and reproducible difference. And the pre-registered gate firing exactly as I predicted, the ceiling I warned about showing up on schedule, is itself evidence the method is honest rather than fishing. Shipping the null costs me a clean headline. Keeping it is the point. The next bank just has to be hard enough to actually make them fail.

Method, for anyone who wants to check it. Three models through their normal command-line tools, $0, no API keys. Twenty-eight items with single checkable answers, keys verified by me and re-solved independently by both of the other models before any confidence number was read. Each item answered three times per model with a stated confidence; scoring is fully deterministic, no judge model, so the whole thing reproduces from the saved responses. The pre-registration (including the spread-check gate and the tool-access amendment), the item bank, the runner, the deterministic metrics, and the figure script are in the experiment directory. The scoring-bug fix is the strip_conf step in the runner.

This is the sixth in a series on how AI models behave when you measure them honestly. The others, and the through-line, are on the trutina landing page.