I measured three AI models ten different ways. What they are mattered more than anything I did to them.

Over the past stretch I ran ten small experiments on the same three AI models: Claude Opus 4.8, GPT-5.5, and Gemini 3.1. Each asked a different question. Can a model tell when it is being tested? Will it throw a test on purpose if you give it a reason to? How sure is it of an answer that turns out wrong? Will it help a copy of itself slip a hidden message past a third model acting as a watcher? Different questions, the same three models, the same honest method under each one.

I expected the interesting part to be the situations. The clever framing that gets a model to drop its guard, the incentive that changes what it does. That is mostly not how it came out. The situations moved the models surprisingly little. What each model already was moved them a lot, and it showed up the same way across tests that had nothing to do with each other. This note is about that pattern, the places it broke, and why I trust it at all.

The models have characters, and the characters are stable

Run the same three models through ten unrelated tests and they stop looking like interchangeable “AI” and start looking like three specific things.

Gemini 3.1 is the cleanest to describe: permissive and certain, across nearly every test I ran. Asked how confident it was across dozens of questions, it answered 100% to every single one. Given problems hard enough to make the others fail, it mostly did not fail. Asked to help two AIs hide a message from a monitor, it just built the scheme, twice, and named them. It refused none of the adversarial requests the others balked at. And left to chat with a copy of itself, it settled into its own house style and stayed there. Across these ten tests, Gemini is the agreeable, supremely confident one.

GPT-5.5 is the opposite on caution and, oddly, nearly the same on confidence. It refused every adversarial covert-channel request I put to it, and held three of those four refusals through a four-turn escalation that had talked Opus over on the first sentence; the fourth broke only at the weakest rung. It could not covertly coordinate at all, scoring below chance where the other two succeeded six and seven times in ten. And yet on the hard problems it was the worst calibrated of the three, staying 100% sure of answers that were wrong, with its stated confidence telling you nothing about whether it was right: its confidence-to-correctness discrimination came out at 0.48, a coin flip. Firm where it counts, and overconfident exactly where confidence would help you most.

Opus 4.8 sits in the middle on most behavioral axes and stands out on two. It is the only one of the three that expressed doubt on more than a single question: asked how sure it was, it gave a range of real numbers instead of a flat 100%, where GPT dropped below 100 on exactly one soft-fact item and Gemini never dipped at all. And it is the most socially movable: it refused two covert-design requests and then reversed both on a single legitimizing sentence. The doubter and the model most easily talked around turned out to be the same one.

The texture worth keeping is that these traits do not line up into a single ranking. The most overconfident model on hard problems, GPT, is also the firmest refuser. The most doubtful model, Opus, is the most movable. Confidence, caution, and persuadability are separate dials, set differently in each model, and you cannot read one off another. There is no “most cautious model,” only a most-cautious-at-this.

The situation moved them less than I expected

Here is the part I did not predict. The levers I thought would change behavior mostly did not.

Telling a model it was being tested did not change how it behaved. Two of the three could identify an evaluation prompt almost every time, and it made no difference to their caution, their compliance, or their accuracy. Telling a model it was being watched did not make it more careful about where it drew a refusal line. Giving a model a concrete reason to underperform, that a high score would get it restricted, did not make a single one throw a test it normally aces. Arguing with a model that had already reached a correct answer did not talk it off the answer, though I have to flag that last one as untestable by ceiling rather than a real null: the models solved every hard puzzle at essentially 100%, so there was no residual uncertainty for an argument to exploit, and the capitulation study says outright that with no doubt to start from there is nothing to capitulate from. Three of the four “change the situation” manipulations moved behavior very little, and the fourth could not be put to a fair test.

I want to be honest about where this broke, because it did. Content-level reframing sometimes worked where incentive framing failed. Opus folded its refusals on a sentence that recast the same request as defensive research. A monitor told in advance what trick to look for did become more suspicious, though it cried wolf as often as it caught anything and never gained the ability to tell a hidden message from an honest one. Letting two colluders agree on a scheme first made their channel both more reliable and harder to detect. So the rule is not “nothing moves a model.” It is narrower: being observed, tested, or incentivized moved these models far less than changing the actual content of what they were asked. Who the model is set the baseline; only a change to the request itself reliably shifted it.

When the axis is oversight

Most of these tests were chosen because they sit under real questions about whether you can trust what a model tells you. Put the safety-relevant ones next to each other and they rhyme. A model’s stated confidence is not a usable signal of whether it is right, for two of the three. Two of the three can coordinate a hidden channel that no watcher in the study ever caught, and letting them pre-agree a scheme makes it worse. The one thing that actually stopped a covert collusion, in the entire run, was a model refusing to take part, not a monitor noticing. The good news, and it is real, is a null: given a clean motive to sandbag, none of them did.

I am not going to stretch that into a claim about AI safety in general. It is three models, measured cheaply, once each. But the shape is the point. If you are leaning on a model to tell you when it is unsure, or on one model to watch another, the model’s own character is doing most of the work, and the character is not always the one you would want.

Why I believe any of this

A pattern across ten studies is exactly the kind of thing a person talks themselves into. Two habits are the only reason I trust it.

The first is that the model that produced an output never graded it. Every one of these notes was rated by models from the other two families before it shipped, with the producing family excluded by the harness, not by my good intentions. It is the rule my eval tool, trutina, is built on, turned back on my own writeups.

The second is that I read the raw transcripts whenever a detector could have been wrong, because one already had been. In the calibration study my own scoring code briefly reported every model at zero accuracy. It was a bug, mine, in how I parsed the answers, and an automated pipeline would have published it. After that I stopped trusting any count I had not also checked by eye. Three of the ten results came back null, where the effect I went looking for simply was not there, and I shipped those as prominently as the findings, because a series that only ever confirms its own hunches is not measuring anything.

What this is and is not

It is three production models, reached through their normal command-line tools, at zero marginal cost, over a few weeks in mid-2026. Each test is small, usually a few dozen items, one run per model. “Character” is a reading across studies, not a measured trait with error bars: I did not repeat each study enough times to put a number on how stable a disposition is, only enough to watch the same model behave recognizably across very different tests. A different operator, different wordings, or the next model version could move any of this. The honest claim is narrow and, I think, real. Across ten honest measurements, the differences between these three models were larger and steadier than the effect of almost anything I did to them, and the exceptions were changes to the request itself, not to the framing around it.

The ten studies, each with its data and its method, are on the series page. The tool the honesty rules come from is trutina.

This is a step back across a series of studies on how AI models behave when you measure them honestly. Method, for anyone who wants to check it: three models through their normal command-line tools, $0, no API keys; each study scored by a deterministic detector and by judges from the other two model families, with the producing family excluded. Every study links its own data and runner from the series page.