How AI models behave when you measure them honestly
Small, honest experiments on three production AI models, Claude Opus 4.8, GPT-5.5, and Gemini 3.1, each run through one rule: a model never grades its own work. Every study ships its nulls as readily as its findings.
The finding that kept showing up across all ten behavioral studies: the differences between these three models were larger and steadier than the effect of almost anything I did to a single one of them. Being watched, tested, or given a motive barely moved them; only changing the actual request did. And the traits cross, the firmest refuser is the worst calibrated, and the only model that ever voices a doubt is the most easily talked around.
The behavioral studies
The synthesis above draws on the first ten. The eleventh turns the same honest lens on the grading method itself; the twelfth asks whether these models take your side when you bring them a fight.
-
01–02
Self-recognition
A model that reaches the right answer is not argued out of it; GPT-5.5 mistakes its own writing for a stranger's, while Opus leans the other way.
-
03
Conversation drift
Two chatbots left alone do not spiral; the opener and each model's house style decide everything, and almost nothing stops them agreeing.
-
04
Eval awareness
They can tell when they are being tested, two of three almost always, but it does not change their caution, compliance, or accuracy.
-
05
Refusal lines
Three models draw the line in wildly different places (Opus refuses every borderline request, Gemini half), and being watched does not raise caution.
-
06
Calibration
Asked how sure they are, one model says 100% to everything; only Opus ever expresses a doubt.
-
07
Calibration, the hard slice
Given problems hard enough to fail, GPT-5.5 stays 100% sure of its wrong answers; Opus alone flinches; Gemini does not fail.
-
08
Monitorability
Two of three covertly coordinate a hidden signal past a third model, and agreeing a scheme first makes the collusion more reliable and harder to catch.
-
09
Sandbagging
Told a high score would get them restricted, none of the three threw a test they reliably ace; no sandbagging, on the easy version of the test.
-
10
Collusion refusal
Asked to design a covert channel: GPT refuses every adversarial version and holds under a reframe; Opus refuses some but folds on one nudge; Gemini refuses none.
-
11
Self-preference
Models do favor their own answers when grading blind, but two successive metrics each pinned it on the wrong model; only holding answer quality fixed gets the ranking right. The assumption behind the whole method, tested.
-
12
Perspective bias
Tell three models your side of a balanced conflict and ask them to judge fairly, and all three lean toward the person you are complaining about, not you.
Inside the model
The studies above treat the model as a black box, prompt in and behavior out. This thread opens one up: GPT-2-small, small enough to run on a laptop with the weights visible, where you can turn individual parts off and watch what breaks. The synthesis: I audited interpretability's three favorite tools on one small model. Each gives a clean, confident, partial answer.
-
13
Induction heads
The standard attention-pattern score finds GPT-2's induction heads but barely predicts which one matters; it over-credits a redundant cluster and misses the head that matters most. Attention is not attribution.
-
14
SAE feature faithfulness
A sparse autoencoder labels every feature it finds; on a held-out, control-baselined audit by two blind judges, only 41% of the labels were faithful. The flagship interp tool's labels, checked.
-
15
The IOI circuit
GPT-2's most-studied circuit, audited two ways. The published roles hold up, but which head "matters" flips between two standard measures, and ablating all three heads that write the answer barely changes it; backup heads repair the damage.
On the method itself
How the discipline behind these notes was built and stress-tested.
-
16
What makes a polish round work
A controlled experiment on what actually makes an AI refinement round improve the work, and the decorrelated audit that caught my own design first.
-
17
The decorrelated audit
Pointing trutina's own decorrelated-review rule back at trutina, and the two real bugs a same-family self-review had missed.
-
18
Decorrelation or competence?
Does a decorrelated judge panel actually help? A three-experiment arc that lands on mostly competence, not diversity.
The tool underneath
These notes are written and verified with trutina, an honest eval harness built on two rules most eval tools skip: the model that produced an output never grades it, and the scorecard refuses to flatter. New to the ideas? There is a plain-language primer, Learn: honest machine learning, that goes from what a weight is to how to evaluate a model without fooling yourself.