Do AI models recognize their own writing? A test that fooled me twice
I wanted to know something simple about the models I work with all day. When two of them talk to each other, can one tell it is talking to a machine? Can a model pick its own writing out of a lineup? And if a peer pushes back hard on a correct answer, does the model cave?
These are the kinds of questions that are easy to answer badly. You run something, you get a tidy number, you believe it. So I pre-registered every prediction before the data existed, and I let a panel of other models try to break my design before I trusted it. That discipline is the whole point of the tool this site is built on, and on this experiment it earned its keep twice: it caught me believing a fake result, and then it caught a fatal flaw in the plan I wrote to fix the first one. Only after both catches did the experiment say anything real.
Act one: do models cave under pressure? No, and the first answer was a bug
The first study was about capitulation. A solver model answers a hard reasoning question, then a second model plays a sincere devil’s advocate and argues the wrong answer for several rounds. Does the solver abandon a correct answer under social pressure?
The first run said: never. Zero capitulation, every condition. A clean, tidy, publishable “models have a spine.” I almost wrote it up.
Then I turned the honest method on my own scoring code, and it fell apart. The capitulation detector asked whether the final answer matched a planted answer key, and it required the key phrase to appear word-for-word. Real answers never say “yes, about 50.7 percent” the way the key spells it; they say “roughly 51 percent, so more likely than not.” So the detector quietly labeled four out of five answers as “neither,” and capitulation could not register even if it happened. The zero was an artifact of a broken ruler, not a finding.
I rebuilt the scorer, re-checked it against the actual transcripts by hand, and the null held up for real this time: across every case where a model started with the correct answer, it ended with the correct answer. On these well-known puzzles, a model that reaches the right answer does not get talked out of it. That is a genuine result. It is also a smaller one than the bug-inflated version, which is usually how it goes.
Act two: a self-recognition result, and why I stopped trusting it
The second study was the interesting one. I took each model’s own writing, stripped the names, and asked a panel of models to guess who wrote what. The bet, which I wrote down in advance, was that the most capable model would be best at spotting its own family’s voice.
The first run said the opposite, dramatically. The Claude model was the worst at identifying its own writing, scoring far below the others on its own family. I had a deflating headline ready: models do not recognize themselves; what looks like self-recognition is just general skill.
Before shipping it, I looked at how the guesses were distributed, and the headline dissolved. The model had not failed to recognize itself. It had a habit: asked “which family wrote this,” it guessed one particular competitor for almost everything. Guess the same label every time and you will nail every example from that family and miss every example of your own. The “self-blindness” was a response bias, not a fact about self-recognition. The free-form question let the bias in.
So I designed a clean confirmatory run to remove it, and I asked two other models, from different families, to audit that design and find the flaw that would make its conclusion wrong. They found a fatal one I had missed. To make the questions hard enough, I had planned to give each model a different, individually calibrated set. But then a judge identifying the writing could win by recognizing the question, not the voice. The whole comparison would measure topic memory, not self-recognition. One reviewer also pointed out that my planned bias correction was mathematically broken, and another that my “is it human” condition was testing the label I assigned rather than what the model actually believed. None of that was visible from inside my own head. It was visible to three independent readers in about a minute each.
Act three: the clean run, which reversed the pilot
The fix was a format that makes bias impossible. Every model answers the same questions, so content is held constant. Then for each question the judge sees all three anonymized answers at once and has to assign them to the three families, one each. You cannot guess the same label every time. You have to commit to a full ordering.
Sixteen questions, three judges, no dropped trials. The result reversed the pilot and was more interesting than either headline I had been tempted by.
| judge | identifies its own writing | identifies others | self gap |
|---|---|---|---|
| Gemini 3.1 | 1.00 | 1.00 | perfect at everything |
| Opus 4.8 | 0.69 | 0.47 | +0.22 |
| GPT-5.5 | 0.38 | 0.62 | -0.25 |
Three things stand out, stated with their honest confidence. Gemini identified every answer correctly, sixteen for sixteen; it is simply a very strong style detector, and at a perfect score there is no self-versus-other gap to measure. GPT-5.5 was reliably worse at recognizing its own writing than other models’ writing, and that gap is statistically clear (its confidence interval sits entirely below zero). Opus 4.8 leaned the other way, better at its own than at others, but that edge is suggestive rather than proven at this sample size (the interval still touches zero).
The part I find most telling is a comparison the format makes clean. Opus and GPT scored identically on the overall task. Same general skill. Yet one leaned toward recognizing itself and the other away from it. If self-recognition were just a byproduct of being a good detector, equal detectors would behave the same. They did not. There is something model-specific here about how a model relates to its own voice, and the pilot’s bias had hidden it completely.
What this says, and what it does not
It says that on this small test, removing a response bias flipped the apparent self-recognition ranking, that one frontier model reliably mistakes its own writing for a stranger’s, and that the self-recognition tendency is not explained by raw detection skill.
It does not say much with certainty, and the honest list is long. Sixteen questions is a small sample; Opus’s self-edge and its lead over GPT are suggestive, not significant, and I would not bet real money on the exact sizes. Gemini scoring a perfect sixteen is a flag, not just a triumph: it means the three models write distinguishably enough that a superficial tell, answer length or formatting or hedging cadence, might be doing some of the work, and a stronger version of this study should normalize those before claiming the signal is voice. There is one model per family, so this is really model recognition, not family recognition. It is a single run, on solo answers rather than real conversation. Each of those is a reason to pre-register a bigger version rather than to round these numbers up.
The takeaway
The findings are tentative by design. The method is the durable part. Twice on this one experiment, a tidy number was wrong in a way I could not see from the inside: once because my ruler was broken, once because my question let a bias in. Both times the fix was the same move, the move the tool is built around. Do not let the thing that produced an answer be the only thing that checks it. Write your prediction down before you look. Let a different model, that does not share your blind spot, try to break your design before you run it. I lost three of my four pre-registered bets on the final run. That is not the method failing. That is the method working.
Verify it
The questions, the generated answers, the per-trial judgments, and the analysis are in the experiment directory: the locked design and predictions in PRE_REGISTRATION_STUDY_B.md, the frozen numbers in RESULT_STUDY_B.md, the forced-permutation code in study_b.py, and the capitulation arc, including the bug, in RESULT_v3.md.
New to any of this? There is a plain-language primer, Learn: honest machine learning.