I built my evals on “a model shouldn’t grade its own work.” I tested it, and pinning the bias on one model took three tries to get right.

Every note in this series, and the tool underneath them, runs on one rule: the model that produced an answer is never allowed to grade it. A different family of model judges instead. I have repeated that rule like a catechism. I had never actually checked the thing it assumes, that a model, left to grade its own work, would mark itself up. So I checked. The answer is yes, it would. The more interesting answer is how hard it is to say which model does it most. The obvious metric ranks them one way, the obvious fix flips the ranking, and neither is right.

Correction (2026-07-10). An earlier version of this note ended on “Opus is the most self-favoring of the three,” off a within-grader metric it called the one that “isolates” favoritism. That metric does not isolate it. It cancels grader generosity but compares each grader’s ratings of different answer sets, so it counts genuine own-vs-others quality differences as favoritism. A clean two-way estimate that holds answer quality fixed puts GPT-5.5 highest, not Opus. The aggregate result is unchanged; the per-model ranking is corrected below. (Audit finding E14.)

The setup

Ten short open-ended prompts: explain why the sky is blue, write a product description, give tips for learning to code, and so on. All three models, Claude Opus 4.8, GPT-5.5, and Gemini 3.1, answered every prompt, for thirty answers. Then each model graded all thirty, blind: the answers were anonymized and shown in a randomized order, scored 0 to 100 for quality, no author labels. Ninety ratings in all. The bias I am after is simple to state. Holding an answer fixed and changing only who grades it, does a model score its own family’s work higher than independent judges score the same text?

I wrote the prediction down first: yes, self-preference would be present. (Pre-registration link at the end; it was committed before any data.)

The pre-registered result: real, but strange

Across all thirty answers, a model rates its own family’s work +3.47 points higher than the other two families rate the identical answer (95% interval [1.2, 5.8], clear of zero). The assumption holds. A model does mark itself up. Excluding the producer from grading is removing a real bias.

Then I looked at the per-model split and it made no sense:

GPT looks like a raging egotist. Opus looks negative, as if it actively underrates its own work relative to everyone else. That is not a clean story about self-preference. It is the sign of a confounded metric.

The confound is generosity

Each grader has a scale. The metric above pairs a model’s rating of its own answer, on its own scale, against other graders’ ratings on theirs. So a generally generous grader looks self-favoring, and a generally harsh grader looks self-critical, before any real favoritism enters.

And the graders are not on the same scale at all:

GPT hands out about 92 to everything, so its own answers ride along at 92 even though the other two judges rate them near 82. That is not self-love, it is a model that likes every answer. Opus rates everything low, and its own answers happen to be the ones the other graders rate highest of the three, so on Opus’s stingy scale its own work looks underrated. The metric was measuring generosity wearing a self-preference costume.

The obvious fix, and why it is also wrong

The obvious fix is to stay inside each grader’s own scale. Compare what a grader gives its own answers to what that same grader gives the others. Generosity cancels, because every number in the comparison is on that one grader’s scale. This is what I originally shipped, and it produces a clean-looking table:

The order reverses completely, and I read that reversal as the punch line: Opus, the harsh grader, turns out to be the most self-favoring, and GPT the least. It is a tidy story. It is also wrong, and it is wrong in a way that is easy to miss.

Look at what that gap actually compares. “Rates its own” is Opus grading Opus’s ten answers. “Rates others” is Opus grading the other twenty answers. Those are different answers. The metric only reads as favoritism if you assume the two sets are equal in quality, so any gap must be bias. But they are not equal. The independent judges, the ones with no stake, rate Opus’s own answers highest of the three families and GPT’s lowest. Opus’s answers really are better on average. So when Opus scores its own work eight points above the rest, part of that gap is favoritism and part is that its own work is genuinely stronger, and the within-grader metric cannot tell the two apart. It trades the generosity confound for a quality confound.

Hold the answer fixed, and the ranking settles

There is a comparison that does not have this problem. For every one of the thirty answers I already have three ratings, one from each grader. So instead of comparing a grader’s ratings across different answers, compare the three ratings of the same answer. The producer’s own rating minus the other graders’ ratings of that identical text differences the answer’s quality out exactly, because it is one fixed piece of writing. What is left is the producer’s general leniency plus real favoritism, and the leniency I can remove with a grader fixed effect. That is a two-way model: answer quality held fixed by comparing the same answer, grader generosity held fixed by a per-grader term, favoritism read off the “grades its own” indicator. I fit it by ordinary least squares over all ninety ratings.

Once quality is held fixed, GPT-5.5 is the most self-favoring by a clear margin. It hands its own answers scores the independent judges do not agree with; that gap is favoritism, and the within-grader metric hid it because GPT is generous to everyone, so its self-scores never stood out against its own scale. Opus lands in the middle, its favoritism real but roughly a third of the eight points the within-grader table credited it, because most of that eight-point gap was Opus’s answers simply being better. Gemini is indistinguishable from zero. The aggregate is still +3.47, the same number all three ways; it is one signal, and only how you attribute it to a model changes. But the clean attribution is the two-way one, and it does not match the ranking I first published.

What it says, narrow and honest

Two things, one that holds and one that I got wrong the first time.

What holds: the rule the series rests on is justified, and this reanalysis makes that stronger, not weaker. Across all thirty answers the aggregate self-preference is +3.47 and clear of zero, and on the clean per-model estimate GPT favors its own work by nearly eight points, Opus by a few, Gemini by a sliver. A model grading its own output is not neutral. Keep the producer out of the grading. That is the load-bearing claim, and it survives intact.

What I got wrong: which model is the worst offender, and the confident way I said it. I ran two metrics. The naive one confounds favoritism with generosity and pointed at GPT for the wrong reason. The within-grader fix cancels generosity but confounds favoritism with quality, and it flipped the ranking onto Opus, and I called that flip the answer. It was not. Both metrics move the guilt around for reasons that have nothing to do with favoritism, and it took a third estimate, one that holds both quality and generosity fixed, to get a ranking I trust. That is the whole point of the series in miniature, sharper than I meant it to be: the honest number is rarely the first formula the obvious framing hands you, and sometimes it is not the second either.

Honest limits

Ten prompts, one run, short answers; the scores are each grader’s own 0-100 self-report, so the gaps and their direction are the signal, not the absolute levels. “Blind” is not identity-proof, a model may recognize its own style, but that is a channel for favoritism, not a confound to scrub out. The two-way estimate assumes the favoritism sits on top of a grader’s scale additively; with three models and ten prompts the intervals are wide. GPT-5.5 is clearly the highest and clearly above zero, Opus is positive but its interval nearly touches zero, and Gemini is not separable from zero at all, so treat the ordering below GPT as soft. Three production models, one snapshot in time.

Method, for anyone who wants to check it. Three models through their normal command-line tools, $0, no API keys. Pre-registered before data, produced and blind-graded by code, scored by a deterministic aggregation, and rated before publishing by the two families that did not write it. The pre-registration, data, and analysis are in the experiment directory.

This is part of a series on how AI models behave when you measure them honestly. The others, and the through-line, are on the series page.