Lab notebook · Kevin Madson

Notes on machines that grade themselves.

I run small, pre-registered experiments on how AI models behave, mostly Claude, and write down what happened. That includes the bets I lost.

they shouldn't. that is the whole point

Try it: who catches the bugs?

Claude alone: 23 of 30 bugs caught

Gemini alone caught 27 and Codex 21. All three together caught all 30.

20 code snippets with 30 planted defects, reviewed blind by three AI models. Switch reviewers on and off to see what each one misses. From the blind spots study, June 2026.

The notes

New, November 2026: What changed since July. Three of these experiments, run again on four current Claude models. A green underline on a card means its main claim was retested and held. Red means it was revised.

Crossed out

Things I believed, and what the data said instead. Corrections stay on the page.

  1. Claude Opus is the grader that most favors its own work.

    GPT-5.5 is, once answer quality is held fixed: +7.6 points against +2.7.

    Corrected July 10, 2026 after a reanalysis. Self-preference study

  2. Three different AI reviewers will catch far more than the best one alone.

    Only 1.1 times as many. Under the bar I set in advance.

    Pre-registered June 13, 2026. Blind spots study

  3. My write-ups say exactly what the data says.

    An audit of 620 claims found 18 errors. All 18 are fixed.

    Full audit of every published note, July 10, 2026.

Lab rules

Five rules every experiment here follows. If a team adopts nothing else, adopt these.

  • Nothing grades its own output. The reviewer is a different model, and the harness decides who it is.
  • Every score cites evidence. A grade with no fact behind it gets lowered until it has one.
  • Cheap checks run first. Plain, model-free tests run on every change. Expensive judges only look at what is new.
  • A human outranks the automation. A person checks what the machines cannot, and wins any disagreement.
  • Say what was not measured. "Estimate" and "one run" stay attached to the number.

Read the field guide