Lab notebook · Kevin Madson
Notes on machines that grade themselves.
I run small, pre-registered experiments on how AI models behave, mostly Claude, and write down what happened. That includes the bets I lost.
they shouldn't. that is the whole point
Try it: who catches the bugs?
Claude alone: 23 of 30 bugs caught
Gemini alone caught 27 and Codex 21. All three together caught all 30.
The notes
New, November 2026: What changed since July. Three of these experiments, run again on four current Claude models. A green underline on a card means its main claim was retested and held. Red means it was revised.
-
harness
An honest eval harness, and ten studies of model behavior
An eval harness where the model that did the work never judges it, plus ten studies of how production models behave.
code -
experiment
Do AI code reviewers have complementary blind spots?
Three AI reviewers together caught all 30 planted bugs. The best one alone caught 27.
code -
field guide
How to know if your AI is actually working
The five rules I would hand a team that adopts nothing else.
code -
safety
The gate that edits a gate is rejected by the gate it edits
A small gate that decides whether an agent may change its own rules. It only ever proposes, it never writes.
code -
experiment
Ten rounds of an agent improving one game
A 71-check test suite and what each extra round of AI polish actually costs.
code -
method
Rating your own work without lying to yourself
Two scores, craft and fit. A grade with no evidence behind it gets lowered until it has some.
code -
method
An eval framework where the doer never grades itself
A review loop where each reviewer's vote counts for as much trust as it has earned.
code -
side project
Audioscan: one decode pass instead of three ffmpeg shellouts
One decode pass instead of three separate ffmpeg calls.
code
Crossed out
Things I believed, and what the data said instead. Corrections stay on the page.
-
Claude Opus is the grader that most favors its own work.GPT-5.5 is, once answer quality is held fixed: +7.6 points against +2.7.
Corrected July 10, 2026 after a reanalysis. Self-preference study
-
Three different AI reviewers will catch far more than the best one alone.Only 1.1 times as many. Under the bar I set in advance.
Pre-registered June 13, 2026. Blind spots study
-
My write-ups say exactly what the data says.An audit of 620 claims found 18 errors. All 18 are fixed.
Full audit of every published note, July 10, 2026.
Lab rules
Five rules every experiment here follows. If a team adopts nothing else, adopt these.
- Nothing grades its own output. The reviewer is a different model, and the harness decides who it is.
- Every score cites evidence. A grade with no fact behind it gets lowered until it has one.
- Cheap checks run first. Plain, model-free tests run on every change. Expensive judges only look at what is new.
- A human outranks the automation. A person checks what the machines cannot, and wins any disagreement.
- Say what was not measured. "Estimate" and "one run" stay attached to the number.