Auditing an honest-eval tool with its own method

trutina is a small eval harness with one rule: a model never grades its own work. Every output is scored by judges from different model families, and the producer’s family is excluded by the harness. After I published it, I turned that rule on the tool itself, and a cross-family panel caught two real bugs that a same-family self-review had walked straight past. Here is what happened.

The setup

trutina takes a spec (a prompt under test, the model that runs it, a few cases, and a panel of judge models from different lineages) and produces a scorecard: a deterministic floor of cheap exact checks, then the decorrelated judges, then an honesty layer that reports how much the judges agree, refuses to round a score up, and names what it could not measure.

Experiment 1: self-review versus a decorrelated panel, on the tool itself

First I had a panel of Claude agents review the trutina codebase. Same family as the code’s author. They were careful and found a few real things, around three.

Then I ran the tool’s own rule: a cross-family panel with the author’s family excluded. GPT-5.5 and Gemini 3.1 Pro audited the whole codebase, and a free local Qwen-7B took a third pass file by file. They surfaced about fifteen issues, and two were real bugs in the honesty machinery that the self-review had walked past:

• Score inflation. A judge that failed to return valid JSON was dropped from the score average instead of counting as a zero. So one good judge plus one failed judge produced a flattering number rather than a penalized one. The function’s own docstring claimed it did not do this. It did.
• A producer-exclusion bypass. A producer label with a vendor prefix (think openrouter/anthropic/claude) did not resolve to a known family, so the rule that excludes the producer’s family quietly excluded nothing, and a same-family judge could grade the producer’s own output. For an honesty tool that is the worst possible bug, and it was reachable from an ordinary spec.

I fixed both, plus the smaller findings, and pinned each with a test. Then I re-ran the same cross-family rating on the fixed code. It rose from about 7.5 to 7.8, and the agreement between the two judges went from 0.90 to 0.96. By the tool’s own metric the panel is now near-unanimous, which is what makes the number worth trusting.

The bugs are not the point. The point is that the self-review rationalized them away and the decorrelated panel did not, which is the exact argument the tool exists to make, demonstrated on the tool.

Experiment 2: a cheap local judge is useful but soft

In a four-family run (Claude produced support-ticket replies; GPT-5.5, Gemini, and a free local Qwen-7B judged), the local model behaved like a credible panel member. It returned well-formed verdicts every time and agreed with the frontier judges most of the time, with panel agreement around 0.82.

The same run, as trutina renders it: the verdict hero, the cross-family agreement reading, and the “what was not measured” panel (this run had no blind spots to report, so it says so rather than staying silent). The per-case scores match the table below. Regenerate it with img/render-scorecard.sh.

Here are the per-judge scores behind that card:

The local 7B tracks the direction but is softer. The case that matters is the out-of-policy refund: the reply invented a “30-day” refund policy when the real window is 14 days. Gemini caught that factual error (its 5 names it explicitly); the local 7B did not, and scored a lenient 7.

I want to be careful here: this is one run over three cases, not a finding. There is no false-positive or false-negative rate, no human-labeled baseline, and the agreement number I am quoting is just mean pairwise score distance, which cannot tell genuine diversity from noise. What it does is motivate the design choice trutina ships: judge the bulk of your volume on the free local model, and escalate only the low-confidence cases to a frontier judge. Cheap for the easy ones, careful for the ones that matter.

What would make these real

To turn the second story from an anecdote into a result, I would measure judge agreement against human labels on a real set of cases, run a single-frontier-judge baseline against the panel to show the panel buys something, and replace the heuristic confidence thresholds with values that have an empirical basis. That work is not done, and the tool says so about itself, in the same “what was not measured” panel it prints for every run.

Takeaway

The honest version of “I built an eval tool” is “I built an eval tool, turned it on itself, it caught real bugs I had missed, and I can show the rating going up after I fixed them.” Decorrelated review beat self-review on the codebase that argues decorrelated review beats self-review.

Verify it: the four-family sample and its scorecard, and the audit fixes with a test pinning each one, are all in this repo.