What makes a polish round work, and how the experiment nearly fooled me

When you refine something in rounds, what actually makes the next pass better? I built a controlled experiment to find out. The honest method's first catch was a flaw in my own experiment. Its second was the answer I had bet on.

Kevin Madson · June 2026 · 7 min read

If someone forwarded this to you: a lot of my work runs through AI models that write and revise text and code. A “polish round” is one pass where you hand the draft back and ask for it better. The trick I lean on is decorrelated review: instead of letting the model that wrote something grade its own work (a student marking their own exam), a different family of model checks it, like a second opinion from a doctor who did not make the first diagnosis. This note is about turning that trick on the question of polishing itself, and on my own experiment.

github.com/KiwiMaddog2020/trutina kevinmadson@protonmail.com

The question was simple. When you refine something in rounds, what makes the next round improve the work at a real margin? Is it planning? A second opinion? Just trying again? I built a controlled experiment to answer it, and the first thing the honest method did was catch a confound in my own design that would have handed me a false answer. The second thing it did was overturn the answer I had bet on.

The setup

Hold the writer constant and vary only the process around it, so any difference traces to the process and not the model. A fixed model drafts a short technical explanation. Then one polish round is applied under five recipes: naive (“make it better”), self-critique then fix, a cross-family critic then fix, the same plus an explicit plan, and the same plus an honest verify gate that re-checks and only revises if the round genuinely improved things. Quality gets measured two ways. A deterministic floor: a checklist of planted, code-checkable criteria (did the explanation state the required facts, avoid the common misconception, stay in the length window), scored by code, not opinion. And a panel of judge models from other families, scoring the craft, with the family that wrote the text excluded from grading. This is the discipline of my eval tool, trutina, pointed at the act of improvement itself.

Act I: the honest method caught the experiment first

Before trusting a single number, I ran trutina’s core rule on the experiment’s own design: a panel of independent reviewers, each told to find the thing that would make the conclusion wrong. It came back with four problems serious enough to sink the headline.

Decorrelation was tangled up with critic count. My “decorrelated” arm used two critics; the self-critique arm used one. So any win could be “a second opinion of any kind,” or just twice the editing notes, rather than the cross-family independence I wanted to credit. The verify gate was also a judge: the model deciding whether a draft should be revised was the same family that then scored a third of the result, which is the exact “grading your own work” leak the tool exists to stop, moved one step downstream. There was no spend-matched control, so “the fancy recipe won” could just mean “the fancy recipe got more attempts.” And the honest gate I most wanted to test never actually fired, which made its arm a copy of the one before it.

None of that was visible from the inside. A self-review would have shipped the tangle and reported a tidy ranking. The cross-family review refused to. That is the whole argument of the tool, demonstrated on its author.

The wall, stated plainly

Midway through, the hosted judge budget ran out. The good judge models are pay-per-token, and the small free allowance was gone. Rather than pay to finish a design the audit had just shown was confounded, I rebuilt the one slice that stays rigorous for free: the deterministic floor. Its outcome is scored by code, not by a model, so a weak or biased judge cannot corrupt it. The price is scope. This measures objective adherence to a spec, not craft or accuracy. I come back to that.

I also fixed the audit’s first complaint head on, by making the decorrelation test count-matched. One same-family critic versus one cross-family critic isolates decorrelation at a count of one. Two versus two isolates it at a count of two. Now the only thing that changes between the compared arms is family independence. The writer was a small local model (Qwen 2.5 7B) running free on my laptop; the critics were Llama 3.1 8B and Mistral 7B.

Act II: the result, which refuted my prediction

I wrote my bet down before looking: decorrelation would be the main lift. Ten tasks, three rough drafts each, thirty paired runs.

Both real effects came from critic count, not decorrelation. The two contrasts that move each add a pass; the two that isolate family independence are null, and at two critics decorrelation is if anything slightly negative. A single critique of any kind was not reliably better than a plain rewrite. The count effect showed up in six of ten tasks, and it clustered exactly where there was room to improve: the hard explanations (herd immunity, p-values) moved, the easy ones already at ceiling did not.

My prediction was wrong, and in a useful way. The audit had warned that naive designs confuse “decorrelation helped” with “we reviewed twice.” Separate the two and the diversity I bet on evaporates. The plain number of independent passes was carrying the whole effect.

Act III: does the next round keep improving? Mostly no

That is about how many critics to use in one round. The other half of the question is across rounds. If one good round helps, do two or three help more? I ran the winning recipe (two cross-family critics) and a naive rewrite for three sequential rounds, each polishing the last.

The gain is front-loaded. With the good recipe, round one delivered a large, real lift (+0.129, interval [+0.07, +0.19], p=0.003). Round two added nothing and was slightly negative (-0.064). Round three was flat. The three-round total (+0.053) came out smaller than round one alone, because the later rounds ate into it: once the checklist items are covered, a second editing pass mostly shuffles satisfied points around and occasionally drops one. The first coat of paint does the work; the next two move paint. The naive recipe never produced a real round at all.

So on objective adherence, the honest answer to “what keeps the next round improving” is: not much. One good round captures the gain. The lever is the quality of that first round, not the count of rounds.

What this does and does not say

It says that for a polish round to improve the work at a real margin on objective, checkable criteria, the highest-leverage move here was another independent critique pass in the first round. Not a more diverse reviewer. Not (a separate arm, still pending a funded run) an explicit plan. And not a second or third round.

It does not say decorrelation is useless. The floor checks whether required points are present, which is a checklist, not judgment. Decorrelation’s real payoff lives on the axis this free run could not measure: catching factual errors and hallucinations that a same-family reviewer is blind to in exactly the way the writer was. A separate trutina run found precisely that on a different task, where two frontier judges caught a model inventing a refund policy. That axis needs paid frontier judges, and it is the next run.

The honest caveats, all of them. Verbosity is a partial confound: more critique means more notes means a longer rewrite that trips more checklist items. The correlation is about 0.36, the same inside the headline contrast, real but far from the 0.8 a pure-length story would need, and the winning arm beat the wordier one (it added 39.5 words and still won; the loser added 54.2). The writer was one small local model with small-model critics, so the result is scoped to that. The sample is thirty runs. The floor is a proxy for adherence, not quality. The cross-model refutation I wanted from a second engine could not run headless that night, so the check here is statistical, not a second model’s read.

The takeaway

Two things, one narrow and one general.

Narrow: in this experiment the secret sauce of a productive polish round was the number of independent looks in one good first round. Not their family diversity. Not a planning step. Not the number of rounds. Measured where improvement is objectively checkable.

General: when you find that a decorrelated review helped, control for how many review passes you ran before you credit the diversity. You may be paying for variety and getting the benefit of simply doing the work twice. The honest way to know is to hold the count fixed and vary only the thing you think matters, and to let an independent reviewer check your design before you trust your own numbers. Mine caught me twice.

The corrected, pre-registered protocol and the frontier-graded run that would test the craft axis are written up and ready.

Verify it. The run, the per-run data, and the analysis are in the experiment directory: the frozen numbers in RESULT.md, the paired statistics in analyze_floor.py and analyze_rounds.py, and the corrected PROTOCOL_V2.md. The companion note, where I first turned trutina’s rule on trutina’s own code, is here. The three-experiment arc on whether a decorrelated judge panel actually helps, which lands on “mostly competence, not diversity,” is here.