In GPT-2’s most famous circuit, the head that writes the answer ranks 141st when you turn it off
If you have read one mechanistic-interpretability result, it is probably this one. For a sentence like “When Mary and John went to the store, John gave a drink to ___”, GPT-2-small reliably says “Mary”, the name that appeared once. Wang et al. reverse-engineered the circuit that does it (indirect-object identification, IOI) and gave its parts job titles: name mover heads copy the right name to the output, negative name mover heads push the other way, S-inhibition heads suppress the repeated name, and induction, duplicate-token, and previous-token heads feed them upstream. It is the canonical “we opened the model and found the gears” story.
I wanted to know if the gears survive a direct causal audit, the same lens that, in an earlier note, showed the standard induction-head detector barely predicts which induction head matters. So I measured every head’s importance to IOI two ways that the field treats as roughly interchangeable, and they turned out to tell almost opposite stories.
The setup
256 IOI sentences, balanced across both name orders, with single-token names so positions line up. The metric is the logit difference: how much more logit the model gives the correct name (the indirect object) than the repeated one. Clean, GPT-2-small scores +3.89 (interval +3.70 to +4.08) and gets the right name 99.6% of the time. The circuit works.
Then, for all 144 heads, two importance measures:
- Ablation (total effect): turn the head off (replace its output with its average) and see how far the logit difference falls. The intuitive “what breaks if I remove this part”.
- Direct attribution (direct effect): read what the head writes straight to the answer, its contribution along the direct path to the logits. The intuitive “what is this part actually saying”.
The circuit is real
First, the good news for the published story, because it is real and worth stating plainly. Every one of the top 10 heads by ablation is a named circuit head. The three name movers are the top three heads by direct attribution, exactly their job. And the sharpest thing I pre-registered, that the two negative name movers (10.7 and 11.10) would have negative effect, holds perfectly: they rank 144th and 143rd of 144 on both measures, the only heads that clearly hurt the correct answer. A blind guess would not put them there. The circuit Wang et al. described is genuinely in the model.
But “which head matters” depends entirely on how you ask
Now the part that should give anyone pause who reads a single importance number off a head. Here are five circuit heads with their rank (of 144) under each measure:
| head | role | direct-effect rank | ablation rank |
|---|---|---|---|
| 9.9 | name mover | 1 | 8 |
| 9.6 | name mover | 2 | 141 |
| 10.0 | name mover | 3 | 15 |
| 8.6 | S-inhibition | 139 | 1 |
| 5.5 | induction | 34 | 2 |
Read across the rows. Head 9.6 writes the second-largest direct contribution to the answer in the entire model, and ranks 141st of 144 when you ablate it (turning it off actually nudges the answer up). Head 8.6 is the single most important head by ablation and 139th by direct attribution: it matters enormously and writes essentially nothing to the answer itself. The name movers dominate direct attribution and scatter under ablation; the S-inhibition and induction heads dominate ablation and vanish under direct attribution. The two measures do not just disagree at the margin. They nearly invert each other.
There is a clean reason, and it is not that one measure is wrong. Direct attribution finds the heads that write the answer (the name movers). Ablation finds the heads the answer depends on, which are the upstream heads (S-inhibition, induction) feeding the writers, plus it is confounded by what comes next.
Turn off all three heads that write the answer, and the answer survives
Here is the confound, and it is the most interesting result. The three name movers produce nearly all of the direct signal for the correct name. So ablate all three at once and the logit difference should collapse. It does not move: the three heads’ individual ablation effects sum to +0.20, but ablating all three together changes the logit difference by -0.20 (very slightly up), against a clean +3.89. Removing all the writers at once does even less than the small things removing them one at a time suggested.
The model repairs itself. When the primary name movers are removed, other heads (the documented backup name movers) wake up and do the job instead. That is why head 9.9 can write the single largest direct contribution to the answer yet barely register when ablated: knock it out and a backup covers the gap. Ablation, the most intuitive importance test there is, systematically understates exactly the heads doing the most visible work, because the model routes around the damage.
What this does and does not mean
It is not a debunking. The IOI circuit is real, the roles are right, and the negative-name-mover prediction landing on both measures is a genuine, specific confirmation. What the audit shows is narrower and, I think, more useful: a head does not have an importance, it has an importance per measure, and two standard measures the field reaches for can rank the same head 1st and 141st. If you characterize a circuit with one of them, or with an attention pattern that looks like one of them, you will get a confident, clean, and partial picture, and self-repair means even “I turned it off and nothing happened” is not the safe test it sounds like.
The honest scope. Mean-ablation takes a head off its natural distribution and measures total rather than direct effect; direct attribution only sees the final linear write and misses effects routed through later layers; neither is the iterative path-patching the original used, so this is an audit of the published circuit, not a re-derivation of it. One model, one task. And one of my own predictions was too weak in a telling way: I expected the name movers to be individually modest but to “jointly matter”, and the self-repair was stronger than that, even jointly they barely move the output.
It is the same lesson as the rest of this series, now from the inside. The induction detector found the heads but not their importance. The SAE labels named the features but did not match them. Here the circuit is correctly drawn, and still, the simplest question you can ask of it, which part matters, does not have one answer.
Method, for anyone who wants to check it. GPT-2-small via TransformerLens, run locally, $0, no API. 256 single-token ABBA/BABA IOI prompts balanced over name order; metric is final-token logit difference (IO minus S). Total effect = change in logit difference under per-head mean-ablation; direct effect = per-head direct logit attribution onto the IO-minus-S direction at the final position; joint ablation over the three name-mover heads. The pre-registration (predictions and the decorrelated methods audit that added the direct-effect measure and caught a sign error, both before any data), the code, and the full 144-head table are in the experiment directory.
This is the third white-box note in a series on how AI models behave when you measure them honestly. The others, and the through-line, are on the trutina landing page.