←

I audited interpretability’s three favorite tools on one small model. Each gives a clean, confident, partial answer.

Most of the studies in this series treat an AI model as a black box: prompt in, behavior out, which is all the commercial models let you see. This thread does the opposite. It takes GPT-2-small, old and small enough to run on a laptop with every weight exposed, and turns each of interpretability’s standard tools on it to ask a simple question: when the tool tells you what is going on inside the model, is it right?

Three tools, three notes. Each one finds something genuinely real. And each one, looked at carefully, turns out to answer a question slightly different from the one the field trusts it to answer. It is the same gap three times, and it is worth seeing together.

Tool one: the attention score for finding induction heads

Induction heads are the parts that let a model continue a repeated pattern (having seen “…Mary… Mary”, predict that the next token after a later “Mary” repeats what followed last time). The standard way to find them is an attention score: does the head attend back to the token that previously followed the current one? In the first note the score works as advertised for finding them. But it is a bad guide to which one matters: its ranking correlates just 0.14 with what actually happens when you ablate each head. It over-credits a redundant cluster no single member is needed for, and misses the first-layer head with the largest causal effect. The tool measures attention. You wanted importance. Those are not the same thing.

Tool two: the labels on sparse-autoencoder features

The sparse autoencoder is interpretability’s current flagship. It breaks a model’s activations into thousands of features, each given a human-readable label like “scanning devices” or “the Golden Gate Bridge”, and the promise is that you can read the model’s mind in plain English. In the second note I took 80 of those labels to held-out text the feature fires on and asked two independent judges, plus my own hand-labels, whether each firing actually matched its label. Corrected for a control baseline, the answer was about 40% (corrected precision 0.38 against a near-zero control). Only 41% of labels were faithful by a majority bar. Some features were perfect; a real tail were confabulated, a “movie titles” feature whose top activation is a piece of lab equipment. The label is written from a feature’s top examples and trusted as its meaning. On fresh text it is right less than half the time.

Tool three: measuring which head matters in a known circuit

The indirect-object-identification circuit is the textbook “we reverse-engineered the gears” result, with heads assigned named jobs. In the third note I confirmed the circuit is real, and then measured each head’s importance two standard ways: ablation (turn it off, see what breaks) and direct attribution (read what it writes to the answer). The two nearly invert. One name-mover head ranks 2nd by what it writes and 141st of 144 by what breaks when you remove it. And turn off all three heads that write the answer at once, and the answer barely changes, because backup heads wake up and repair the damage. There is no single fact “how much this head matters”. There is one number per measure, and the measures disagree.

It is the same gap every time

Line the three up and the pattern is hard to miss. Each tool measures something that is correlated with what you care about, and gets trusted as if it were the thing itself:

None of these tools is broken. They find real heads, real features, real circuits. The error is in the read-off: treating a proxy as the target, and reporting the clean confident number the proxy gives without checking it against the thing it stands in for. And the fix is the same boring move each time. Take the claim somewhere it was not fitted, held-out text, a second measure, a direct causal test, a judge who did not write the label, and see if it survives.

Which is exactly what the black-box half of this series kept finding from the outside. There, the obvious metric and the real one kept coming apart once you controlled properly: the firmest refuser was the worst calibrated, the model that “favors its own answers” flipped once you corrected for how generously each one grades, the folk belief that a model “takes your side” inverted on a fair test. Same discipline, inside and out: a claim is never allowed to grade itself, you measure the proxy and the target both, and you ship the gap between them.

One more thing, since this series is as much about the method as the models. The honesty loop that is supposed to catch the models caught me three times before these notes went up. It flagged a head I had confidently mislabeled “the previous-token head” (it was not), an off-by-one in how I reported head rankings, and a sign error in the IOI design that a decorrelated reviewer caught before any data was collected. The cross-check is not a courtesy you extend to the model. It is the thing standing between you and a clean, confident, wrong result, and you are not exempt from it.

What to actually take from this

Not that interpretability does not work. On a laptop, for free, these tools found real induction heads, real monosemantic features, and a real circuit, which is remarkable. The narrower and more useful takeaway is for anyone reading or producing interpretability claims: a single tool’s verdict on what a part does or how much it matters is a strong lead, not a conclusion. The tools are good at proposing. They are weak at adjudicating, and they do not tell you which of their answers is the partial one. Triangulate before you believe. The cross-check is cheap, and as all three of these notes show, it is where the actual finding lives.

The scope, plainly: one small model, one instance of each tool, the simplest honest version of each audit. None of it is the last word on its tool. But the gap shows up so cleanly, and in the same shape, in all three that I would be surprised to find it absent at scale.

The three studies, with full method, pre-registration, and code: induction heads · SAE feature faithfulness · the IOI circuit. All run locally on GPT-2-small via TransformerLens and sae_lens, $0, no API. The black-box half of the series measures the same kind of gap from the outside, on three production models.