An SAE labels every feature it finds. On a fair test, the labels hold up about 40% of the time.

The biggest current bet in interpretability is the sparse autoencoder. Run one on a model’s activations and it decomposes them into thousands of sparse “features”, each meant to stand for a single human concept. Then an automated pass writes each feature a label from its top-activating examples, “legal terminology”, “the Golden Gate Bridge”, “scanning devices”, and the promise is that you can now read off what the model is thinking in plain English. The labels are everywhere, on Neuronpedia, in papers, in dashboards. The question almost nobody puts to them directly is whether they are true.

They are easy to over-trust for a specific reason: the label is written from the feature’s very top examples and then, usually, judged on those same examples. That is grading your own homework. The honest test is to take the label to fresh text the feature fires on, and to a skeptic who did not write it, and ask: does this feature actually fire on its stated concept, and on that and not much else? So I ran exactly that, on a laptop, for free.

The setup

The widely used gpt2-small-res-jb sparse autoencoder, 24,576 features on the residual stream of GPT-2-small. I ran a held-out 100,000-token slice of varied text (the Pile) through the model and the SAE, and sampled 80 features that were active and carried a Neuronpedia label, stratified across the full range of how often features fire so the sample is not just the busiest ones. For each feature I pulled the 8 contexts where it fired hardest, plus 8 control contexts where it did not fire at all, shuffled them together, and handed them to two judges from different model families than the one that wrote the labels. Blind to which contexts were which, they rated each: does the marked token, in this context, match the feature’s label? Yes, no, or unclear.

The controls are the key piece. If a judge will call a vague label a “match” for any old text, the control contexts catch it. So the headline number is corrected precision: how often the firing contexts match the label, minus how often the controls do. That subtracts away the rate at which a label-agnostic reader would say yes anyway.

The labels are right about 40% of the time

Across the 80 features, corrected precision averaged 0.38, with a bootstrap interval of 0.30 to 0.46. The control rate was 0.016, almost zero, so the features are not noise: they fire on their concept far more than on random text. But a feature’s top activations match its own label only about two times in five. By the obvious bar, a feature counts as faithful if a clear majority of its top firings match its label, 41% of the labels passed. Loosen the bar all the way down and it barely helps: even requiring just 30% of firings to match, only 54% of labels clear it. Tighten it to 70% and one in five survives.

The average hides a split. Some features are exactly what they say. The feature labeled “scanning or scanning devices” fires on scanning, scans, MRI, scanner, cleanly, and on nothing in its controls, for a perfect corrected precision of 1.0, and a handful of other features score 1.0 too. A “camping” feature is faithful but looser, firing on camp, camping, campsite at a corrected precision around 0.56. These are real, monosemantic, and correctly named. The other tail is where the trouble is. A feature labeled “phrases related to entertainment such as movie titles” has, as its single strongest activation in 100,000 tokens, the phrase “Rotary Microtome latest Spencer”, a piece of lab equipment. A feature labeled as a reference to “Helvetia, which could be a reference to Switzerland” turns out to fire on the letter-string “hel” wherever it appears, in the Norwegian word helg, in Heliospheric, in helms, and never once on Switzerland. A feature labeled “personal reflections or anecdotes” mostly fires on whitespace and sentence-boundary tokens. For these the label describes the feature’s actual behavior no better than random text does, and sometimes worse: the movie-titles and personal-reflections features both land below zero, a negative corrected precision, while the Helvetia feature sits right at the chance baseline, matching its label no more often than random tokens would. Several features across the tail scored below zero.

Why you can believe the audit

The weak point of a test like this is the judge, so I pinned it down three ways. The two judges, from different families, agreed with each other on 90% of the 1,280 context calls. I also hand-labeled 64 contexts myself, blind to firing-versus-control, before seeing the aggregate, and the judges agreed with me on 91% of those. And the near-zero control rate means the judges were genuinely discriminating, not waving everything through. This is the same rule the rest of this series runs on, the thing that makes a claim is never allowed to grade it: the auto-interp label is the claim, and a different model on held-out activations, checked against my own hand labels and a control baseline, is the grader.

I had pre-registered the bet that fewer than 60% of labels would be faithful and that they would beat chance but leak. Both held: 41% faithful, well above the 0.016 floor, well below clean.

What this does and does not show

The honest scope, because it is easy to overread in both directions. This is one SAE, one layer of one small model, one label source, and the main number above is precision, whether a feature’s firings match its label, not recall, whether the feature fires on every instance of the concept. The judges are models, mitigated by using two families, a human gold check, and a control, but still models.

I did attempt recall as a follow-up, and the honest report is that it is much harder to measure than precision. For the faithful features, I searched a fresh held-out corpus for natural instances of each concept (excluding the feature’s own trigger words, so the test is whether it fires on the concept expressed in other words, not just its trigger), and had two judges confirm the genuine instances. Most features turned up too few confirmed instances to measure, so by the pre-registration this result is exploratory, not a number I will stand behind. The signal it did show points the same way as the precision tail: a handful of features genuinely generalize, firing hard on their concept in fresh wording, but most behave like narrow detectors of one trigger token and fire on almost none of the same concept written differently. If that holds up under denser retrieval and a bigger corpus, which is the real next step, it means even the labels that pass precision are often broader than the feature: the feature is a word detector wearing a concept’s name.

What it does show is narrow and, I think, important to say plainly. It is not that sparse autoencoders do not work: the features fire on their concepts far above chance, and four in ten of these labels are genuinely accurate. It is that the human-readable label sitting next to a feature is right less than half the time on held-out text, and a real fraction of labels describe their feature worse than a coin flip would. If you read SAE feature labels the way the field invites you to, as what the feature means, you will be wrong most of the time, and the labels give you no signal about which times. The thing that tells you is the boring one: take the feature to fresh text it fires on, and check.

That is the same shape as the other notes here. The induction-head detector found the heads but not their importance; the behavioral notes kept finding the obvious measurement and the real one coming apart. Here the obvious measurement is the label, and the real one is what the feature does when you are not looking at the examples that named it.

Method, for anyone who wants to check it. GPT-2-small and the gpt2-small-res-jb SAE through TransformerLens and sae_lens, run locally, $0, no API. A held-out 100k-token Pile slice; 80 labeled active features sampled stratified by firing frequency; per feature 8 top-firing contexts plus 8 non-firing controls, judged blind by two different-family models (yes/no/unclear), corrected precision = firing-match minus control-match. Validated against a 64-context human gold set (91% agreement) and inter-judge agreement (90%). The exploratory recall pass (its own pre-registered design: keyword retrieval over a fresh corpus with the feature’s trigger words removed, two-judge confirmation, firing measured against each feature’s own median activation, and the underpowered-by-design outcome) is in the same directory. The pre-registration (with the bar set before judging), the code, the labels, and the full per-feature table are in the experiment directory.

This is the second white-box note in a series on how AI models behave when you measure them honestly. The others, and the through-line, are on the trutina landing page.