Listing the Fish: Why Interpretability Might Just Be Biology

Listing all the fish

A long time ago I worked in a freshwater aquatic biology lab, and one of our jobs was to list the fish. All of them. Every known freshwater species in the twenty largest lakes in the world, pulled out of books, research papers, and whatever other scientific documents we could get, and entered one at a time: scientific name, depth, everything like that.

It was exhausting and it was not glamorous. Nobody writes a grant proposal that says “we will type in fish.” But the catalog is what made everything downstream possible. You cannot study how a lake ecosystem responds to warming if you do not know what lives in it.

“There are plenty of fish in the sea” is an idiom because the number feels infinite. It is not. It is large, finite, and knowable, and the way you come to know it is one fish at a time.

I think the same thing is about to be true of neurons in neural networks, and a new essay, Neural Networks Are Already Clean, is the best argument I have seen for it.

What Neural Networks Are Already Clean actually claims

Nick Cammarata's essay, Neural Networks Are Already Clean, argues that networks are far more interpretable than the field has assumed. For years the working theory was superposition: networks have more concepts than neurons, so they cram unrelated features into shared neurons, and you have to untangle that mess with a decomposition method before you can read anything. The essay says the mess was mostly real but the diagnosis was wrong.

The evidence runs in three parts. First, the canonical superposition example, a 2019 circuit where a “dog” neuron appeared to secretly carry a “car” feature, turned out to be a shared pose detector, and no second real superposition circuit has been published since. Second, the “uninterpretable” last layer of InceptionV1 went from roughly 5% readable with 2023-era tools to about 99% in a modern ConvNeXt of the same size, with no change to the data. Residual connections, LayerNorm, and better feature visualization did the work.

Third, and most relevant to anyone shipping language models, Cammarata audited neurons in Qwen3-32B directly. His estimate: about 90% of neuron poles have one main job, about 7% have two related jobs, and only 2 to 4% are arbitrary mixtures. He is careful to say those figures depend on thresholds and definitions.

The mechanism he proposes is what he calls recursive generalization. A neuron that fires on “iy” grows to fire on “diy,” then on “do it yourself,” which contains no “iy” at all. Neurons grow outward from a core in locally sensible steps. What looks like a neuron with several unrelated jobs is usually one job you have not yet traced back to its origin.

His conclusion is a bitter lesson for interpretability: rather than engineering ever-cleverer decompositions, scale up ordinary investigation of the network as it stands.

This is ichthyology

If Cammarata is right, interpretability stops being a methods problem and becomes a cataloguing problem. That is a very different kind of science, and it is one biology has been doing for three hundred years.

Consider what we were really doing in that lab. We did not have a theory that generated the fish. We had a population that was finite, structured, and knowable only by direct observation. Each species had a range, a depth, a diet, a set of relatives. The relationships only became visible once enough individual entries existed to compare. The structure of a lake came out of the list, slowly, as it filled in.

Neurons in a large model look the same. A 32-billion-parameter model has a finite number of them. Each one, per the audit in Neural Networks Are Already Clean, mostly has one job, a range of inputs it responds to, a set of upstream neurons it depends on, and downstream neurons it feeds. That is a species description. The recursive generalization story even gives you a phylogeny: you can watch a neuron's job grow over training the way a taxonomist traces how a genus radiated across a lake system.

The tools map over too. A neuron's activation record plays the part of the pinned specimen. The rubric that decides whether a neuron has one job or two is a dichotomous key. The canonical example that fixes what a neuron means works like the type specimen in a museum drawer, and the catalog, cross-referenced and shared across institutions, is what makes the next researcher faster than the last.

Cammarata makes this point himself when he says understanding compounds: each explained neuron constrains the explanation of its neighbors. That is exactly how a species list works. The thousandth fish is easier to place than the tenth, because you already know what family it cannot belong to.

The network, in other words, may be a great deal more biological than the engineering framing ever suggested. We built it, but we did not design its internals any more than we designed a lake. We set the conditions and let it grow. The natural way to study a thing that grew is natural history.

What a natural-history program looks like

If you take the analogy seriously,

The unit of output becomes the entry, not the paper. Biology's great cataloguing projects were not individual publications; they were shared databases that thousands of people contributed to over decades. Interpretability would need the same thing: a per-model atlas where each neuron has a stable identifier, a job description, canonical examples, upstream and downstream connections, and a confidence grade. The deliverable for a frontier lab would be the atlas for its production model.

Where the analogy strains

How it breaks down. Fish do not change when you look at them. Neurons are only meaningful in the context of a specific set of weights, and a fine-tune or a retrain reshuffles the catalog. Our lake list was good for decades. A neuron atlas might be good for one model version. The economics of cataloguing depend heavily on how much of the atlas survives a new training run, and nobody knows that yet.

The catalog has a scale problem biology never faced. There are roughly 35,000 known fish species. A frontier model has tens of millions of neurons across its layers, and the industry is heading toward trillions of parameters. At that scale the cataloguers will themselves have to be models, which means the audit inherits whatever blind spots the auditing model has.

Neurons may not be the right unit. The numbers in Neural Networks Are Already Clean are about individual neurons in MLP layers. A lot of what matters in a transformer lives in the residual stream and in attention heads, where superposition arguments have always been strongest.

The cleanliness number is a judgment call. Cammarata says so himself: 90% with one main job depends on where you set the threshold and how generous you are with “related.”

A 2 to 4% residue in a 32B model is still tens of thousands of neurons, and there is no guarantee the dangerous behavior lives in the clean 90% rather than the messy tail.

Why this matters for agent control

At Clear Harness we think about controlling AI agents from the outside: what they are allowed to touch, what they did, and whether what they did matches what they were asked. That is necessary work whether or not the inside of the model is ever legible.

But the two approaches are not in competition, and the result in Neural Networks Are Already Clean makes them complementary in a concrete way. External control tells you what an agent did. A neuron atlas would tell you why, and eventually it would let you check the why before the agent acts rather than after. Mature safety disciplines have both. Aviation has the flight recorder and the maintenance manual. Medicine has the chart and the anatomy textbook.

It will arrive the unglamorous way, entry by entry, the way field guides did. The teams that will benefit most are the ones already instrumenting agent behavior from the outside, like Clear Harness, because we are the ones who will have the behavioral data to line up against the neurons.

Sources