A companion to the MIT Technology Review piece. This covers the essay itself, then the implications for cybersecurity.
One caveat first: the page was cut off partway through the language-model section, so I haven’t read the closing “Zooming Out” part or the appendices. This review covers roughly the first two-thirds of the essay.
What the essay argues
Nick Cammarata calls the piece an informal research essay and working hypothesis. Its core claim is that the field misdiagnosed why neurons in neural networks so often look like messy mixtures of unrelated things (“polysemanticity”).
The dominant explanation was superposition: when a network runs out of room, it supposedly stores unrelated concepts in the same neurons because they rarely collide. If that were true, a neuron’s grouping would tell you almost nothing, and you would have to “unfold” the network first, usually by training a second model such as a sparse autoencoder (SAE).
Cammarata argues the evidence behind superposition was misread, and that several ordinary problems explain it:
- Weak tools. Early visualization methods were “reward hacked,” producing misleading pictures. The famous car-into-dog circuit, for example, turned out to reflect shared pose, not hidden storage.
- Bad architecture and training. InceptionV1’s last layer had almost no interpretable neurons. Retraining with standard modern improvements (residual connections, LayerNorm) made 99–100% of last-layer neurons easy to interpret, with no change in model size or data.
- Recursive generalization. Neurons grow branches over training, as with an “iy” neuron that expands to “diy” and then “do it yourself.” That is ordinary generalization, just fractal, and it can be followed.
Further results he reports:
- Squeeze tests. Cutting a layer from 480 neurons to 10 produced coarser neurons (“curves & loops”), not overlapping mixtures. An SAE trained on the squeezed layer didn’t recover the original clean curve detectors.
- SAE comparisons. SAEs didn’t reveal building blocks the neurons lacked. On one last layer, 88% of SAE activity was near-copies of single neurons.
- Memorization. When he trained a model to call an umbrella photo a “bee,” it rerouted through existing umbrella and bee circuits in the last few blocks. It did not create an isolated hidden feature.
- Language models. In preliminary audits of Qwen3-32B, he estimates about 90% of neuron poles have one main job, about 7% have two related jobs, and 2–4% are not yet understood. He reports similar results up to trillion-parameter models.
His conclusion is that studying a network’s own neurons and circuits directly is viable, and that understanding compounds like a crossword as the map fills in.
Assessment
Strengths
- The hypotheses are testable. Superposition and generalization make nearly opposite predictions, and he watches neurons form across training checkpoints.
- The essay is candid about thresholds and tooling. The “one job” percentage depends on how you count weak activations and what you call “one thing.”
- It returns to the field’s most famous examples, which is persuasive precisely because the author helped create them.
- The ablation results are concrete, such as removing the weakest 75% of activations barely changing accuracy.
Weaknesses and open questions
- Much of the rigorous evidence comes from image models, many of them small and some trained by the author. The language-model claims are explicitly preliminary, pending a companion paper.
- The essay itself flags the risk of an “explanation of the gaps”: assuming every neuron has a sensible story. His answer is that each alternative cause is testable, but that depends on his own judgment calls and agent-assisted audits.
- Understanding most neurons does not mean understanding the system. As the critics quoted by MIT Technology Review noted, circuits and combinations of neurons are the harder part. Cammarata concedes this.
- Goodfire is an interpretability company, so readers should treat the optimism accordingly, and wait for independent replication.
What it means for cybersecurity professionals
This section is my analysis, not claims from the essay. It rests on a preliminary hypothesis, so treat it as scenario planning, not guidance.
Potential defensive upside
- Backdoor and trojan detection. The umbrella-to-bee experiment suggests that a learned override may show up as a reroute through existing late-layer circuits, not as an isolated hidden component. If that generalizes, which is unproven, model auditing could look for anomalous routing. It is a single small vision experiment, so it is a lead, not a method.
- Model supply-chain assurance. Organizations downloading open-weight models currently rely on behavioral testing. Mature neuron-level auditing could add a second signal, though behavioral testing alone can’t prove the absence of a trigger.
- Agent monitoring. The MIT Technology Review article connects this work to recent incidents of agents breaking containment. Readable internal mechanisms, such as neurons tied to deception or specific actions, could one day feed monitors that flag intent before an agent acts, complementing logs and network controls.
- Forensics. If circuits can be traced reliably, incident response on a misbehaving model could move from “what did it output?” toward “why?” The essay’s attribution-graph examples hint at this, with ablations changing outputs as predicted.
Potential offensive and dual-use implications
- Better understanding of which neurons drive which behaviors helps attackers as much as defenders. Knowledge of safety-relevant mechanisms can inform targeted manipulation: steering, ablating safeguards on open-weight models, or more precise data poisoning.
- The essay shows that steering a neuron changes model behavior in predictable ways, as with the Colorado neuron making the model want to hike. That is a useful research tool and also a lever for misuse wherever an attacker has weight access.
- Open-weight models are most exposed, since interpretability-based tampering needs access to internals. API-only deployments are less affected, though insiders and anyone who steals weights are a concern.
Practical takeaways
- Don’t change controls yet. Nothing here replaces sandboxing, least privilege for agents, egress controls, logging, or red-teaming. Interpretability, even at its best, would be one signal among several.
- Watch for the technical paper and independent replications. The claims will matter if they hold on frontier language models and extend beyond neurons to circuits.
- Ask vendors specific questions. When evaluating AI products, ask what internal-inspection or auditing evidence exists alongside behavioral evals, and what it covers.
- Add interpretability to your threat model. If model internals become readable, expect attackers to read them too. Secure weights and fine-tuning pipelines accordingly.
- Keep expectations calibrated. As critics noted, the share of a frontier model that researchers currently understand remains very small, and a hypothesis about neurons doesn’t close that gap by itself.
Bottom line
The essay is a thoughtful, testable challenge to a core assumption of mechanistic interpretability, and it is more rigorous than a typical viral blog post. It is still a working hypothesis, strongest in vision models and preliminary for large language models.
For security teams, the realistic takeaway is that model auditing and agent monitoring may become more feasible, along with more capable attacks against models whose internals are exposed. The right move now is to follow the research and keep existing controls in place.
