Paper, researchers, and primary source
Major lab research / interpretability
Toy Models of Superposition uses small neural networks to study how more features than available dimensions can be represented through overlapping directions.
Contribution 1
Modeled superposition as a consequence of sparse feature importance.
Contribution 2
Connected feature geometry to interference and phase transitions.
Contribution 3
Provided tractable experiments for testing mechanistic-interpretability hypotheses.
Research context
interpretability / 2022
Toy Models of Superposition places superposition inside the broader interpretability discussion at Anthropic, with interpretability supplying a second analytical lens. Its contribution chain has three links: Modeled superposition as a consequence of sparse feature importance; Connected feature geometry to interference and phase transitions; and Provided tractable experiments for testing mechanistic-interpretability hypotheses. This framing makes features a property to inspect within the study, not a label that settles later deployments. Its builder-facing implication is that individual neurons need not map cleanly to individual concepts, limiting naive neuron-level explanations of large models.
Methods and evidence reading
1 cataloged method notes
Method 1
The experimental design in Toy Models of Superposition tests modeled superposition as a consequence of sparse feature importance and connected feature geometry to interference and phase transitions against the paper's documented baselines, datasets, model variants, or systems workloads.
How to read the evidence
A careful reading of Toy Models of Superposition starts with the experiment's declared scope, not the reputation of Anthropic. The editorial method record pairs two moves: Modeled superposition as a consequence of sparse feature importance; and Connected feature geometry to interference and phase transitions. The outcome-facing contribution is: Provided tractable experiments for testing mechanistic-interpretability hypotheses. This supports the bounded implication that individual neurons need not map cleanly to individual concepts, limiting naive neuron-level explanations of large models. It does not remove the source limit that the superposition comparison in Toy Models of Superposition is interpretable only alongside compute budget, comparison baselines, task distribution, architecture choices, evaluation protocol, and documented data, which limits claims about unseen deployments. Follow-on evaluation should therefore vary interpretability while retaining an explicit superposition baseline. A new evaluation of Toy Models of Superposition should disclose its superposition comparator, isolate interpretability changes, and retain observations that qualify Connected feature geometry to interference and phase transitions before deployment review.
Findings in the source record
1 paper-specific findings
- The reported evidence in Toy Models of Superposition supports provided tractable experiments for testing mechanistic-interpretability hypotheses.
Practical implication for AI builders
Anthropic / 2022
Proposed BrokenGPT application
Research blueprint / proposed status
Proposed route placement / superposition
For a proposed BrokenGPT experiment based on Toy Models of Superposition, present interpretability signals as experimental diagnostics with uncertainty, and avoid treating a single activation or feature label as a complete causal explanation. Keep the superposition path isolated, versioned, and attributable to this research record.
Proposed acceptance test / interpretability
Validate the proposed superposition route against the paper's reported outcome: Provided tractable experiments for testing mechanistic-interpretability hypotheses. Use replication, feature recovery, model-scale transfer, and causal intervention to evaluate Toy Models of Superposition, but retain a distinct interpretability ledger so the proposed superposition path cannot hide concentrated failures.
Proposed decision boundary / features
Balance explanatory detail, faithfulness, and generality before promoting the proposed features design. Because A deployment review should isolate changed operating conditions, another user population, a later model revision, new hardware, and a different product when translating the superposition contribution into a different system, adoption remains conditional on replay under BrokenGPT's selected model, runtime, and policy configuration.
Limitations, verification, and source
Boundaries recorded with the paper
Limitations
- The superposition comparison in Toy Models of Superposition is interpretable only alongside compute budget, comparison baselines, task distribution, architecture choices, evaluation protocol, and documented data, which limits claims about unseen deployments.
- A deployment review should isolate changed operating conditions, another user population, a later model revision, new hardware, and a different product when translating the superposition contribution into a different system.
PRIMARY SOURCES
- 01Toy Models of Superposition
Anthropic — Primary primary arXiv paper / 21 September 2022 / Nelson Elhage, Tristan Hume, Catherine Olsson, and 13 more