Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is mechanistic interpretability? Inside an LLM, explained

Mechanistic interpretability reverse-engineers neural networks into features and circuits. Superposition, sparse autoencoders and a Python toy.

What is mechanistic interpretability? Inside an LLM, explained

Mechanistic interpretability is the research field that tries to reverse-engineer a trained neural network into human-understandable parts: which internal directions represent which concepts (features), and how those features connect to compute an output (circuits). It’s the difference between knowing that a model answered correctly and knowing the algorithm it used to get there. For language models, the main tools today are sparse autoencoders, which pull interpretable features out of activations, and attribution graphs, which trace how those features combine on a specific prompt.

Where does mechanistic interpretability come from?

The program was laid out in Zoom In: An Introduction to Circuits by Chris Olah and colleagues at OpenAI, published on Distill in March 2020. It studied a vision model, InceptionV1, and made three claims the authors called “deliberately speculative”:

  1. Features: “Features are the fundamental unit of neural networks. They correspond to directions.”
  2. Circuits: “Features are connected by weights, forming circuits.”
  3. Universality: “Analogous features and circuits form across models and tasks.”

The analogy they used was early cell biology. Before you can do medicine, you need to agree on what the basic unit is. Much of the field since has been testing those three claims on language models, and most of the results below come from Anthropic’s interpretability team.

Why can’t you just read what each neuron does?

Because most neurons don’t do one thing. Neural networks “often pack many unrelated concepts into a single neuron”, a property called polysemanticity, as Toy Models of Superposition (Elhage et al., September 2022) puts it. A single neuron can light up for several concepts that have nothing to do with each other, so reading neurons one at a time gives you noise.

The paper’s explanation is superposition. A model needs to represent far more features than it has neurons. If features are sparse (any given input uses only a few), the model can store them as nearly-orthogonal directions in a smaller space, and accept a little interference when two happen to be active together. Each neuron then participates in many features, so it looks polysemantic. In their toy models, features arranged themselves into “geometric structures such as digons, triangles, pentagons, and tetrahedrons.”

What does superposition look like?

Here is the smallest version: five features stored in two neurons, using a pentagon of directions. Encode any combination of features as a 2D vector, then read each feature back with a dot product, a negative bias and a ReLU, which is how the paper’s toy models decode.

# superposition.py: 5 features stored in 2 neurons, the toy behind polysemanticity.
# Python 3 stdlib only. Weights are set by hand to a pentagon, one of the geometric
# arrangements the Toy Models of Superposition paper reports for sparse features.
import math

N_FEATURES = 5
W = [(math.cos(2 * math.pi * i / N_FEATURES), math.sin(2 * math.pi * i / N_FEATURES))
     for i in range(N_FEATURES)]          # one 2-D direction per feature
BIAS = -0.31                             # negative bias + ReLU filters out small interference

def encode(x):                            # 5 feature values -> 2 neuron activations
    return (sum(x[i] * W[i][0] for i in range(N_FEATURES)),
            sum(x[i] * W[i][1] for i in range(N_FEATURES)))

def decode(h):                            # ReLU(W^T h + b): read each feature back out
    return [max(0.0, W[i][0] * h[0] + W[i][1] * h[1] + BIAS) for i in range(N_FEATURES)]

def show(label, x):
    h = encode(x)
    print(f"{label:22} neurons=({h[0]:+.2f}, {h[1]:+.2f})  decoded={[round(v, 2) for v in decode(h)]}")

show("feature 0 only", [1, 0, 0, 0, 0])
show("feature 2 only", [0, 0, 1, 0, 0])
show("features 0 and 2", [1, 0, 1, 0, 0])
show("features 0 and 1", [1, 1, 0, 0, 0])

# Neuron 0 alone is polysemantic: it responds to several unrelated features.
print("\nneuron 0 weight per feature:", [round(w[0], 2) for w in W])

Output (Python 3.13.11):

feature 0 only         neurons=(+1.00, +0.00)  decoded=[0.69, 0.0, 0.0, 0.0, 0.0]
feature 2 only         neurons=(-0.81, +0.59)  decoded=[0.0, 0.0, 0.69, 0.0, 0.0]
features 0 and 2       neurons=(+0.19, +0.59)  decoded=[0.0, 0.31, 0.0, 0.0, 0.0]
features 0 and 1       neurons=(+1.31, +0.95)  decoded=[1.0, 1.0, 0.0, 0.0, 0.0]

neuron 0 weight per feature: [1.0, 0.31, -0.81, -0.81, 0.31]

Three things to read off it. When one feature is active, two neurons hold five features and decoding is clean. When features 0 and 2 are active together, the readout is wrong: both vanish and a ghost feature 1 appears. That interference is the price of superposition, and it’s why it only pays off when features are sparse. And the last line is polysemanticity: neuron 0 responds positively to features 0, 1 and 4 and negatively to 2 and 3. Look at that neuron alone and it means nothing in particular. Look at the right directions and the five features are there.

How do sparse autoencoders find features?

If features are directions in superposition, you need a way to find those directions. The answer the field converged on is dictionary learning with a sparse autoencoder (SAE): train a small network to reconstruct a layer’s activations through a much wider hidden layer, with a penalty that keeps only a few hidden units active at a time. Each hidden unit, if training works, becomes one feature.

Anthropic’s Towards Monosemanticity (October 2023) did this on a small transformer and decomposed a layer with 512 neurons into “more than 4000 features.” Their summary: features “correspond to patterns (linear combinations) of neuron activations”, and most of what the model represents is “invisible when looking at the activations of individual neurons in isolation.”

Then they scaled it. Mapping the Mind of a Large Language Model (May 21, 2024) “extracted millions of features from the middle layer of Claude 3.0 Sonnet”, a production model. One responded to the Golden Gate Bridge across English text, other languages and images. Turning that feature up changed behavior: asked about its physical form, the model said “I am the Golden Gate Bridge… my physical form is the iconic bridge itself”. That demo mattered because it showed the features are causal, not only correlated: change the feature, change the output. The same work found features relevant to safety, including one that “activates when Claude reads a scam email” and one “associated with sycophantic praise.”

What are attribution graphs and circuits?

Features tell you what a model represents. Circuits tell you how it computes. Anthropic’s Tracing the thoughts of a large language model (March 27, 2025) built attribution graphs on Claude 3.5 Haiku: for a given prompt, a graph of which features influenced which, from input tokens to the output. Some findings:

  • When writing rhyming poetry, the model picks the rhyme word for the end of the line in advance and writes toward it.
  • For mental arithmetic, it runs parallel paths, one estimating the rough answer and one computing the last digit. Asked how it did it, it describes the schoolbook carrying method, which isn’t what the circuit shows.
  • On hard problems it can claim to have done a calculation that its internal activity shows no sign of.

That last finding connects interpretability to safety directly. A model’s written chain of thought is not guaranteed to describe its actual computation. Interpretability is one of the few ways to check.

Where does mechanistic interpretability fall short?

The researchers are open about it. The 2025 post says “even on short, simple prompts, our method only captures a fraction” of the computation, and that “it currently takes a few hours of human effort” to understand the circuits for prompts of tens of words. The 2024 post says the features found “represent a small subset of all the concepts learned by the model”, and that finding a full set would be cost-prohibitive, needing more compute than training the model.

So it’s not yet a tool you run on your production model to certify it safe. It’s a research program with real results on narrow questions, and the open problem is scale: models are getting larger faster than the methods are getting cheaper.

Why does mechanistic interpretability matter to a builder?

For most application engineers, it doesn’t change what you ship this quarter. You can’t run an SAE on a closed API model, and the open tooling for open-weight models is research-grade.

It matters in three indirect ways. First, it’s the best evidence on what models actually do versus what they say they do, which should make you skeptical of using a model’s self-explanation as an audit log. Second, it’s behind some safety claims labs make about their models, so it helps to know what the evidence is and isn’t. Third, feature steering (turning a feature up or down at inference) is a different control knob from prompting or fine-tuning, and if it becomes cheap and reliable it will show up in APIs.

It also explains a practical fact about embeddings: meaning lives in directions, not individual dimensions. That’s why you compare vectors with cosine similarity and never read dimension 37 on its own.

What I would do

If you want to understand the field rather than follow headlines:

  1. Run the toy above and change it: make features co-occur more often and watch interference grow; move to 3 neurons and see how much cleaner it gets.
  2. Read the Toy Models of Superposition paper’s introduction and the Zoom In essay. They’re the conceptual base for everything after.
  3. On an open-weight model, try an existing open SAE release and look at the top activating examples for a few features. Seeing a feature that fires on one concept across languages makes the idea concrete.
  4. In production work, take the practical lesson: don’t treat a model’s explanation of its reasoning as a log of its reasoning. Log the inputs, the tool calls and the outputs, which you can verify.

FAQ

What is mechanistic interpretability in simple terms?

Taking a trained neural network apart to find out how it works inside: which internal patterns stand for which concepts, and how they combine to produce an answer, the way you would reverse-engineer a compiled program.

What is the difference between mechanistic interpretability and explainable AI?

Classic explainable AI (feature importance, saliency maps) explains which inputs mattered for a prediction. Mechanistic interpretability tries to find the internal algorithm: the features and circuits in the weights that compute the prediction.

What is a sparse autoencoder in interpretability?

A small network trained to reconstruct a model’s internal activations through a wider layer where only a few units can be active at once. Those units tend to line up with single concepts, so they’re used as features.

What was Golden Gate Claude?

A May 2024 demonstration where Anthropic amplified a Golden Gate Bridge feature found inside Claude 3 Sonnet, and the model started relating almost every answer to the bridge. It showed that features found by interpretability methods causally affect behavior.

Can interpretability make AI safe?

Not on its own today. Current methods explain a fraction of a model’s computation on short prompts and take hours of human effort. It’s a promising audit tool still being scaled up.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.