Mechanistic Interpretability

Reverse-engineering the black box

Mechanistic interpretability is the discipline of reverse-engineering artificial neural networks to improve understanding of the mechanisms that generate their observed behaviours. Instead of just evaluating behavior based on inputs and outputs, practitioners dissect the model's individual neurons and abstract latent spaces to map out their processes and derive the exact mathematical basis driving their behavior. The ultimate goal is to enhance AI safety and reliability while improving design engineering knowledge.

Core Methods

Locating What Matters

Attribution — activation patching, path patching, and the logit lens trace which components causally drive a given output.

Decomposing Representations

Sparse autoencoders and dictionary learning pull superposed activations apart into sparse, human-interpretable features.

Finding Circuits

Automated circuit discovery and attribution graphs chain those features into a readable, layer-by-layer computational graph.

Validation

Synthetic ground-truth tasks (like the HMM sandbox below) and causal scrubbing test whether a proposed circuit actually explains the behavior.

1 / 15

Slide 1: Decrypting neural networks using the mathematics of Hidden Markov Models

Why continuous, high-dimensional layers resist intuitive inspection in the first place.

Slide 2: Continuous neural network layers create a fundamental interpretation barrier

Why HMMs make a useful ground-truth benchmark: known discrete states, known transition probabilities.

Slide 3: Hidden Markov Models act as a mathematical Rosetta stone for interpretability

Comparing the two systems head-to-head across computing space, state visibility, and transition logic.

Slide 4: The structural divide between discrete Hidden Markov Models and continuous neural networks

The experimental pipeline: a known generator trains the model that researchers then try to reverse-engineer.

Slide 5: Reverse-engineering black-box learners using known HMM generators

Three angles of proof the sandbox is built to deliver, from statistics to timing.

Slide 6: The HMM sandbox validates interpretability across three dimensions

The core question up front: do trained networks recover real probabilities, or just shallow shortcuts?

Slide 7: Testing whether neural networks internalize true probability distributions

Comparing the exact mathematical answer against what the trained network actually produced.

Slide 8: Transformers approximate optimal Bayesian forward filters

How an unstructured internal representation resolves into an organized geometric shape.

Slide 9: Transformers map probability distributions into structured belief geometry

The extraction method, step by step, from raw activations to a recovered transition matrix.

Slide 10: Markovian circuit tracing extracts discrete hidden states from continuous layers

The mismatch a network has to overcome: discrete jumps on one side, smooth trajectories on the other.

Slide 11: Neural networks must emulate discrete stochastic logic using continuous trajectories

The dynamical motif researchers found for how a continuous system executes a discrete jump.

Slide 12: Kick-neurons trigger sudden state jumps between activation clusters

Each mechanistic hypothesis paired with the mathematical result the HMM sandbox confirmed.

Slide 13: The sandbox upgrades mechanistic hypotheses into mathematical proofs

Bringing belief geometry and kick-neurons together into one picture of the network's internal simulation.

Slide 14: Synthesis — a continuous neural manifold physically restructured to simulate an HMM

Closing frame: why a fully known sandbox matters for validating the next generation of circuit-finding tools.

Slide 15: Establishing absolute ground truth for next-generation circuit-finding tools

Open Source MI Tools

The mechanistic interpretability (MI) ecosystem has matured rapidly, evolving from ad-hoc PyTorch scripts into a rich toolkit of specialized frameworks. Depending on whether you are looking to patch activations, train Sparse Autoencoders (SAEs), or visualize attention circuits, the open-source landscape offers tailored tools for each layer of the stack.

Mechanistic Interpretability Tools
Category	Tool	Description	Best For	Organization	Source
Core Interventions & Activation Hooking	TransformerLens	The standard framework in MI research, originally created by Neel Nanda. Re-implements popular transformers (e.g., Llama, Gemma, GPT-2) into a standardized format with explicit, named `HookPoints` throughout the architecture, plus built-in utilities for activation caching, logit lens readouts, and causal activation patching.	Analyzing circuit behavior, prompt interventions, and reverse-engineering attention heads on standard transformer architectures.	TransformerLensOrg (community) · originated by Neel Nanda, ex‑Anthropic	GitHub
Core Interventions & Activation Hooking	NNsight (and NDIF)	A flexible framework that operates directly on native PyTorch models without requiring manual model re-implementation. Intercepts and modifies tensor streams contextually, and supports remote execution on frontier-scale models hosted on remote clusters.	Large-scale models (70B+ parameters), non-standard architectures, or experiments requiring distributed multi-GPU interventions.	Northeastern University · National Deep Inference Fabric (NDIF)	GitHub
Core Interventions & Activation Hooking	pyvene	A declarative intervention library that treats interventions as first-class primitives. Allows configuring complex intervention schemas — e.g., swapping internal representations across multiple forward runs simultaneously — using clean configuration dictionaries.	Causal mediation analysis, distributed alignment search (DAS), and formal causal abstraction experiments.	Stanford NLP Group	GitHub
Feature Disentanglement & Sparse Autoencoders (SAEs)	SAELens	The go-to toolkit for training, evaluating, and running Sparse Autoencoders on neural activations. Integrates seamlessly with TransformerLens and Hugging Face models to decompose dense activation vectors into monosemantic, human-interpretable features.	Dictionary learning, feature extraction, and steering model outputs via sparse concepts.	Decode Research (Joseph Bloom, Curt Tigges, et al.)	GitHub
Feature Disentanglement & Sparse Autoencoders (SAEs)	Gemma Scope	An open suite of JumpReLU sparse autoencoders trained on every layer and sub-component of the Gemma model family — thousands of pre-trained, open-weight SAEs, saving massive amounts of GPU compute.	—	Google DeepMind	Hugging Face
Feature Disentanglement & Sparse Autoencoders (SAEs)	SAE Bench	A comprehensive benchmark of eight diverse SAE evaluations (unsupervised metrics and downstream tasks), open-sourcing a suite of 200+ SAEs across eight architectures and training algorithms.	—	Independent researchers (Adam Karvonen et al.) · interactive interface hosted by Neuronpedia	GitHub
Circuit Tracing & Automation	Anthropic Circuit-Tracing Tools	Open-source tools (`circuit-tracer`) that map internal computation pathways, generating attribution graphs that visually reveal how concepts and features interact layer-by-layer to produce specific outputs.	Tracing complex multi-step reasoning pathways or understanding safety-relevant behaviors like refusal.	Anthropic (Anthropic Fellows Program)	GitHub
Visual Exploration & Notebook UIs	Neuronpedia	An open-source web platform and interactive interface for exploring SAE features and attribution graphs. Search for abstract concepts (e.g., "code syntax," "deception," or specific entities) and inspect feature activations across dataset samples interactively.	—	Decode Research (Johnny Lin, Joseph Bloom) · supported by Anthropic, Open Philanthropy, and others	Site
Visual Exploration & Notebook UIs	CircuitsVis	A JavaScript/Python visualization library for inline Jupyter notebook analysis — renders interactive attention head patterns, vector projections, and activation patching heatmaps directly in the research environment.	—	TransformerLensOrg · created by Alan Cooney and Neel Nanda	GitHub