In mechanistic interpretability, recent work scrutinizes transformer "circuits" - sparse, mono or multi layer sub computations, that may reflect human understandable functions. Yet, these network circuits are rarely acid-tested for their stability across different instances of the same deep learning architecture. Without this, it remains unclear whether reported circuits emerge universally across labs or turn out to be idiosyncratic to a particular estimation instance, potentially limiting confidence in safety-critical settings. Here, we systematically study stability across-refits in increasingly complex transformer language models of various sizes. We quantify, layer by layer, how similarly attention heads learn representations across independently initialized training runs. Our rigorous experiments show that (1) middle-layer heads are the least stable yet the most representationally distinct; (2) deeper models exhibit stronger mid-depth divergence; (3) unstable heads in deeper layers become more functionally important than their peers from the same layer; (4) applying weight decay optimization substantially improves attention-head stability across random model initializations; and (5) the residual stream is comparatively stable. Our findings establish the cross-instance robustness of circuits as an essential yet underappreciated prerequisite for scalable oversight, drawing contours around possible white-box monitorability of AI systems.
Figures & tables
Figure 1: Experimental workflow for assessing attention head stability. For each of our 26 architectural configurations, we instantiate N GPT-2-small LLMs, each with a different initial random seed. Each LLM is pre-trained from scratch with the OpenWebText or C4 datasets, depending on the architectural configuration. After pre-training, we take P evaluation prompts and compute attention score matrices for each head in each layer. An anchor model m is selected and compared to a pair model n , recursively for each randomly initialized and pre-trained LLM in that architectural subset. Cosine similarity is used to assess the similarity of each attention score matrix from each head, and a “Stability” score is computed from the best matching heads across refits of the architectural subset. See Methods § 3.2 for definitions of these terms.
Figure 2: Only some mid-layer attention heads persist across refits. For each head hi from layer l in an anchor model, We find its best-matching counterparts in the same layer l across other refits and use Shi(m) from Eq. 4 as the head’s stability (top). This plot thus summarizes inter-seed stability for all the heads of the anchor. Consistently, initial and final layers are more stable than middle layers (bottom).
Figure 3: Middle layer attention heads exhibit off-layer matching across re-fits. Cross-layer best match alignment heatmap shows strong diagonal dominance with heads typically matching to same-depth layers but broader off-diagonal spread in mid-layers.
Figure 4: Middle layer attention heads are most unique. Attention heads are most redundant in early and late layers of GPT-like models, while being quite distinct in the intermediate layers.
Figure 5: Instability of attention heads from middle transformer layers increases with increasing prompt length.
Figure 6: AdamW substantially improves attention head seed stability with no significant difference in performance. Top : 8-layer, 8-head, MLP architecture. Bottom : 12-layer, 12-head, MLP (GPT2-small) architecture.
Figure 7: Slmax (blue) and Slmin (red) stability vs. depth. Circles: MLP; squares: attention-only. Dark shades: Adam; light: AdamW. Numbers indicate heads per layer. Architectures trained using AdamW exhibit increased attention head stability.
Figure 8: rlmax (blue) and rlmin (red) relative depth vs. total layers. Circles: MLP; squares: attention-only. Dark shades: Adam; light: AdamW. Numbers indicate heads per layer. lmin tends to occur after lmax across architectures and, for 8- and 12-layer architectures, it occurs around the mid-depth.
Figure 9: The transformer residual stream is relatively much more stable than corresponding attention head that feed into it.
Figure 10: Unstable heads become increasingly more important with the depth of the layer.
Figure 11: Visualizing each attention head’s activation space using meta-SNE, when colored by relative depth. Heads are clustered by their relative position in the computational stack of the transformer model. Top : Adam-based architectures. Bottom : AdamW-based architectures.
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
#
Layers
Heads
Attention-only
Optimizer
Note
1
2
8
True
Adam
2
2
8
True
AdamW
3
2
8
False
Adam
4
2
8
False
AdamW
5
2
16
True
Adam
6
2
16
True
AdamW
Appendix
Table 1: Details of 26 architectures
Figure 13
Figure 12: Residual stream is relatively more stable than its corresponding attention heads.
Figure 13: Residual stream is relatively more stable than its corresponding attention heads.
Figure 31: Layer-wise stability is inversely correlated with average ( ℓ2 ) norm of attention-head query weights, suggesting a link between norm growth and head instability.
Figure 34Figure 35Figure 36
(a) Layers: 12, Heads: 12, Attn-only: False
Figure 36: AdamW substantially controls norm growth. See § 4.5
Transformer language models contain localized reasoning circuits, contiguous layer blocks that improve reasoning when duplicated at inference time. Finding these circuits currently requires brute-force sweeps costing 25 GPU hours per model. We propose CircuitProbe, which predicts circuit locations from activation statistics in under 5 minutes on CPU, providing a speedup of three to four orders of magnitude. We find that reasoning circuits come in two types: stability circuits in early layers, detected through the derivative of representation change, and magnitude circuits in late layers, detected through anomaly scoring. We validate across 9 models spanning 6 architectures, including 2025 models, confirming that CircuitProbe top predictions match or are within 2 layers of the optimal circuit in all validated cases. A scaling experiment across the Qwen 2.5 family reveals that layer duplication consistently benefits models under 3B parameters but degrades performance in 7B+ models, making this a practical scaling technique for small language models. CircuitProbe requires as few as 10 calibration examples and its predictions are stable across English, Hindi, Chinese, and French.
Despite extensive alignment efforts, Large Language Models (LLMs) remain vulnerable to generating unsafe content under adversarial prompting, yet the internal mechanisms by which safety behaviors are implemented remain poorly understood. We study LLM safety from a mechanistic interpretability perspective and characterize a multi-stage safety circuit that organizes refusal behavior, consisting of (i) Harmful Detection Heads that respond to harmful inputs, (ii) Safety Neurons that mediate and stabilize safety signals in the residual stream, and (iii) Refusal Heads that translate these signals into safe response generation. Using targeted attention-head and neuron-level interventions, we provide causal evidence consistent with this circuit organization, showing that suppressing upstream Harmful Detection Heads disrupts downstream refusal behavior and that safety neurons mediate this interaction. We validate that this decomposition recurs across multiple LLM architectures and adversarial attack settings, and use simple, architecture-preserving weight scaling as a mechanistic probe to test its functional relevance. Across six LLMs, circuit-guided scaling improves safety rates under attacks by 26.5%, while incurring only a 1.7% accuracy drop across four standard benchmarks. Overall, our results support a circuit-level interpretation of LLM safety and suggest that mechanistic abstractions can reveal stable and transferable patterns underlying aligned behavior.
Kuan-Lin Chu, Chung-En Sun, Tsui-Wei Weng
Computer Science and Engineering, University of California San Diego · Halıcıoğlu Data Science Institute, University of California San Diego
We present a three-step recipe for identifying attention-head circuits in pretrained transformers. A per-head spectral signal -- the time-integrated participation ratio of each head's attention output -- ranks heads doing sustained content-dependent computation without labels or attribution gradients. A task-pattern screen filters this general indicator into a task-specific candidate circuit, and group ablation against a matched-random control completes the causal claim. We validate across an 8x parameter range (51M to 1B-active / 7B-total), two architecture families (dense, mixture-of-experts), and four pretraining pipelines. The recipe ports: a 2-6 head induction circuit is causally necessary in every model tested, with a 94-100% drop in synthetic-induction top-1 after ablation. The spectral signal is predictive without supervision: on six independent seeds of a 51M-parameter probe model, the same computation identifies the seed-specific circuit on each seed. The fraction of heads doing identifiable specialized computation is conserved at 17-19% across the Pythia family (124M to 410M), while specific induction circuits stay 3-11 heads -- sublinear in total head count. This paper is the methodology anchor of a three-paper program; companion papers extend the recipe to developmental trajectories during pretraining and to composed-task circuits where pattern selectivity decouples from task-causal structure.