cs.LGApr 4, 2026

Automated Attention Pattern Discovery at Scale in Large Language Models

Authors: Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes, Arie van Deursen, Maliheh Izadi

Organizations: Delft University of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models have found success by scaling up capabilities to work in general settings. The same can unfortunately not be said for interpretability methods. The current trend in mechanistic interpretability is to provide precise explanations of specific behaviors in controlled settings. These often do not generalize, or are too resource intensive for larger studies. In this work we propose to study repeated behaviors in large language models by mining completion scenarios in Java code datasets, through exploiting the structured nature of code. We collect the attention patterns generated in the attention heads to demonstrate that they are scalable signals for global interpretability of model components. We show that vision models offer a promising direction for analyzing attention patterns at scale. To demonstrate this, we introduce the Attention Pattern - Masked Autoencoder(AP-MAE), a vision transformer-based model that efficiently reconstructs masked attention patterns. Experiments on StarCoder2 show that AP-MAE (i) reconstructs masked attention patterns with high accuracy, (ii) generalizes across unseen models with minimal degradation, (iii) reveals recurring patterns across inferences, (iv) predicts whether a generation will be correct without access to ground truth, with accuracies ranging from 55% to 70% depending on the task, and (v) enables targeted interventions that increase accuracy by 13.6% when applied selectively, but cause collapse when applied excessively. These results establish attention patterns as a scalable signal for interpretability and demonstrate that AP-MAE provides a transferable foundation for both analysis and intervention in large language models. Beyond its standalone value, AP-MAE also serves as a selection procedure to guide fine-grained mechanistic approaches. We release code and models to support future work in large-scale interpretability.

Figures & tables

Model InputsEncoderDecoder
Pattern Size256256Layers2424Layers88
Patch Size3232Dim512512Dim512512
Mask Ratio0.50.5Heads1616Heads88
Batch Size480480MLP20482048MLP20482048
Learning Rate1.44×10−31.44\times 10^{-3}    
Table 1: Training and architecture parameters for AP-MAE.
Figure 1: Comparison of attention pattern reconstruction methods: (a) raw attention pattern, (b) raw attention pattern with pixel values scaled for visualization, (c) log normalized attention pattern. Comparing the reconstructions in (b) and (c), we see that scaling the attention patterns prior to training AP-MAE is essential.
Table 2: Cross-evaluation of AP-MAE models. The Trained column refers to the target LLM that generated attention patterns the AP-MAE model was trained on. Evaluated refers to the target LLM that generated attention patterns that were used only for evaluation. Diagonal entries denote the traditional training/evaluation setup, while off-diagonal entries show the performance of AP-MAE on attention patterns generated by an ’unseen’ model.
Figure 2: Comparison of different clustering results: (a) examples of different patterns found by clustering, (b) attention patterns within a single cluster, (c) attention patterns generated by noise , as described in Section 5.1 .
Figure 3: Distribution of the number of clusters in a head
Figure 4: Performance of target LLMs on the studied tasks, and the accuracy of the CatBoost classifier
Figure 5: Difference in mean SHAP values per cluster for the CatBoost classifiers, classifying predictions for the End of Line task across all target sizes
Figure 6: Maximum difference in mean SHAP values per cluster explaining the global effect of each pattern in each head on the correctness classifier
Figure 7: Effects of interventions based on SHAP values showing the net change in correct token generation as heads are progressively zeroed. (a) Global results across tasks, shown on a logarithmic x-axis for clarity, (b) End-of-Line token prediction, the best-performing classification task, (c) String Literal completion, the worst-performing classification task,
Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: SHAP values for the assignment operators task.
Figure 9: SHAP values for the boolean literals task.
Figure 10: SHAP values for the boolean operators task.
Figure 11: SHAP values for the closing brackets task.
Figure 12: SHAP values for the end-of-line token task.
Figure 13: SHAP values for the identifiers task.
Figure 14: SHAP values for the mathematical operators task.
Figure 15: SHAP values for the numeric literals task.
Figure 16: SHAP values for the random masking task.
Figure 17: SHAP values for the string literals task.
Figure 18: Net change in number of correct generations after intervention for the assignment operators task.
Figure 19: Net change in number of correct generations after intervention for the boolean literals task.
Figure 20: Net change in number of correct generations after intervention for the boolean operators task.
Figure 21: Net change in number of correct generations after intervention for the closing brackets task.
Figure 22: Net change in number of correct generations after intervention for the end-of-line token task.
Figure 23: Net change in number of correct generations after intervention for the identifiers task.
Figure 24: Net change in number of correct generations after intervention for the mathematical operators task.
Figure 25: Net change in number of correct generations after intervention for the numeric literals task.
Figure 26: Net change in number of correct generations after intervention for the random masking task.
Figure 27: Net change in number of correct generations after intervention for the string literals task.
More in this topic

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 338 papers

Explore similar work

CardsList
Aug 6, 2026

Scaling Inherently Interpretable Language Models

Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.

Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail +7

Guide Labs Team

Language Model SteeringLLM InterpretabilityDiffusion Language ModelsRepresentation Geometry in Language ModelsLLM Training
AskPDF
May 9, 2026

Architecture, Not Scale: Circuit Localization in Large Language Models

Mechanistic interpretability assumes that circuit analysis becomes harder as models scale. We challenge this assumption by showing that the attention architecture matters more than parameter count. Studying three circuit types across Pythia and Qwen2.5, we find that grouped query attention produces circuits that are far more concentrated and mechanistically stable than standard multi-head attention at comparable scales. The same concentration pattern holds across indirect object identification, induction heads, and factual recall. Within a single architecture family (Qwen2.5), factual recall circuits undergo a discrete phase transition above a critical scale, collapsing to a single bottleneck rather than degrading gradually. These findings suggest that some architectural choices make large models more tractable to study and that interpretability difficulty is not a fixed consequence of model size.

Sohan Venkatesh

Manipal Institute of Technology Bengaluru.

Transformer InterpretabilityGrouped-Query AttentionCircuit DiscoveryAttention Head AnalysisMechanistic InterpretabilityCircuits in Language Models
AskPDF
May 12, 2026

Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models

Large language models have achieved remarkable capabilities across diverse tasks, yet their internal decision-making processes remain largely opaque, limiting our ability to inspect, control, and systematically improve them. This opacity motivates a growing body of research in mechanistic interpretability, with sparse autoencoders (SAEs) emerging as one of the most promising tools for decomposing model activations into sparse, interpretable feature representations. We introduce Qwen-Scope, an open-source suite of SAEs built on the Qwen model family, comprising 14 groups of SAEs across 7 model variants from the Qwen3 and Qwen3.5 series, covering both dense and mixture-of-expert architectures. Built on top of these SAEs, we show that SAEs can go beyond post-hoc analysis to serve as practical interfaces for model development along four directions: (i) inference-time steering, where SAE feature directions control language, concepts, and preferences without modifying model weights; (ii) evaluation analysis, where activated SAE features provide a representation-level proxy for benchmark redundancy and capability coverage; (iii) data-centric workflows, where SAE features support multilingual toxicity classification and safety-oriented data synthesis; and (iv) post-training optimization, where SAE-derived signals are incorporated into supervised fine-tuning and reinforcement learning objectives to mitigate undesirable behaviors such as code-switching and repetition. Together, these results demonstrate that SAEs can serve not only as post-hoc analysis tools, but also as reusable representation-level interfaces for diagnosing, controlling, evaluating, and improving large language models. By open-sourcing Qwen-Scope, we aim to support mechanistic research and accelerate practical workflows that connect model internals to downstream behavior.

Boyi Deng, Xu Wang, Yaoning Wang +15

Qwen Team

LLM InterpretabilitySparse AutoencodersActivation Steering
AskPDF