LLM Interpretability

LLM: Large Language Model

Large language model interpretability asks what information models represent, how it moves through layers and components, and which computations cause particular answers or behaviors. Researchers use probes and representation geometry, sparse autoencoders, attribution and circuit tracing, activation patching or ablation, and steering to study factual recall, mathematical reasoning, hallucination, refusal, and in-context learning. These methods can reveal distributed circuits and layer-wise signals, and sometimes support targeted interventions without retraining. A central challenge is that information being decodable—or a direction being steerable—does not prove the model uses it; causal faithfulness, robustness across prompts and model families, and reliable evaluation remain open tensions.

Latest papers

781 papers82 papers in the last four weeks, up 12% on the four weeks before. 0.5% of all new papers.

Darkness Visible: Reading the Exception Handler of a Language Model
Apr 6, 2026Peter BaloghConcept Localization in Language Models LLM Interpretability
How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models
Apr 6, 2026Gregory N. FrankLLM Interpretability LLM Safety
Chief Scientist, Molt AI Corp.
NeuroSymbolic AI for Legal AI-TRISM: Trustworthy, Reliable, Interpretable, Safe Models
Apr 5, 2026Deepa Tilwani, Yash Saxena, Ankur Padia +2LLM Reliability Neuro-Symbolic AI
Department of Computer Science, AI Institute, University of South Carolina, Columbia, South Carolina, USA · Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, Baltimore, Maryland, USA · Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA
Unmasking Hallucinations: A Causal Graph-Attention Perspective on Factual Reliability in Large Language Models
Apr 5, 2026Sailesh kiran kurra, Shiek Ruksana, Vishal BorusuLLM Hallucination Mitigation Factual Knowledge in Language Models
Amazon,Texas, United States of America · Vasavi College of Engineering, Telangana,India
Automated Attention Pattern Discovery at Scale in Large Language Models
Apr 4, 2026Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes +2Transformer Interpretability LLM Interpretability
Delft University of Technology
Testing the Limits of Truth Directions in LLMs
Apr 4, 2026Angelos Poulis, Mark Crovella, Evimaria TerziLLM Interpretability Linear Probing
Department of Computer Science, Boston University · Faculty of Computing & Data Sciences, Boston University
Entropy and Attention Dynamics in Small Language Models: A Trace-Level Structural Analysis on the TruthfulQA Benchmark
Apr 4, 2026Adeyemi Adeseye, Aisvarya Adeseye, Hannu Tenhunen +1Small Language Models LLM Honesty
Brilloconnetz Partners avoin yhtiö, Turku, Finland · University of Turku, Turku, Finland · Royal Institute of Technology, Stockholm, Sweden
Verbalizing LLMs' assumptions to explain and control sycophancy
Apr 3, 2026Myra Cheng, Isabel Sieh, Humishka Zope +7LLM Sycophancy Human-AI Interaction
Stanford University · University of Oxford · The University of Texas at Austin
NeuReasoner: Towards Explainable, Controllable, and Unified Reasoning via Mixture-of-Neurons
Apr 3, 2026Haonan Dong, Kehan Jiang, Haoran Ye +3Large Reasoning Models Efficient Language Model Reasoning
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · School of Software and Microelectronics, Peking University
GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamics
Apr 3, 2026Yujing Wang, Yuanbang Liang, Yukun Lai +2LLM Interpretability LLM Uncertainty Estimation
Beihang University · Cardiff University · King’s College London
On the Geometric Structure of Layer Updates in Deep Language Models
Apr 2, 2026Jun-Sik YooRepresentation Geometry in Language Models LLM Interpretability
Institute of Basic Science, Korea University, 145 Anam-ro, Seoul, South Korea 02841.
VISTA: Visualization of Token Attribution via Efficient Analysis
Apr 2, 2026Syed Ahmed, Bharathi Vokkaliga Ganesh, Jagadish Babu P +8LLM Interpretability Perturbation-Based Feature Attribution
Responsible AI Office Infosys Limited, Bangalore, India
The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Apr 2, 2026Jeremy Herbst, Stefan Wermter, Jae Hee LeeMixture-of-Experts Interpretability Mixture-of-Experts Models
Department of Informatics, University of Hamburg, Hamburg, Germany.
Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations
Apr 2, 2026Shou-Tzu Han, Rodrigue Rizk, KC SantoshLanguage Model Robustness LLM Reasoning
Department of Computer Science, University of South Dakota
Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
Apr 1, 2026Tianyi Zhao, Yinhan He, Wendy Zheng +2Confidence Estimation in Language Models Language Model Calibration
University of Virginia
Friends and Grandmothers in Silico: Localizing Entity Cells in Language Models
Apr 1, 2026Itay Yona, Dan Barzilay, Michael Karasik +1Concept Localization in Language Models LLM Interpretability
Mentaleap · Independent Researcher · Tel Aviv University
Therefore I am. I Think
Apr 1, 2026Esakkivel Esakkiraja, Sai Rajeswar, Denis Akhiyarov +1LLM Decision-Making Implicit Reasoning in Language Models
Khoury College of Computer Sciences Northeastern University · Mila, ServiceNow Research · ServiceNow
In-context superposition: human-like working memory interference in large language models
Apr 1, 2026Hua-Dong Xiong, Li Ji-An, Jiaqi Huang +3Working Memory in Language Models LLM Interpretability
School of Psychological and Brain Sciences, Georgia Tech · Department of Psychology, New York University · Department of Cognitive Science, Indiana University Bloomington +3
The Overlooked Repetitive Lengthening Form in Sentiment Analysis
Apr 1, 2026Lei Wang, Eduard DragutSentiment Analysis Instruction Tuning
Temple University, Philadelphia, PA, USA
Temporal Dependencies in In-Context Learning: The Role of Induction Heads
Apr 1, 2026Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +3In-Context Learning Temporal Reasoning in Language Models
Department of Computer Science, Indiana University Bloomington
Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?
Apr 1, 2026Luis Frentzen Salim, Lun-Wei Ku, Hsing-Kuo Kenneth PaoMultilingual Language Models LLM Fine-Tuning
Institute of Information Science, Academia Sinica · Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology
From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
Apr 1, 2026Ayan Datta, Mounika Marreddy, Alexander Mehler +2Symbolic Reasoning in Language Models LLM Interpretability
IIIT Hyderabad · Goethe University, Frankfurt am Main, Germany · University of Sheffield, UK
Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations
Mar 31, 2026Haoran Wang, Li Xiong, Kai ShuPrivacy Leakage in Language Models Contextual Integrity
Department of Computer Science, Emory University
Unveiling the Reasoning Process of Large Language Models
Mar 31, 2026Junjie Zhang, Zhen Shen, Xisong Dong +1Symbolic Reasoning in Language Models LLM Interpretability
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity
Mar 31, 2026Zoë Prins, Samuele Punzo, Frank Wildenburg +2LLM Interpretability LLM Evaluation
Graduate School of Informatics, University of Amsterdam · College of Informatics, University of Amsterdam · Department of Medical Informatics, Amsterdam University Medical Center +1
Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States
Mar 31, 2026Dianxing Zhang, Gang Li, Sheng LiLLM Routing LLM Interpretability
Digital China AI Research Institute
Beyond the Answer: Decoding the Behavior of LLMs as Scientific Reasoners
Mar 30, 2026Rohan Pandey, Eric Ye, Michael LiLLM Prompting LLM Interpretability
Carnegie Mellon University
Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs
Mar 29, 2026Utsav Maskey, Mark Dras, Usman NaseemLLM Interpretability LLM Refusal Behavior
Macquarie University
From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents
Mar 27, 2026Trilok Padhi, Ramneet Kaur, Krishiv Agarwal +9LLM Interpretability Representation Probing
Georgia State University, Atlanta, USA · Computer Science Lab, SRI, Menlo Park, USA · Computer Science Department, University of Florida, USA +1
Weight Tying Biases Token Embeddings Towards the Output Space
Mar 27, 2026Antonio Lopardo, Avyukth Harish, Catherine Arnett +1Language Modeling Representation Geometry in Language Models
EleutherAI · University of California Berkeley

Apr 6, 2026J/K move · Enter open · S save

Darkness Visible: Reading the Exception Handler of a Language Model

Peter Balogh

Concept Localization in Language Models LLM Interpretability Circuits in Language Models

Open paper AskPDF arXiv

The final MLP of GPT-2 Small exhibits a fully legible routing program -- 27 named neurons organized into a three-tier exception handler -- while the knowledge it routes remains entangled across ~3,040 residual neurons. We decompose all 3,072 neurons (to numerical precision) into: 5 fused Core neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover -- where MLP intervention shifts from helpful to harmful -- is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect -- GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2