LLM Interpretability

LLM: Large Language Model

Large language model interpretability asks what information models represent, how it moves through layers and components, and which computations cause particular answers or behaviors. Researchers use probes and representation geometry, sparse autoencoders, attribution and circuit tracing, activation patching or ablation, and steering to study factual recall, mathematical reasoning, hallucination, refusal, and in-context learning. These methods can reveal distributed circuits and layer-wise signals, and sometimes support targeted interventions without retraining. A central challenge is that information being decodable—or a direction being steerable—does not prove the model uses it; causal faithfulness, robustness across prompts and model families, and reliable evaluation remain open tensions.

Latest papers

781 papers82 papers in the last four weeks, up 12% on the four weeks before. 0.5% of all new papers.

Shared Lexical Task Representations Explain Behavioral Variability In LLMs
Apr 23, 2026Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez +5Prompt Sensitivity LLM Prompting
Brown University · MIT · Northeastern
Trust but Verify: Introducing DAVinCI -- A Framework for Dual Attribution and Verification in Claim Inference for Language Models
Apr 23, 2026Vipula Rawte, Ryan Rossi, Franck Dernoncourt +1Claim Attribution in Language Models LLM Answer Verification
Adobe · Adobe Research
Slot Machines: How LLMs Keep Track of Multiple Entities
Apr 22, 2026Paul C. Bogdan, Jack LindseyLLM Interpretability Language Model Probing
Anthropic Fellows Program · Anthropic
TabSHAP
Apr 22, 2026Aryan Chaudhary, Prateek Agarwal, Tejasvi AlladiFeature Attribution SHAP Feature Attribution
Department of Computer Science Birla Institute of Technology and Science India
Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs
Apr 22, 2026Krishiv Agarwal, Ramneet Kaur, Colin Samplawski +6Language Model Safety Evaluation LLM Auditing
University of Florida · SRI
Dual-Stance Evaluation of Sycophancy: The Structure of Agreement and the Limits of Intervention
Apr 22, 2026Matthew James BuchanLLM Sycophancy Language Model Steering
Independent Researcher, United Kingdom.
LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures
Apr 22, 2026Yuhang Wu, Qinyuan Liu, Qiuyang Zhao +1LLM Interpretability Language Model Robustness
China Electronic Technology Nanhu Research Institute, Jiaxing, China
Surrogate modeling for interpreting black-box LLMs in medical predictions
Apr 22, 2026Changho Han, Songsoo Kim, Dong Won Kim +4Surrogate Modeling LLM Interpretability
Medical Big Data Research Center, Seoul National University Medical Research Center, Seoul National University College of Medicine, Seoul, Republic of Korea · Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, Republic of Korea. · Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA 02139 +5
R2IF: Aligning Reasoning with Decisions via Composite Rewards for Interpretable LLM Function Calling
Apr 22, 2026Aijia Cheng, Kailong Wang, Ling Shi +1RL for Language Models LLM Interpretability
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University, Shanghai, China · Huazhong University of Science and Technology · Nanyang Technological University
Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders
Apr 21, 2026Het Patel, Tiejin Chen, Hua Wei +2LLM Interpretability LLM Reliability
University of California, Riverside, CA · Arizona State University, Tempe, AZ
How Do Answer Tokens Read Reasoning Traces? Self-Reading Patterns in Thinking LLMs for Quantitative Reasoning
Apr 21, 2026Haoyang Chen, Yi Liu, Jianzhi Shao +3LLM Interpretability Mathematical Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University, China · Alibaba Group, China · Ant Group, China
Cell-Based Representation of Relational Binding in Language Models
Apr 21, 2026Qin Dai, Benjamin Heinzerling, Kentaro InuiLLM Interpretability Representation Probing
1Tohoku University · 2RIKEN AIP · 3MBZUAI
Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams
Apr 20, 2026Isaac Llorente-SaguerLanguage Model Safety Evaluation LLM Interpretability
Independent Researcher
Different Paths to Harmful Compliance: Behavioral Side Effects and Mechanistic Divergence Across LLM Jailbreaks
Apr 20, 2026Md Rysul Kabir, Zoran TiganjLanguage Model Safety Evaluation LLM Interpretability
Department of Computer Science Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
Understanding the Prompt Sensitivity
Apr 20, 2026Yang Liu, Chenhui ChuPrompt Sensitivity LLM Prompting
Kyoto University
Reasoning Models Know What's Important, and Encode It in Their Activations
Apr 20, 2026Yaniv Nikankin, Martin Tutek, Tomer Ashuach +2LLM Interpretability LLM Reasoning
Technion · University of Zagreb, FER · MIT +1
Exploring Concreteness Through a Figurative Lens
Apr 20, 2026Saptarshi Ghosh, Tianyu JiangFigurative Language Understanding LLM Interpretability
University of Cincinnati
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyMixture-of-Experts Interpretability LLM Interpretability
Independent · Georgia Institute of Technology · Goodfire
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Apr 20, 2026Rongyuan Tan, Jue Zhang, Zhuozhao Li +3LLM Evaluation Feature Attribution
1Southern University of Science and Technology, China · *Work done during an internship at Microsoft. · 2Microsoft
Language models recognize dropout and Gaussian noise applied to their activations
Apr 19, 2026Damiano Fornasiere, Mirko Bronzi, Spencer Kitts +3LLM Interpretability Language Model Introspection
LawZero
Are Emotion and Rhetoric Neurons in LLM? Neuron Recognition and Adaptive Masking for Emotion-Rhetoric Prediction Steering
Apr 19, 2026Li Zheng, Xin Zhang, Shuyi He +5Language Model Steering LLM Interpretability
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China · Ant International · School of Computing Technologies, RMIT University, Australia
Where is the Mind? Persona Vectors and LLM Individuation
Apr 18, 2026Pierre Beckmann, Patrick ButlinLLM Interpretability Personality Modeling in Language Models
MATS · EPFL & Idiap Research Institute · Eleos AI Research
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
Apr 17, 2026Max Henning Höth, Kristian Kersting, Björn Deiseroth +1CoT Faithfulness Feature Attribution
Aleph Alpha Research Lab1141 · TU Darmstadt Hessian.AI Lab1141
Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures
Apr 17, 2026Yutong Gao, Qinglin Meng, Yuan Zhou +1Transformer Interpretability LLM Interpretability
MOE Key Lab of Computational Linguistics, Peking University · Nanjing University of Science and Technology · Purdue University +1
Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
Apr 17, 2026Tanja Baeumel, Josef van Genabith, Simon OstermannLLM Interpretability Mechanistic Interpretability
German Research Center for Artificial Intelligence (DFKI) · Saarland University · Center for European Research in Trusted AI (CERTAIN)
LLM Reasoning Is Latent, Not the Chain of Thought
Apr 17, 2026Wenshuo WangLLM Interpretability CoT Reasoning
School of Future Technology, South China University of Technology, China
LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance
Apr 16, 2026Jack Wei Lun Shi, Minghao Dang, Wawan Solihin +1Feature Attribution LLM Interpretability
Department of Civil and Environmental Engineering, National University of Singapore, 117576, Singapore · School of Transportation Science and Engineering, Harbin Institute of Technology, Harbin 150090, China · Research and Innovation, NovaCITYNETS Pte.Ltd., 348615, Singapore
LLMbench: A Comparative Close Reading Workbench for Large Language Models
Apr 16, 2026David M. BerryLLM Evaluation LLM Interpretability
University of Sussex
Mechanistic Decoding of Cognitive Constructs in Large Language Models
Apr 16, 2026Yitong Shou, Manhao GuanLLM Interpretability Causal Interventions in Language Models
Zhejiang University, Hangzhou 310027, China. · The authors are with the College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China.
Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance
Apr 15, 2026Bar Alon, Itamar Zimerman, Lior WolfFaithfulness of Language Model Explanations LLM Interpretability
Blavatnik School of Computer Science, Tel Aviv University

Apr 23, 2026J/K move · Enter open · S save

Shared Lexical Task Representations Explain Behavioral Variability In LLMs

Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez, Eric Todd+4

Brown University · MIT · Northeastern

Prompt Sensitivity LLM Prompting LLM Interpretability Few-Shot Prompting Mechanistic Interpretability Attention Mechanisms

Open paper AskPDF arXiv

One of the most common complaints about large language models (LLMs) is their prompt sensitivity -- that is, the fact that their ability to perform a task or provide a correct answer to a question can depend unpredictably on the way the question is posed. We investigate this variation by comparing two very different but commonly-used styles of prompting: instruction-based prompts, which describe the task in natural language, and example-based prompts, which provide in-context few-shot demonstration pairs to illustrate the task. We find that, despite large variation in performance as a function of the prompt, the model engages some common underlying mechanisms across different prompts of a task. Specifically, we identify task-specific attention heads whose outputs literally describe the task -- which we dub lexical task heads -- and show that these heads are shared across prompting styles and trigger subsequent answer production. We further find that behavioral variation between prompts can be explained by the degree to which these heads are activated, and that failures are at least sometimes due to competing task representations that dilute the signal of the target task. Our results together present an increasingly clear picture of how LLMs' internal representations can explain behavior that otherwise seems idiosyncratic to users and developers.