LLM Interpretability

LLM: Large Language Model

Large language model interpretability asks what information models represent, how it moves through layers and components, and which computations cause particular answers or behaviors. Researchers use probes and representation geometry, sparse autoencoders, attribution and circuit tracing, activation patching or ablation, and steering to study factual recall, mathematical reasoning, hallucination, refusal, and in-context learning. These methods can reveal distributed circuits and layer-wise signals, and sometimes support targeted interventions without retraining. A central challenge is that information being decodable—or a direction being steerable—does not prove the model uses it; causal faithfulness, robustness across prompts and model families, and reliable evaluation remain open tensions.

Latest papers

781 papers82 papers in the last four weeks, up 12% on the four weeks before. 0.5% of all new papers.

Applied Explainability for Large Language Models: A Comparative Study
Apr 15, 2026Venkata Abhinandan KancharlaLLM Interpretability Explainability Evaluation
NRI Institute of Technology
Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs
Apr 14, 2026Vishal Pramanik, Maisha Maliha, Nathaniel D. Bastian +1Feature Attribution LLM Interpretability
Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL 32611, USA · School of Computer Science, University of Oklahoma, Norman, OK 73019, USA · Department of Electrical Engineering and Computer Science, United States Military Academy, West Point, NY 10996, USA
IDEA: An Interpretable and Editable Decision-Making Framework for LLMs via Verbal-to-Numeric Calibration
Apr 14, 2026Yanji He, Yuxin Jiang, Yiwen Wu +3LLM Decision-Making Language Model Calibration
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd · The Hong Kong University of Science and Technology
Latent Planning Emerges with Scale
Apr 14, 2026Michael Hanna, Emmanuel AmeisenLLM Planning Implicit Reasoning in Language Models
ILLC, University of Amsterdam · Anthropic
Phantom Paradoxes: Language Models Treat Unresolved Circular Truth Dependence as Contradiction
Apr 13, 2026Ji Ho BaeLLM Interpretability LLM Reliability
JRTI
Robust Explanations for User Trust in Enterprise NLP Systems
Apr 13, 2026Guilin Zhang, Kai Zhao, Jeffrey Friedman +3Explainable Artificial Intelligence LLM Interpretability
Workday AI Research
Psychological Concept Neurons: Can Neural Control Bias Probing and Shift Generation in LLMs?
Apr 13, 2026Yuto Harada, Hiro Taiyo HamadaPersonality Modeling in Language Models LLM Interpretability
Araya Inc.
A Mechanistic Analysis of Looped Reasoning Language Models
Apr 13, 2026Hugh Blayney, Álvaro Arroyo, Johan Obando-Ceron +4Recurrent Transformers LLM Interpretability
University of Oxford · Mila – Quebec AI Institute · Universit´e de Montréal +1
Why Do Large Language Models Generate Harmful Content?
Apr 13, 2026Rajesh Ganguli, Raha MoraffahLLM Safety LLM Interpretability
Worcester Polytechnic Institute
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
Apr 13, 2026Pengfeng Li, Chen Huang, Chaoqun Hao +4Causal Reasoning in Language Models LLM Evaluation
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
CausalGaze: Unveiling Hallucinations via Counterfactual Graph Intervention in Large Language Models
Apr 13, 2026Linggang Kong, Lei Wu, Yunlong Zhang +4LLM Hallucination Detection Hallucination in Language Models
College of Electronic Engineering, National University of Defense Technology · Anhui Province Key Laboratory of Cyberspace Security Situation Awareness and Evaluation · Institute of Computer Application, China Academy of Engineering Physics
Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
Apr 12, 2026Jakub Binkowski, Kamil Adamczewski, Tomasz KajdanowiczLLM Hallucination Detection Attention Sinks
Department of Artificial Intelligence, Wroclaw University of Science and Technology
Steered LLM Activations are Non-Surjective
Apr 10, 2026Aayush Mishra, Daniel Khashabi, Anqi LiuActivation Steering LLM Interpretability
Johns Hopkins University
The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Apr 10, 2026Xi Wang, Soufiane Hayou, Eric NalisnickMixture-of-Experts Expert Specialization Mixture of Experts
Johns Hopkins University
Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning
Apr 10, 2026Bryan Cheng, Jasper ZhangIn-Context Learning LLM Interpretability
William A. Shine Great Neck South High School
What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal
Apr 9, 2026Stephen Cheng, Sarah Wiegreffe, Dinesh ManochaLanguage Model Steering LLM Interpretability
University of Maryland, College Park
Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models
Apr 9, 2026Yating Wang, Wenting Zhao, Yaqi Zhao +3Knowledge Editing Language Model Editing
School of Software, Shandong University, Jinan, China · Salesforce AI Research
Linear Representations of Hierarchical Concepts in Language Models
Apr 9, 2026Masaki Sakata, Benjamin Heinzerling, Takumi Ito +2Hierarchical Representation Learning LLM Interpretability
Tohoku University · RIKEN · Langsmith Inc. +2
Emotion Concepts and their Function in a Large Language Model
Apr 9, 2026Nicholas Sofroniew, Isaac Kauvar, William Saunders +13Emotion Representation in Language Models LLM Interpretability
Anthropic
Reasoning Fails Where Step Flow Breaks
Apr 8, 2026Xiaoyu Xu, Yulan Pan, Xiaosong Yuan +4LLM Reasoning Large Reasoning Models
Shanghai Jiao Tong University · Fuzhou University · Jilin University
Latent Structure of Affective Representations in Large Language Models
Apr 8, 2026Benjamin J. Choi, Melanie WeberEmotion Representation in Language Models Representation Geometry in Language Models
Harvard University
Distributed Interpretability and Control for Large Language Models
Apr 7, 2026Dev Arpan Desai, Shaoyi Huang, Zining ZhuLLM Interpretability Activation Steering
Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
Apr 7, 2026Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1LLM Evaluation Long-Context Language Model Evaluation
Information Science Cornell University · School of Communication and Culture Aarhus University
The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
Apr 7, 2026Michael Rizvi-Martel, Guillaume Rabusseau, Marius MosbachContinuous Latent Reasoning LLM Interpretability
Mila – Quebec AI Institute · Universit´e de Montr´eal · McGill University
MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models
Apr 7, 2026Ji-jun Park, Soo-joon Choi, Jiwon Jeong +2LLM Interpretability Language Model Probing
Dongguk University
How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism
Apr 7, 2026Elisabetta Rocchetti, Alfio FerraraLLM Interpretability Instruction Following
Department of Computer Science, Università degli Studi di Milano
Disentangling MLP Neuron Weights in Vocabulary Space
Apr 7, 2026Asaf Avrahamy, Yoav Gur-Arieh, Mor GevaLLM Interpretability Transformer Interpretability
Blavatnik School of Computer Science and AI, Tel Aviv University
What are They Thinking? Delineation, Probing, and Tracking of Concepts in LLMs
Apr 7, 2026Mohamed Abdelwahab, Michelle Yu Collins, Sihan Chen +5Linear Probing LLM Interpretability
The Edward S. Rogers Sr. Department of Electrical and Computer Engineering University of Toronto
Attribution-Driven Explainable Intrusion Detection with Encoder-Based Large Language Models
Apr 7, 2026Umesh Biswas, Shafqat Hasan, Syed Mohammed Farhan +2Explainable Intrusion Detection LLMs for Cybersecurity
Computer Science & Engineering, Mississippi State University
Are Latent Reasoning Models Easily Interpretable?
Apr 6, 2026Connor Dilgren, Sarah WiegreffeImplicit Reasoning in Language Models LLM Reasoning
Department of Computer Science, University of Maryland, College Park, MD, USA

Apr 15, 2026J/K move · Enter open · S save

Applied Explainability for Large Language Models: A Comparative Study

Venkata Abhinandan Kancharla

NRI Institute of Technology

LLM Interpretability Explainability Evaluation Transformer Interpretability Feature Attribution Faithfulness of Language Model Explanations

Open paper AskPDF arXiv

Large language models (LLMs) achieve strong performance across many natural language processing tasks, yet their decision processes remain difficult to interpret. This lack of transparency creates challenges for trust, debugging, and deployment in real-world systems. This paper presents an applied comparative study of three explainability techniques: Integrated Gradients, Attention Rollout, and SHAP, on a fine-tuned DistilBERT model for SST-2 sentiment classification. Rather than proposing new methods, the focus is on evaluating the practical behavior of existing approaches under a consistent and reproducible setup. The results show that gradient-based attribution provides more stable and intuitive explanations, while attention-based methods are computationally efficient but less aligned with prediction-relevant features. Model-agnostic approaches offer flexibility but introduce higher computational cost and variability. This work highlights key trade-offs between explainability methods and emphasizes their role as diagnostic tools rather than definitive explanations. The findings provide practical insights for researchers and engineers working with transformer-based NLP systems. This is a preprint and has not undergone peer review.