LLM Interpretability

LLM: Large Language Model

Large language model interpretability asks what information models represent, how it moves through layers and components, and which computations cause particular answers or behaviors. Researchers use probes and representation geometry, sparse autoencoders, attribution and circuit tracing, activation patching or ablation, and steering to study factual recall, mathematical reasoning, hallucination, refusal, and in-context learning. These methods can reveal distributed circuits and layer-wise signals, and sometimes support targeted interventions without retraining. A central challenge is that information being decodable—or a direction being steerable—does not prove the model uses it; causal faithfulness, robustness across prompts and model families, and reliable evaluation remain open tensions.

Latest papers

781 papers82 papers in the last four weeks, up 12% on the four weeks before. 0.5% of all new papers.

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models
Oct 29, 2026Aishik Nagar, Cheston TanRL for Language Model Reasoning LLM Interpretability
Carnegie Mellon University · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR) · Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)
Spokes, not a Ring: Revisiting the Weekday Manifold in Language Models
Oct 27, 2026Diksha Gupta, Rieke Fruengel, Stefan HeimersheimRepresentation Geometry in Language Models LLM Interpretability
Pivotal Research
Mechanistic Failure Modes of Weight Pruning in Transformer Language Models
Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLLM Pruning Attention Head Pruning
Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka
FinLLM-KG: Knowledge Graph-Enhanced LLMs for Financial Sentiment Analysis in Algorithmic Trading
Oct 26, 2026Giorgos Iacovides, Oliver Hannibal, Wuyang Zhou +1Financial Sentiment Analysis Knowledge Augmentation for Language Models
Department of Electrical and Electronic Engineering, Imperial College London, UK
Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery
Oct 23, 2026Marek Masiak, Andrzej Szablewski, Nicola Cancedda +1Sparse Fine-Tuning LLM Interpretability
Department of Computer Science, University of Cambridge · Neuralk-AI
Natural Language Autoencoders Identify Without Explaining
Oct 23, 2026Kristina Šekrst, Barbara GeldLLM Interpretability Faithfulness of Language Model Explanations
A Mathematical Theory of the Jacobian Lens
Oct 22, 2026Mitchell Ostrow, Daoyuan Qian, Toni Liu +1LLM Interpretability Representation Geometry in Language Models
Department of Brain and Cognitive Sciences Massachusetts Institute of Technology · Department of Physics Cornell University
The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models
Oct 22, 2026Shihui Li, Mette Andersen, Martha LewisFigurative Language Understanding Representation Geometry in Language Models
Institute for Logic, Language, and Computation, University of Amsterdam
Verbalizing Activation Differences with Natural Language Autoencoders
Oct 22, 2026Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli +3LLM Interpretability Autoencoders
University of Southern Denmark
Meaning Contracts: Local Semantic Geometry in Large Language Models
Oct 22, 2026Andrés Daniel Godoy OrtizRepresentation Geometry in Language Models LLM Interpretability
Imperial College London
Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models
Oct 22, 2026Qince Luan, Jianhui Xu, Yang XuCausal Interventions in Language Models LLM Interpretability
Southern University of Science and Technology (SUSTech) · Independent Scholar
Probing and Steering for Safety-Relevant Behaviours in Large Language Models
Oct 21, 2026Nimun Bajwa, Massimo PoesioActivation Steering Language Model Probing
Queen Mary University of London
Language Impairment as a Lens onto Large Language Models
Oct 20, 2026Muxuan He, Wang Bill Zhu, Yue Chen +5LLM Interpretability Circuits in Language Models
University of Southern California · University of California, Berkeley · Independent Researcher +1
Where Do Legal Factors Live in LLMs?
Oct 20, 2026Li Zhang, Kevin AshleyLLM Interpretability Legal Reasoning
University of Pittsburgh, USA
Are Linguistic and Arithmetic Circuits Dissociated in LLMs?
Oct 20, 2026Riku Kisako, Tatsuki Kuribayashi, Ryohei SasanoCircuits in Language Models Numerical Reasoning in Language Models
Nagoya University, Japan · MBZUAI, UAE · Tohoku University, Japan
How Fine-Tuning Objectives Shape Layer-Wise Information in LLM Hallucination Detection
Oct 20, 2026Kaiyang Wan, Forrest Sheng Bao, Amin Ahmad +1Fine-Tuning Language Models LLM Hallucination Detection
INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · PAIR Systems, Inc., Mountain View, California
From Encoding to Generation: Towards Empathetic Large Language Models
Oct 20, 2026Samhita Bollepally, Takashi Yamauchi, Marissa WestEmotion Representation in Language Models Emotion-Aware Dialogue Generation
Correcting Attribution Patching with a Few Exact Measurements
Oct 19, 2026Luyang Zhang, Minxing Zheng, Jialu WangLLM Interpretability Activation Patching
Carnegie Mellon University · University of California, Santa Cruz
Shaped SAEs: Expanding the Toolkit for Mechanistic Interpretability
Oct 19, 2026Oveys Delafrooz Noroozi, Bhagyashree Puranik, Upamanyu MadhowLLM Interpretability Sparse Autoencoders
University of California, Santa Barbara
Predicting Linear Relational Decodability Without the Relation
Oct 19, 2026Jonas Rohweder, Adrian Mülthaler, Faridoun Mehri +2LLM Interpretability Representation Learning
Center for Information and Language Processing (CIS), LMU Munich · Ubiquitous Knowledge Processing Lab (UKP Lab), TU Darmstadt · Munich Center for Machine Learning (MCML) +1
DeltaTrace: Efficient Signed Attribution for Reasoning Language Models
Oct 19, 2026Chen Liang, Daoping Wu, Donghua Yang +4Feature Attribution LLM Interpretability
Harbin Institute of Technology · Zhejiang Lab
From Stories to Features: Scaling Concept Vectors for AI Alignment & Interpretability
Oct 17, 2026Dmitry Scherbakov, Oleg Chernikov, Victoria Khamash +5LLM Interpretability LLM Alignment
Do LMs Know What Variables Do? Interpreting Code Representations via Variable Roles
Oct 16, 2026Célian Vasson, Daniil Ognev, Kentaro Inui +1LLM Interpretability Code Language Models
Sorbonne Université · MBZUAI · RIKEN AIP +1
Verbalising Model's Computation for Chain-of-Thought Faithfulness Detection
Oct 16, 2026Yujing Wang, Yulan He, Wenya Wang +1CoT Faithfulness Faithfulness of Language Model Reasoning
King’s College London · Nanyang Technological University
What LLMs Do in the Shadows: Reading Context Reliance from the Unembedding Dark Space
Oct 16, 2026Alexandra Kuleshova, Andrey Volodichev, Daria Kotova +1LLM Hallucination Detection Faithfulness of Language Model Reasoning
Applied AI Institute
Do Knowledge and Reasoning Oriented Benchmarks Induce Stable Representation Signatures in LLMs?
Oct 16, 2026Parsa Rahimi, Azim Dehghani AmirabadLLM Interpretability Language Model Probing
EPFL, Idiap Research Institute · Independent Researcher
Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics
Oct 15, 2026Jian Gu, Hongyu Zhang, Chunyang Chen +1LLM Interpretability Mechanistic Interpretability
Monash University · Chongqing University · Technical University of Munich
Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
Oct 15, 2026Sekh Mainul Islam, Pepa Atanasova, Isabelle AugensteinMixture-of-Experts Expert Specialization Personalized Language Models
University of Copenhagen
Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
Oct 15, 2026Camila Kolling, Mariya TonevaNeural Encoding LLM Interpretability
Max Planck Institute for Software Systems (MPI-SWS)
Isolating a Language Model's Judgment with a Label-Free Subspace
Oct 15, 2026Chaemin Jang, Donggyu Lee, Jeesoo Kim +2LLM Interpretability Representation Disentanglement
Korea Advanced Institute of Science and Technology (KAIST)

Oct 29, 2026J/K move · Enter open · S save

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

Aishik Nagar, Cheston Tan

Carnegie Mellon University · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR) · Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)

RL for Language Model Reasoning LLM Interpretability

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly described as acquiring "reasoning" skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analyzable logical domain, namely Tic-Tac-Toe, and trace how internal representations evolve across reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting the emergence of game reasoning. Mechanistic interpretability, however, paints a different picture: features extracted with sparse autoencoders (SAEs) reveal that models primarily adapt to rely more heavily on information already explicit in the prompt, such as whose turn it is, game progression and board occupancy. In comparison, high-level concepts like board symmetries and strategic forks remain weakly represented, raising concerns that reasoning benchmarks overestimate model abilities. This suggests that RL-driven "reasoning" may conflate task-specific adaptations to input information with reasoning ability. We provide, for the first time: (i) a systematic interpretability pipeline tracing representation dynamics across RL training in small-scale LLMs; (ii) an extension of SAE-based feature discovery to hypothesis-driven testing in a finite logical domain; and (iii) an interpretability-based demonstration that RL post-training neither adds nor removes concept information but reorganizes which features dominate the representation, with causal interventions locating the decision downstream of it, in the late-layer state.