LLM Interpretability
LLM: Large Language Model
Overview
Large language model interpretability asks what information models represent, how it moves through layers and components, and which computations cause particular answers or behaviors. Researchers use probes and representation geometry, sparse autoencoders, attribution and circuit tracing, activation patching or ablation, and steering to study factual recall, mathematical reasoning, hallucination, refusal, and in-context learning. These methods can reveal distributed circuits and layer-wise signals, and sometimes support targeted interventions without retraining. A central challenge is that information being decodable—or a direction being steerable—does not prove the model uses it; causal faithfulness, robustness across prompts and model families, and reliable evaluation remain open tensions.
Latest papers
841 papers55 papers in the last four weeks, down 36% on the four weeks before. 0.4% of all new papers.
Tracing Moral Foundations in Large Language Models
Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders
Bridging Mechanistic Interpretability and Prompt Engineering with Gradient Ascent for Interpretable Persona Control
AlignSAE: Concept-Aligned Sparse Autoencoders
Latent Debate: A Surrogate Framework for Interpreting LLM Thinking towards Binary Decisions
Rep2Text: Decoding Full Text from a Single LLM Token Representation
TOPol: Reconstructing and Interpreting Multidimensional Semantic Polarity Fields
Fine-grained Analysis of Brain-LLM Alignment through Input Attribution
Base Models Know How to Reason, Thinking Models Learn When
Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures
Vis-CoT: A Human-in-the-Loop Framework for Interactive Visualization and Intervention in LLM Chain-of-Thought Reasoning
A Financial Brain Scan of the LLM
Unraveling the cognitive patterns of Large Language Models through module communities
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
BiasGym: A Simple and Generalizable Framework for Analyzing and Removing Biases through Injection
BiasGym for reliably injecting, analyzing, and mitigating conceptual associations of biases within LLMs. BiasGym consists of two modules: Inject, which injects specific biases into the model via token-based fine-tuning while keeping the model frozen, followed by two debiasing methods that leverage these injected signals to identify and reliably suppress (Scope) or Steer the components responsible for biased behavior. Our framework enables consistent bias elicitation for better localization of bias conceptual association in the model space, supports targeted debiasing without degrading performance on downstream tasks, and generalizes to biases unseen during fine-tuning. We demonstrate the effectiveness of our proposed framework in reducing real-world stereotypes (e.g., people from Italy being `reckless drivers'), showing its utility for both safety interventions and interpretability research.Too Categorical to be Human: Emotion Concepts in LLMs and Humans
LLMs Encode Harmfulness and Refusal Separately
Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
On the Effect of Uncertainty on Layer-wise Inference Dynamics
The Trilemma of Truth in Large Language Models
Cross-Layer Discrete Concept Discovery for Interpreting Language Models
InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
Localizing Persona Representations in LLMs
When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoning
Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
irrelevant'' contextual information in the input prompt. Specifically, LMs assign significantly higher logits (or probabilities) to any tokens that have previously appeared in the context prompt, even for random tokens. This suggests that contextual entrainment is a mechanistic phenomenon, occurring independently of the relevance or semantic relation of the tokens to the question or the rest of the sentence. We find statistically significant evidence that the magnitude of contextual entrainment is influenced by semantic factors. Counterfactual prompts have a greater effect compared to factual ones, suggesting that while contextual entrainment is a mechanistic phenomenon, it is modulated by semantic factors. We hypothesise that there is a circuit of attention heads -- the entrainment heads -- that corresponds to the contextual entrainment phenomenon. Using a novel entrainment head discovery method based on differentiable masking, we identify these heads across various settings. When we turn off'' these heads, i.e., set their outputs to zero, the effect of contextual entrainment is significantly attenuated, causing the model to generate output that capitulates to what it would produce if no distracting context were provided. Our discovery of contextual entrainment, along with our investigation into LM distraction via the entrainment heads, marks a key step towards the mechanistic analysis and mitigation of the distraction problem.