Causal Interventions in Language Models

Follow this topic Ask about this topic

This field asks which internal states, features, heads, neurons, or context elements actually influence a language model’s decisions, rather than merely correlate with them. Researchers use counterfactual prompt and evidence edits, activation patching and ablation, causal tracing, sparse-autoencoder features, and inference-time steering or weight edits to test mechanisms and control behavior. Applications include checking whether chain-of-thought and retrieved evidence drive answers, and modifying refusal, hallucination, bias, tool use, or reasoning while preserving unrelated capabilities. Results show that readable representations are often not causally used or easily steered; effects vary by layer, task, model, and context, while redundancy and collateral effects make reliable, selective interventions a continuing challenge.

Latest papers

489 papers83 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis
Mar 12, 2026Abdullah Al Mofael, Lisa M. Kuhn, Ghassan Alkadi +1Language Modeling Circuits in Language Models
∗Department of Computer Science, Southeastern Louisiana University, Hammond, LA, USA · †Department of Mathematics, Southeastern Louisiana University, Hammond, LA, USA
Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviors
Mar 12, 2026Pengcheng Wen, Yanxu Zhu, Jiapeng Sun +5CoT Reasoning LLM Alignment
Hong Kong University of Science and Technology · Beijing Jiaotong University
Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning
Mar 11, 2026Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza FerozLLM Reasoning with Graphs Causal Interpretability
Daffodil International University, Dhaka, Bangladesh · New York University, Brooklyn, NY, USA
Do Large Language Models Get Caught in Hofstadter-Mobius Loops?
Mar 10, 2026Jaroslaw HryszkoLLM Safety Framing Effects in Language Models
Faculty of Mathematics and Computer Science, Jagiellonian University, Krak´ow, Poland
From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions
Mar 9, 2026Rishab Alagharu, Ishneet Sukhvinder Singh, Shaibi Shamsudeen +2Language Model Steering LLM Refusal Behavior
Algoverse AI Research · Carnegie Mellon University · University of Maryland
Engineering Verifiable Modularity in Transformers via Per-Layer Supervision
Mar 8, 2026J. Clayton KerceTransformer Interpretability Mechanistic Interpretability
Georgia Tech Research Institute
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA
Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
Mar 2, 2026Kyle Cox, Darius Kianersi, Adrià Garriga-AlonsoFaithfulness of Language Model Explanations LLM Interpretability
Independent · Cambridge Boston Alignment Initiative (CBAI)
Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
Feb 24, 2026Shiwei Tan, Hengyi Wang, Weiyi Qin +3Object Hallucination in VLMs VLM Hallucination Mitigation
Rutgers University · Meta Ranking AI
Steering at the Source: Style Modulation Heads for Robust Persona Control
Feb 24, 2026Yoshihiro Izawa, Gouki Minegishi, Koshi Eguchi +2Language Model Steering Causal Interventions in Language Models
The University of Tokyo, Tokyo, Japan · National Institute of Informatics Research and Development Center for Large Language Models, Japan
Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations
Feb 22, 2026Ahmed Karim, Fatima Sheaib, Zein Khamis +3Hallucination in Language Models LLM Interpretability
Hassana Labs · University of Oxford
Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions
Feb 22, 2026Yao YanNumerical Reasoning in Language Models Circuits in Language Models
Chongqing Normal University Chongqing Key Lab of Cognitive Intelligence and Intelligent Finance Chongqing, China
Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency
Feb 18, 2026Victoria Lin, Xinnuo Xu, Rachel Lawrence +4Counterfactual Reasoning in Language Models Causal Reasoning in Language Models
Carnegie Mellon University · Microsoft Research Cambridge · Microsoft Research India
Causality is Key for Interpretability Claims to Generalise
Feb 18, 2026Shruti Joshi, Aaron Mueller, David Klindt +3LLM Interpretability Causal Interpretability
Mila - Québec AI Institute & Université de Montréal · Boston University · Cold Spring Harbor Laboratory +1
Activation Steering via Generative Causal Mediation
Feb 17, 2026Aruna Sankaranarayanan, Amir Zur, Atticus Geiger +1Language Model Steering Causal Interventions in Language Models
CSAIL, MIT · Pr(AI)2R · Stanford University +1
Causal Effect Estimation with Latent Textual Treatments
Feb 17, 2026Omri Feldman, Amar Venugopal, Jann Spiess +1Causal Effect Estimation Causal Inference
School of Computer Science and Engineering, The Hebrew University of Jerusalem · Department of Economics, Stanford University · Graduate School of Business, Stanford University
Directional Reasoning Trajectory Change (DRTC): Identifying Critical Trace Segments in Reasoning Models
Feb 17, 2026Waldemar ChangLLM Interpretability Reasoning Trace Analysis
Johns Hopkins University
Patches of Nonlinearity: Instruction Vectors in Large Language Models
Feb 8, 2026Irina Bigoulaeva, Jonas Rohweder, Subhabrata Dutta +1Transformer Interpretability LLM Interpretability
Ubiquitous Knowledge Processing Lab (UKP Lab) · Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany
ReBeCA: Causality-Inspired Behavior Analysis for Understanding the Self-Reflection of Language Models
Feb 6, 2026Tianqiang Yan, Lizhen Qu, Sihan Shang +6LLM Self-Refinement Causal Interpretability
Monash University · Harbin Institute of Technology, Shenzhen · Hainan University +2
Causal Front-Door Adjustment for Robust Jailbreak Attacks on LLMs
Feb 5, 2026Yao Zhou, Zeen Song, Wenwen Qiang +2Adversarial Attacks on LLMs LLM Jailbreak Attacks
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Towards Isolated Interventions via Almost Orthogonal Features in Language Models
Feb 4, 2026Moritz Miller, Florent Draye, Bernhard SchölkopfDisentangled Representation Learning Representation Learning
Max Planck Institute for Intelligent Systems, Tübingen, Germany · ETH Zurich, Switzerland · Max Planck ETH Center for Learning Systems +1
$C$ - $ΔΘ$ : Circuit-Restricted Weight Arithmetic for Selective Refusal
Feb 4, 2026Aditya Kasliwal, Pratinav Seth, Vinay Kumar SankarapuLLM Refusal Behavior Causal Interventions in Language Models
Lexsi Labs
Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models
Feb 3, 2026Anish Sathyanarayanan, Aditya Nagarsekar, Aarush RathoreCoT Faithfulness Faithfulness of Language Model Explanations
BITS Pilani, K. K. Birla Goa Campus
There Is More to Refusal in Large Language Models than a Single Direction
Feb 2, 2026Faaiz Joad, Majd Hawasly, Sabri Boughorbel +2LLM Interpretability LLM Refusal Behavior
Qatar Computing Research Institute, HBKU, Doha, Qatar
Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoning
Feb 2, 2026Yadong Wang, Haodong Chen, Yu Tian +3Sparse Autoencoders Continuous Latent Reasoning
Nanjing University of Aeronautics and Astronautics · State Key Laboratory of Ocean Sensing, ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou, 311215, China · Security Capability Center
Cross-Lingual Activation Steering for Multilingual Language Models
Jan 23, 2026Rhitabrat Pokharel, Ameeta Agrawal, Tanay NagarMultilingual Language Models Language Model Steering
Department of Computer Science, Portland State University, USA · Independent Researcher
SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics
Jan 22, 2026Silvia Casola, Ryan Soh-Eun Shim, Felicia Körner +2Summarization Evaluation Language Model Steering
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Language Science and Technology, Saarland University, Germany
Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues
Jan 19, 2026Shiyue Hu, Ruizhe Li, Yanjun GaoLLM Interpretability Mechanistic Interpretability
University of Colorado Anschutz · University of Colorado Boulder · University of Aberdeen
To Copy or Not to Copy: Copying Is Easier to Induce Than Recall
Jan 17, 2026Mehrdad Farahani, Franziska Penzkofer, Richard JohanssonKnowledge Conflicts in Language Models LLM Interpretability
1Chalmers University of Technology · University of Gothenburg
Steering Language Models Before They Speak: Logit-Level Interventions
Jan 16, 2026Hyeseon An, Shinwoo Park, Hyundong Jin +1Language Model Steering Controllable Text Generation
Department of Computer Science, Yonsei University, Seoul, South Korea

Mar 12, 2026J/K move · Enter open · S save

Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis

Abdullah Al Mofael, Lisa M. Kuhn, Ghassan Alkadi, Kuo-Pao Yang

∗Department of Computer Science, Southeastern Louisiana University, Hammond, LA, USA · †Department of Mathematics, Southeastern Louisiana University, Hammond, LA, USA

Language Modeling Circuits in Language Models Transformer Interpretability Causal Interventions in Language Models

Open paper AskPDF arXiv

Negation remains a persistent challenge for modern language models, often causing reversed meanings or factual errors. In this work, we conduct a causal analysis of how GPT-2 Small internally processes such linguistic transformations. We examine its hidden representations at both the layer and head level. Our analysis is based on a self-curated 12,000-pair dataset of matched affirmative and negated sentences, covering multiple linguistic templates and forms of negation. To quantify this behavior, we define a metric, the Negation Effect Score (NES), which measures the model's sensitivity in distinguishing between affirmative statements and their negations. We carried out two key interventions to probe causal structure. In activation patching, internal activations from affirmative sentences were inserted into their negated counterparts to see how meaning shifted. In ablation, specific attention heads were temporarily disabled to observe how logical polarity changed. Together, these steps revealed how negation signals move and evolve through GPT-2's layers. Our findings indicate that this capability is not widespread; instead, it is highly concentrated within a limited number of mid-layer attention heads, primarily within layers 4 to 6. Ablating these specific components directly disrupts the model's negation sensitivity: on our in-domain, ablation increased NES (indicating weaker negation sensitivity), and re-introducing cached affirmative activations (rescue) increased NES further, confirming that these heads carry affirmative signal rather than restoring baseline behavior. On xNot360, ablation slightly decreased NES and rescue restored performance above baseline. This pattern demonstrates that these causal patterns are consistent across various negation forms and remain detectable on the external xNot360 benchmark, though with smaller magnitude.