Activation Patching

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 14

All topics

Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions
Sep 30, 2026Beiming Liu, Minjie ChenCircuits in Language Models Causal Interpretability
PetroChina Southwest Oil and Gasfield Company, China
Social Circuits behind Multi-agent Echo Chambers
Sep 28, 2026Chuiyang Meng, Wenlu Yu, Ming Tang +1Multi-Agent LLM Systems Multi-Agent Systems
Simon Fraser University · University of Alberta · Southern University of Science and Technology
StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions
Sep 1, 2026Chao Gao, Haijiang Liu, Qiyuan Li +3Prompt Sensitivity Multiple-Choice Question Answering
Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands · School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China · Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, China
Cross-Layer Interaction under Weight-Space Ablation: A Closed-Form Attention Jacobian Bound and a Test on a Real Pretrained Model
Aug 4, 2026Abdallah KhemaisTransformer Interpretability Weight-Space Analysis
ISITCOM, University of Sousse
A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation
Aug 4, 2026Abdallah KhemaisTransformer Interpretability Weight-Space Analysis
ISITCOM, University of Sousse
Cost Accounting for Reactive Computational Graphs: Exhaustive Sweeps, Sequential Mutation, and the Backward-Locality Gap
Jul 18, 2026Abdallah KhemaisActivation Patching
ISITCOM, University of Sousse
The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching
Jun 25, 2026Sankaran Vaidyanathan, David Arbour, Aaron Mueller +2Causal Effect Estimation Causal Attribution
University of Massachusetts Amherst · Adobe Research · Boston University
Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process
Jun 19, 2026Hail Hochman, Natalie Shapira, Yoav GoldbergLLM Interpretability Factual Knowledge in Language Models
Bar-Ilan University · Northeastern University
Beyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Components
Jun 12, 2026Yifeng Guo, Jin-Hong Du, Xiang ChenMechanistic Interpretability Causal Interventions in Language Models
St. Jude Children’s Research Hospital Memphis, TN · The University of Hong Kong Hong Kong, CN
Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers
Jun 6, 2026Philip QuirkeAttention Head Analysis Mechanistic Interpretability
Martian
When Attribution Patching Lies: Diagnosis and a Second-Order Correction
Jun 5, 2026Luyang Zhang, Jialu WangGradient-Based Attribution LLM Interpretability
1Carnegie Mellon University · University of California, Santa Cruz
Transformer Field Theory: A Response-Theoretic Approach to Mechanistic Interpretability
May 24, 2026David N. Olivieri, Antonio F. Pérez RodríguezTransformer Interpretability Mechanistic Interpretability
Departamento de Informática, Universidade de Vigo, Spain · Independent Researcher, Spain
Instruction Tuning Changes How Upstream State Conditions Late Readout: A Cross-Patching Diagnostic
May 8, 2026Yifan ZhouTransformer Interpretability Instruction Tuning
University of California, Los Angeles
Patch-Effect Graph Kernels for LLM Interpretability
May 7, 2026Ruben Fernandez-Boullon, David N. OlivieriGraph Representation Learning Mechanistic Interpretability
Department of Computer Science, University of Vigo, Ourense 32004, Spain

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions

Beiming Liu, Minjie Chen

PetroChina Southwest Oil and Gasfield Company, China

Circuits in Language Models Causal Interpretability Causal Interventions in Language Models Activation Patching

Open paper AskPDF arXiv

Causal interventions such as activation patching and distributed alignment search (DAS) are the main tool for making mechanistic claims about neural networks. Recent work showed that these interventions routinely push representations off the model's natural distribution, and that such divergence is sometimes harmless and sometimes pernicious: it can recruit pathways the model never uses on natural inputs, so that an intervention produces the expected answer through the wrong mechanism. No method currently tells the two cases apart. We make this question testable by planting hidden pathways inside pretrained language models; the pathways are silent on every benchmark prompt by construction, so which interventions depend on them is known exactly. Across 72 configurations and 100,800 interventions on GPT-2 small, we find three things. (i) Nearest-neighbour and local-PCA distances at the intervention site, as used in prior work, score below chance (AUROC 0.35-0.47) at picking out interventions that give the right answer through a planted pathway. (ii) Hidden-Pathway Contribution (HPC), a label-free test that clamps downstream units to the regime of natural runs with the same output and measures how much of the decision disappears, flags pathway-dominated interventions with AUROC >= 0.99 when the pathway shows up as unit-level out-of-regime activity, but fails when every unit stays within its natural range, which we identify as the open problem. (iii) Optimised interventions actively seek hidden pathways: on a gender task, DAS routes 90-95% of its successes through planted pathways for three of four families, and a downstream on-manifold penalty cuts this share to under 5% at a cost of 6-11 points of success rate. In unmodified GPT-2, successful interventions show almost no unit-level out-of-regime reliance.

Momentum

Related topics

Latest papers 14