Circuits in Language Models

Follow this topic Ask about this topic

Circuits in language models are groups of attention heads, MLP neurons, and learned features whose interactions support behaviors such as induction, factual recall, arithmetic, refusal, and multilingual routing. Researchers locate them with activation patching and ablation, attribution graphs, and sparse autoencoders or transcoders, then test their causal roles on controlled tasks such as indirect object identification and greater-than. Some computations appear compact and reusable, while others are distributed, context-dependent, or reorganized by architecture and post-training. A central challenge is showing that a discovered circuit is causally faithful and task-specific rather than an artifact of prompts, analysis choices, or intervention side effects.

Latest papers

122 papers27 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models
Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLLM Pruning Attention Head Pruning
Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka
The Emergence and Sparsification of a Syntactic Circuit
Oct 23, 2026Qing Yao, Sasha Boguraev, Tiago Pimentel +1Circuits in Language Models Syntactic Generalization in Language Models
The University of Texas at Austin · ETH Zurich
Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery
Oct 23, 2026Marek Masiak, Andrzej Szablewski, Nicola Cancedda +1Sparse Fine-Tuning LLM Interpretability
Department of Computer Science, University of Cambridge · Neuralk-AI
Concentrated Causal Influence, Distributed Diagnostic Information: Cognitive Impairment Decisions in Large Audio-Language Models
Oct 23, 2026Xiutian Zhao, Luqi Sun, Björn Schuller +1Audio-Language Models Circuits in Language Models
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
How Depth Growth and Looping Shape In-Context Learning in Language Models
Oct 22, 2026Ferdinand Kapl, Kaitlin Maile, Stefan Bauer +1In-Context Learning Parameter Sharing
Technical University of Munich · Helmholtz AI, Munich · Munich Center for Machine Learning (MCML) +1
Language Impairment as a Lens onto Large Language Models
Oct 20, 2026Muxuan He, Wang Bill Zhu, Yue Chen +5LLM Interpretability Circuits in Language Models
University of Southern California · University of California, Berkeley · Independent Researcher +1
Are Linguistic and Arithmetic Circuits Dissociated in LLMs?
Oct 20, 2026Riku Kisako, Tatsuki Kuribayashi, Ryohei SasanoCircuits in Language Models Numerical Reasoning in Language Models
Nagoya University, Japan · MBZUAI, UAE · Tohoku University, Japan
Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
Oct 16, 2026Prakhar Gupta, Terry Jingchen Zhang, Zhijing JinCognitive Bias in Language Models Circuits in Language Models
University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford +2
Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation
Oct 15, 2026Atai Waris Khan, Abdullah Mazhar, Raghu Kumar +1Multilingual Language Models Low-Resource MT
Indraprastha Institute of Information Technology Delhi
Do Emergent Misalignment Models Share a Circuit?
Oct 14, 2026Muhammad Haseeb, Muhammad TahirEmergent Misalignment in Language Models Circuits in Language Models
Lahore University of Management Sciences (LUMS)
Interpreting Language Model Parameters
Oct 14, 2026Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin +5Mechanistic Interpretability Transformer Interpretability
Goodfire · MATS · Independent
Causal Syntax Circuits in CodeLMs: Budget and Probe Decide What Looks Localised
Oct 14, 2026Harsh Kumar, Tanmay Shailesh Joshi, Rahul Maity +1Circuits in Language Models Code Language Models
National Institute of Technology Agartala
From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models
Oct 13, 2026Jinghan Zheng, Jiajun ShiCircuits in Language Models Causal Interventions in Language Models
Dalian University of Technology
Task Context Shapes Component Roles in Language Models
Oct 9, 2026Sovesh Mohapatra, Dani S BassettLLM Interpretability Circuits in Language Models
University of Pennsylvania · Yale University
Language Models Have Interpretable Narrative Mechanisms
Oct 9, 2026David Yiyang Liu, Aditya Joshi, Paul Dawson +1LLM Interpretability Circuits in Language Models
UNSW Sydney · McGill University
Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs
Oct 7, 2026Seunghan Kim, Minyeong Choe, Hyunil Kim +1Cross-Lingual Reasoning in Language Models Multilingual Language Models
Chosun University · Soongsil University
SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
Oct 7, 2026Miao Yu, Hao Huang, Lu Yuan +3Refusal Mechanisms in Language Models LLM Safety Alignment
The University of Hong Kong (HKU) · Chinese Academy of Sciences (CAS) · Information Engineering University (IEU) +1
Finding the Heads and the Neurons Responsible for Network Information Retrieval in Language Models
Oct 6, 2026Md Abdul Kadir, Md Mohasin Hossain, Daniel SonntagAttention Head Analysis LLM Interpretability
University of Oldenburg, Oldenburg, Germany · German Research Center for Artificial Intelligence · Saarland University, Saarbrucken, Germany
Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?
Oct 2, 2026Tejas Dahiya, Cole BlondinCircuits in Language Models LLM Training
University of Chicago · Independent Researcher
Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models
Oct 2, 2026Sankaran Vaidyanathan, Rafal Urbaniak, Emily Bunnapradist +2Circuits in Language Models Mechanistic Interpretability
Basis Research Institute · University of Massachusetts Amherst · MIT
Every Ablation Is a Dose: Counterweights and the Semblance of Self-Repair
Oct 1, 2026Areeb Ahmad, Pratinav Seth, Vinay Kumar SankarapuMechanistic Interpretability Circuits in Language Models
Lexsi Labs
Component and Dimension Sparsity in Transformer Refusal Mechanisms
Sep 30, 2026Vincent Siu, Glenn Grant-Richards, Vlad Pavlovich +3Refusal Behavior in Language Models Transformer Interpretability
UC Santa Cruz · UCLA · UC Berkeley
Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions
Sep 30, 2026Beiming Liu, Minjie ChenCircuits in Language Models Causal Interpretability
PetroChina Southwest Oil and Gasfield Company, China
Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space
Sep 29, 2026Tue M. Cao, Lisiane Pruinelli, My T. ThaiMechanistic Interpretability Circuits in Language Models
Department of Computer and Information Science and Engineering, University of Florida, Gainesville, FL 32611, USA · College of Nursing, University of Florida, Gainesville, FL 32611, USA
RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learning
Sep 29, 2026Chuanpu Liu, Miao Yu, Yikai Cai +5Circuits in Language Models LLM Reasoning
Beijing University of Posts and Telecommunications · University of Hong Kong · Nanyang Technological University +1
In-Context Learning Amplifies a Latent Symbolic Circuit
Sep 28, 2026Melissa WesselLLM Interpretability In-Context Learning
Independent Researcher.
Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?
Sep 28, 2026Li Zhang, Chuqin Geng, Mark Zhang +4Faithfulness of Language Model Explanations Mechanistic Interpretability
University of Toronto · McGill University · Tsinghua University
Social Circuits behind Multi-agent Echo Chambers
Sep 28, 2026Chuiyang Meng, Wenlu Yu, Ming Tang +1Multi-Agent LLM Systems Multi-Agent Systems
Simon Fraser University · University of Alberta · Southern University of Science and Technology
Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training
Sep 27, 2026Adam ElimadiMechanistic Interpretability Circuits in Language Models
Independent Cambridge, MA
What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use
Sep 26, 2026Yixiao Chen, Ke Cheng, Jiangtao Guan +5Language Model Pretraining Factual Knowledge in Language Models
AI Data, Tencent · Harvard University · AMS, Tencent

Oct 27, 2026J/K move · Enter open · S save

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models

Nabeelah Faumi, Deshan Sumanathilaka

Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

LLM Pruning Attention Head Pruning LLM Interpretability Circuits in Language Models

Open paper AskPDF arXiv

Pruned language models are judged almost entirely by accuracy and perplexity, which assumes that preserved output behaviour implies preserved internal computation. We test this assumption with a factorial pruning study on GPT-2, crossing two importance criteria (magnitude, random), two component types (attention heads, MLP neurons) and three sparsity levels (3%, 50%, 70%). Using the fully characterised indirect object identification (IOI) circuit as ground truth, we measure circuit integrity with activation patching, path patching and direct logit attribution. We find two qualitatively distinct failure modes. Magnitude head pruning >= zeroes the circuit: output-facing heads are removed, and the model falls to near-chance. Magnitude neuron pruning at 30% inverts it: all 26 circuit heads remain intact, but Name Mover contributions flip sign, and the model systematically prefers the wrong name (17.7% accuracy). Perplexity ranks it as less damaged than random head pruning (74.6 vs 140.1). A replication of the Greater-Than circuit shows that the critical component type depends on the circuit's architecture. These results show that output metrics and structural checks can both miss mechanistic damage, motivating circuit-level causal analysis as a complementary evaluation of compressed language models.