Addressing the Reasoning Gap: Mechanistic Circuit-Based Knowledge Editing in Large Language Models

Tianyi Zhao; Yinhan He; Wendy Zheng; Chen Chen

cs.CLApr 7, 2026

Addressing the Reasoning Gap: Mechanistic Circuit-Based Knowledge Editing in Large Language Models

Authors: Tianyi Zhao, Yinhan He, Wendy Zheng, Chen Chen

Organizations: University of Virginia

Abstract

Deploying Large Language Models (LLMs) in real-world dynamic environments raises the challenge of updating their pre-trained knowledge. While existing knowledge editing methods can reliably patch isolated facts, they frequently suffer from a reasoning gap, where the model recalls the edited fact but fails to utilize it in multi-step reasoning chains. Our analysis shows that multi-hop reasoning relies on sparse circuitry beyond direct factual recall, with functionally differentiated components contributing to intermediate processing and final-answer prediction. To bridge this gap, we introduce MCircKE (Mechanistic Circuit-based Knowledge Editing), a mechanistically grounded framework that enables a map-and-adapt editing procedure. MCircKE first identifies the causal circuits responsible for a specific reasoning task, capturing both the storage of the fact and the routing of its logical consequences. It then surgically update parameters exclusively within this mapped circuit. Extensive experiments on the MQuAKE-series benchmarks demonstrate the effectiveness of the proposed method for multi-hop reasoning in knowledge editing.

Figures & tables

Figure 1: Mechanistic analysis of multi-hop factual reasoning in GPT-2 Large. (a) Aggregate heatmap of attribution scores across 100 instances reveals distinct layer-wise computational regimes. (b) A sparse causal subgraph for a representative two-hop instance reveals a layer-localized routing pathway rather than a dense activation pattern. (c) Gray histogram shows the number of nodes active in the multi-hop circuit but inactive in the single-hop circuit; the red curve shows their aggregated causal importance. The clear mid-to-late-layer surge reveals additional important components that remain dormant during single-hop retrieval. (d) Causal ablation distinguishes routing/bridge heads from destination heads by their effects on the intermediate entity and final answer, respectively.

Figure 2: Layer distribution of attention-head roles. Points denote heads with non-negligible indirect effects; curves show within-role layer distributions and dashed lines their medians.

Figure 3: Example clean and corrupted inputs.

Model	Method	Overall M-Acc.	#hops			S-Acc.
Model	Method	Overall M-Acc.	2-hop	3-hop	4-hop	S-Acc.
GPT-2 Large	LoRA	24.0858	39.3886	13.4390	16.8724	48.947
	ROME	14.9939	17.2546	13.1162	14.4033	52.956
	MEMIT	7.7804	10.9053	5.7805	6.0357	35.009
	WISE	8.1138	11.6990	5.9272	5.9442	21.421
	AlphaEdit	8.5028	13.2275	5.6925	5.5327	44.093

Table 1: Performance comparison under different hop settings on GPT-2 Large and GPT-2 XL.

Method	M-hop (2-hop / 3-hop / 4-hop)	S-hop
LoRA	9.769 (12.61 / 6.72 / 10.11)	47.579
ROME	23.23 (32.89 / 17.55 / 17.06)	52.497
CaKE	56.318 ( 61.84 / 46.77 / 62.59 )	85.277
MCircKE	59.253 ( 64.61 / 49.41 / 66.26 )	85.219

Table 2: Performance comparison under different hop settings on GPT-J.

$\left\|\Delta\right\|$ (%)	MMLU	CSQA
LoRA	0.34	1.24
ROME	0.06	0.41
AlphaEdit	0.21	0.25
CaKE	0.09	0.49
MCircKE	0.10	0.24

Table 3: Locality performance.

Figure 4: Effect of the number of edges retained in the discovered circuit graphs.

Method	Single-hop	Multi-hop
Full Graph	61.40	39.30
Only MLPs	58.85	26.00
Random Edges	26.35	15.13
MCircKE	73.17	67.73

Table 4: Ablation Results.

Figure 5: Representation-level effects of editing. (a) Editing induces progressively larger hidden-state shifts across depth. (b) First-hop edits substantially elevate the counterfactual bridge entity in logit-lens rank.

Figure 6: Cross-instance circuit consistency.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	M-Acc	S-Acc
LoRA	31.23	35.87
ROME	39.03	85.77
CaKE	35.40	84.53
MCircKE	46.20	87.93

Appendix

Table 5: Results on Qwen2.5-7b / MQuAKE-CF-3K-v2.

Method	M-Acc	S-Acc
LoRA	17.47	20.25
ROME	16.10	60.80
CaKE	47.73	74.90
MCircKE	55.50	78.80

Appendix

Table 6: Results on GPT-2 XL / MQuAKE-CF-3K.

	MQuAKE-CF-3K	MQuAKE-T
LoRA	10.39	19.39
ROME	17.53	5.19
CaKE	34.97	40.29
MCircKE	47.02	98.94

Appendix

Table 7: Results on GPT-2 Large (M-Acc).

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 84 papers

Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping
Mar 6, 2026Zhenyu Lei, Qiong Wu, Jianxiong Dong +4Language Model Editing LLM Reasoning
University of Virginia · AT&T Chief Data Office · Florida State University
Learning to Edit Knowledge via Instruction-based Chain-of-Thought Prompting
Apr 7, 2026Jinhu Fu, Yan Bai, Longzhu He +4Knowledge Editing Language Model Editing
Beijing University of Posts and Telecommunications · Peking University · Huawei Technologies Ltd. +1
CRANE: Knowledge Editing for Reasoning MLLMs
Jun 8, 2026Han Huang, Hao Wang, Mengqi Zhang +3Knowledge Editing Knowledge Conflicts in Language Models
University of Chinese Academy of Sciences, Beijing 101408, China, and also with the New Laboratory of Pattern Recognition (NLPR), CASIA, Beijing 100045, China · Harbin Institute of Technology, Weihai 264209, China · School of Computer Science and Technology, Shandong University, Qingdao 266237, China

Mar 6, 2026J/K move · Enter open · S save

Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping

Zhenyu Lei, Qiong Wu, Jianxiong Dong, Yinhan He+3

University of Virginia · AT&T Chief Data Office · Florida State University

Language Model Editing LLM Reasoning Circuits in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) often exhibit flawed reasoning ability that undermines reliability. Existing approaches to improving reasoning typically treat it as a general and monolithic skill, applying broad training which is inefficient and unable to target specific reasoning errors. We introduce Reasoning Editing, a paradigm for selectively modifying specific reasoning patterns in LLMs while preserving other reasoning pathways. This task presents a fundamental trade-off between Generality, the ability of an edit to generalize across different tasks sharing the same reasoning pattern, and Locality, the ability to preserve other reasoning capabilities. Through systematic investigation, we uncover the Circuit-Interference Law: Edit interference between reasoning patterns is proportional to the overlap of their neural circuits. Guided by this principle, we propose REdit, the first framework to actively reshape neural circuits before editing, thereby modulating interference between reasoning patterns and mitigating the trade-off. REdit integrates three components: (i) Contrastive Circuit Reshaping, which directly addresses the generality-locality trade-off by disentangling overlapping circuits; (ii) Meta-Contrastive Learning, which extends transferability to novel reasoning patterns; and (iii) Dual-Level Protection, which preserves preexisting abilities by constraining reshaping update directions and regularizing task-level predictions. Extensive experiments with Qwen-2.5-3B on propositional logic reasoning tasks across three difficulty levels demonstrate that REdit consistently achieves superior generality and locality compared to baselines, with additional validation in mathematics showing broader potential. Our code is available at https://github.com/LzyFischer/REdit.

Abstract

Knowledge Editing

Explore similar work

Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping

Learning to Edit Knowledge via Instruction-based Chain-of-Thought Prompting

CRANE: Knowledge Editing for Reasoning MLLMs