Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models

Yating Wang; Wenting Zhao; Yaqi Zhao; Yongshun Gong; Yilong Yin; Haoliang Sun

cs.CLApr 9, 2026

Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models

Authors: Yating Wang, Wenting Zhao, Yaqi Zhao, Yongshun Gong, Yilong Yin, Haoliang Sun

Organizations: School of Software, Shandong University, Jinan, China · Salesforce AI Research

Abstract

Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances. Yet most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention. This assumption does not hold for rule-level knowledge, where a single rule must remain consistent across multiple interdependent forms. We investigate this problem through a mechanistic study of rule-level knowledge editing. To support this study, we extend the RuleEdit benchmark from 80 to 200 manually verified rules spanning mathematics and physics. Fine-grained causal tracing reveals a form-specific organization of rule knowledge in transformer layers: formulas and descriptions are concentrated in earlier layers, while instances are more associated with middle layers. These results suggest that rule knowledge is not uniformly localized, and therefore cannot be reliably edited by a single-layer or contiguous-block intervention. Based on this insight, we propose Distributed Multi-Layer Editing (DMLE), which applies a shared early-layer update to formulas and descriptions and a separate middle-layer update to instances. While remaining competitive on standard editing metrics, DMLE achieves substantially stronger rule-level editing performance. On average, it improves instance portability and rule understanding by 13.91 and 50.19 percentage points, respectively, over the strongest baseline across GPT-J-6B, Qwen2.5-7B, Qwen2-7B, and LLaMA-3-8B. The code is available at https://github.com/Pepper66/DMLE.

Figures & tables

Figure 1: Causal tracing heatmaps on GPT-J-6B. The heatmap shows the average indirect effect (AIE) across MLP layers and token positions for formula, description, and instance. 1st sub and Last sub denote the first and last subject tokens, and Last the final prompt token. Formula and description peak in earlier layers, whereas instance peaks in middle layers.

Panel of Figure 2 — Figure 2: Causal tracing heatmaps on Qwen2-7B, Qwen2.5-7B, and LLaMA-3-8B. Across models, formula and description peak in earlier layers, whereas instance peaks in middle layers, supporting a form-specific layer-wise organization of rule knowledge.

Figure 3: Overview of DMLE. Given a rule with three aligned forms, formula and description share an update

\Delta W_{fd}

, while instance uses a separate update

\Delta W_{i}

. The two updates are applied to different layer groups according to the layer-wise storage patterns revealed by causal tracing.

Base Model	Method	Knowledge Editing			Rule-Level Editing
		Rel. $\uparrow$	Gen. $\uparrow$	Loc. $\uparrow$	IP $\uparrow$	RU $\uparrow$
GPT-J-6B	ROME	80.17	44.33	17.25	20.50	9.13
	MEMIT	88.88	41.83	44.75	23.08	11.25
	GRACE	94.50	1.17	-	4.33	2.00
	PROMPT	67.00	51.16	-	30.92	18.88
	DMLE	95.00	58.17	46.79	59.19	84.92

Table 1: Main results on RuleEdit-200. Rel., Gen., and Loc. are standard knowledge editing metrics, while IP and RU are rule-level metrics that directly evaluate coherent rule editing. All values are reported in percentage (%). Best results are shown in bold, and second-best results are underlined. Locality is omitted for non-parametric methods .

Method	Qwen2-7B				LLaMA-3-8B
Method	Rel. $\uparrow$	Gen. $\uparrow$	IP $\uparrow$	RU $\uparrow$	Rel. $\uparrow$	Gen. $\uparrow$	IP $\uparrow$	RU $\uparrow$
FLSU	97.17	60.33	50.58	82.50	52.67	38.67	35.50	42.25
FLJU	95.83	60.17	51.42	84.25	51.00	38.67	37.25	43.00
SFSU	96.33	61.83	63.92	84.88	52.17	40.33	40.42	44.25
DMLE	98.17	63.33	64.58	87.63	54.16	40.67	41.21	45.88

Table 2: Ablation on update organization and layer application. Results are reported on Qwen2-7B and LLaMA-3-8B. Best results are shown in bold.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

SECTION	TYPE	CONTENT
Metadata	Subject	geometric mean
Metadata	Rule	calculation of geometric mean
Formula	Prompt	The geometric mean of two numbers $a$ and $b$ is calculated by the formula:
Formula	Target_new	$G=\sqrt{a+b}$ .
Formula	Rephrase	The formula for the geometric mean of two values $a$ and $b$ is
Description	Prompt	The geometric mean of two numbers can be described as

Appendix

Table 3: Example of one structured editing case in RuleEdit-200

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 84 papers

Benchmarking Knowledge Editing using Logical Rules
Jun 9, 2026Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera +1Knowledge Editing LLM Evaluation
Data Science Group, Heinz Nixdorf Institute, Paderborn University
The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing
Mar 21, 2026Yuan Cao, Mingyang Wang, Hinrich SchützeLanguage Model Editing Activation Steering
Technical University of Munich · LMU Munich · Munich Center for Machine Learning (MCML)
HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning
Apr 13, 2026Yangfan Wang, Tianyang Sun, Chen Tang +3Model Editing Language Model Editing
Harbin Institute of Technology · AI Research Center, Midea Group (Shanghai) Co., Ltd. · State Key Laboratory of Smart Farm Technologies and Systems +1

Jun 9, 2026J/K move · Enter open · S save

Benchmarking Knowledge Editing using Logical Rules

Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo

Data Science Group, Heinz Nixdorf Institute, Paderborn University

Knowledge Editing LLM Evaluation LLM Reasoning

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLMs is computationally expensive. Therefore, knowledge editing techniques are crucial for maintaining current information and correcting erroneous assertions within pre-trained models. Current benchmarks for knowledge editing primarily focus on recalling edited facts, often neglecting their logical consequences. To address this limitation, we introduce a new benchmark designed to evaluate how knowledge editing methods handle the logical consequences of a single fact edit. Our benchmark extracts relevant logical rules from a knowledge graph for a given edit. Then, it generates multi-hop questions based on these rules to assess the impact on logical consequences. Our findings indicate that while existing knowledge editing approaches can accurately insert direct assertions into LLMs, they frequently fail to inject entailed knowledge. Specifically, experiments with popular methods like ROME and FT reveal a substantial performance gap, up to 24%, between evaluations on directly edited knowledge and on entailed knowledge. This highlights the critical need for semantics-aware evaluation frameworks in knowledge editing.

Abstract

Knowledge Editing

Explore similar work

Benchmarking Knowledge Editing using Logical Rules

The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing

HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning