Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances. Yet most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention. This assumption does not hold for rule-level knowledge, where a single rule must remain consistent across multiple interdependent forms. We investigate this problem through a mechanistic study of rule-level knowledge editing. To support this study, we extend the RuleEdit benchmark from 80 to 200 manually verified rules spanning mathematics and physics. Fine-grained causal tracing reveals a form-specific organization of rule knowledge in transformer layers: formulas and descriptions are concentrated in earlier layers, while instances are more associated with middle layers. These results suggest that rule knowledge is not uniformly localized, and therefore cannot be reliably edited by a single-layer or contiguous-block intervention. Based on this insight, we propose Distributed Multi-Layer Editing (DMLE), which applies a shared early-layer update to formulas and descriptions and a separate middle-layer update to instances. While remaining competitive on standard editing metrics, DMLE achieves substantially stronger rule-level editing performance. On average, it improves instance portability and rule understanding by 13.91 and 50.19 percentage points, respectively, over the strongest baseline across GPT-J-6B, Qwen2.5-7B, Qwen2-7B, and LLaMA-3-8B. The code is available at https://github.com/Pepper66/DMLE.
Figures & tables
Figure 1: Causal tracing heatmaps on GPT-J-6B. The heatmap shows the average indirect effect (AIE) across MLP layers and token positions for formula, description, and instance. 1st sub and Last sub denote the first and last subject tokens, and Last the final prompt token. Formula and description peak in earlier layers, whereas instance peaks in middle layers.
Figure 2: Causal tracing heatmaps on Qwen2-7B, Qwen2.5-7B, and LLaMA-3-8B. Across models, formula and description peak in earlier layers, whereas instance peaks in middle layers, supporting a form-specific layer-wise organization of rule knowledge.
Figure 3: Overview of DMLE. Given a rule with three aligned forms, formula and description share an update ΔWfd , while instance uses a separate update ΔWi . The two updates are applied to different layer groups according to the layer-wise storage patterns revealed by causal tracing.
Base Model
Method
Knowledge Editing
Rule-Level Editing
Rel. ↑
Gen. ↑
Loc. ↑
IP ↑
RU ↑
GPT-J-6B
ROME
80.17
44.33
17.25
20.50
9.13
MEMIT
88.88
41.83
44.75
23.08
11.25
GRACE
94.50
1.17
-
4.33
2.00
PROMPT
67.00
51.16
-
30.92
18.88
DMLE
95.00
58.17
46.79
59.19
84.92
Table 1: Main results on RuleEdit-200. Rel., Gen., and Loc. are standard knowledge editing metrics, while IP and RU are rule-level metrics that directly evaluate coherent rule editing. All values are reported in percentage (%). Best results are shown in bold, and second-best results are underlined. Locality is omitted for non-parametric methods .
Method
Qwen2-7B
LLaMA-3-8B
Rel. ↑
Gen. ↑
IP ↑
RU ↑
Rel. ↑
Gen. ↑
IP ↑
RU ↑
FLSU
97.17
60.33
50.58
82.50
52.67
38.67
35.50
42.25
FLJU
95.83
60.17
51.42
84.25
51.00
38.67
37.25
43.00
SFSU
96.33
61.83
63.92
84.88
52.17
40.33
40.42
44.25
DMLE
98.17
63.33
64.58
87.63
54.16
40.67
41.21
45.88
Table 2: Ablation on update organization and layer application. Results are reported on Qwen2-7B and LLaMA-3-8B. Best results are shown in bold.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
SECTION
TYPE
CONTENT
Metadata
Subject
geometric mean
Metadata
Rule
calculation of geometric mean
Formula
Prompt
The geometric mean of two numbers a and b is calculated by the formula:
Formula
Target_new
G=a+b .
Formula
Rephrase
The formula for the geometric mean of two values a and b is
Description
Prompt
The geometric mean of two numbers can be described as
Appendix
Table 3: Example of one structured editing case in RuleEdit-200
Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLMs is computationally expensive. Therefore, knowledge editing techniques are crucial for maintaining current information and correcting erroneous assertions within pre-trained models. Current benchmarks for knowledge editing primarily focus on recalling edited facts, often neglecting their logical consequences. To address this limitation, we introduce a new benchmark designed to evaluate how knowledge editing methods handle the logical consequences of a single fact edit. Our benchmark extracts relevant logical rules from a knowledge graph for a given edit. Then, it generates multi-hop questions based on these rules to assess the impact on logical consequences. Our findings indicate that while existing knowledge editing approaches can accurately insert direct assertions into LLMs, they frequently fail to inject entailed knowledge. Specifically, experiments with popular methods like ROME and FT reveal a substantial performance gap, up to 24%, between evaluations on directly edited knowledge and on entailed knowledge. This highlights the critical need for semantics-aware evaluation frameworks in knowledge editing.
Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera +1
Data Science Group, Heinz Nixdorf Institute, Paderborn University
Large language models (LLMs) are increasingly used as knowledge bases, but keeping them up to date requires targeted knowledge editing (KE). However, it remains unclear how edits are implemented inside the model once applied. In this work, we take a mechanistic view of KE using neuron-level knowledge attribution (NLKA). Unlike prior work that focuses on pre-edit causal tracing and localization, we use post-edit attribution -- contrasting successful and failed edits -- to isolate the computations that shift when an edit succeeds. Across representative KE methods, we find a consistent pattern: mid-to-late attention predominantly promotes the new target, while attention and FFN modules cooperate to suppress the original fact. Motivated by these findings, we propose MEGA, a MEchanism-Guided Activation steering method that performs attention-residual interventions in attribution-aligned regions without modifying model weights. On CounterFact and Popular, MEGA achieves strong editing performance across KE metrics on GPT2-XL and LLaMA2-7B. Overall, our results elevate post-edit attribution from analysis to engineering signal: by pinpointing where and how edits take hold, it powers MEGA to deliver reliable, architecture-agnostic knowledge edits.
Yuan Cao, Mingyang Wang, Hinrich Schütze
Technical University of Munich · LMU Munich · Munich Center for Machine Learning (MCML)
Lifelong model editing (LME) aims to sequentially rectify outdated or inaccurate knowledge in deployed LLMs while minimizing side effects on unrelated inputs. However, existing approaches typically apply parameter perturbations to a static and dense set of LLM layers for all editing instances. This practice is counter-intuitive, as we hypothesize that different pieces of knowledge are stored in distinct layers of the model. Neglecting this layer-wise specificity can impede adaptability in integrating new knowledge and result in catastrophic forgetting for both general and previously edited knowledge. To address this, we propose HiEdit, a hierarchical reinforcement learning framework that adaptively identifies the most knowledge-relevant layers for each editing instance. By enabling dynamic, instance-aware layer selection and incorporating an intrinsic reward for sparsity, HiEdit achieves precise, localized updates. Experiments on various LLMs show that HiEdit boosts the performance of the competitive RLEdit by an average of 8.48% with perturbing only half of the layers per edit. Our code is available at: https://github.com/yangfanww/hiedit.
Yangfan Wang, Tianyang Sun, Chen Tang +3
Harbin Institute of Technology · AI Research Center, Midea Group (Shanghai) Co., Ltd. · State Key Laboratory of Smart Farm Technologies and Systems +1