Generalizable Model Editing Could Be More Than Direct Generalization
Authors: Xiaotian Ye, Chunyao Yang, Xiaohan Wang, Junkai Chen, Yongqi Duan, Mengqi Zhang, Shu Wu
Organizations: University of Cambridge · NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of the Chinese Academy of Sciences · HUST · BUPT · Shandong University
A central desideratum in model editing is generalization to complex tasks, which prior work formalizes under a direct generalization paradigm: treating and evaluating LLM behavior as an input-output mapping and expecting a factual update to alter outputs on reasoning tasks. Yet despite substantial empirical effort, the field consensus remains that such generalization is highly challenging and elusive. However, when we deploy the latest models in an in-the-wild chatbot-user setting, we surprisingly find that even classical editing methods exhibit remarkably fluent generalization behavior. This motivates us to revisit the obstacles to generalization and the limitations of the prevailing paradigm. We find that direct generalization is fundamentally constrained both theoretically and practically: theoretically, it suffers from an identifiability issue; and practically, existing approaches attempting to circumvent this are further hindered by the ambiguity of their constructed biases and by vulnerability to being overridden by other factors such as optimization dynamics. We instead identify compositional generalization, empowered by the increasingly strong reasoning and composition abilities of modern LLMs, as a theoretically grounded and empirically validated alternative. We validate this phenomenon across multiple models and editing methods, showing that such generalization behavior is widespread. Overall, we argue that generalizable model editing could be more than direct generalization, and call for a paradigm shift toward more generalizable and practically deployable knowledge update techniques.
Figures & tables
Figure 1
Figure 2: Edit success and overfit rate of unedited model and MEMIT variants on Llama.
Figure 3
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Total
Edit Requests
1489
Efficacy Prompts
1489
Paraphrase Prompts
2062
Subject Specificity Prompts
458
Relation Specificity Prompts
10310
Multihop Prompts
822
Appendix
Table 3: Composition statistics of EVOKE.
Task
Value
Edit Request
{Spike Hughes} originates from London → Philadelphia
Efficacy
{Spike Hughes} originates from (answer: Philadelphia)
Paraphrase
Spike Hughes is native to (answer: Philadelphia)
Subject Specificity
The profession of Spike Hughes is (answer: musician)
Relation Specificity
Max Mosley originates from (answer: London)
Multi-hop Reasoning
What famous food is associated with the city where Spike Hughes originates from? (answer: Cheesesteaks)
Appendix
Table 4: An Example of the EVOKE dataset
Figure 6: Generation examples of MEMIT-edited Qwen3.5 model
Figure 7: Post-edit multi-hop generations of Qwen3.5-9B after MEMIT. In each edit request the struck-out object is the original fact and the colored object is the new fact. Boldface in the response marks the new fact and the answer that completes the hop. The model answers the multi-hop question from the edited fact: the official languages of Switzerland, the places where Tamil is official, and the West Germanic family of German. All case studies presented are derived from actual model outputs with the random seed fixed to 42.
Dimension
LLM Judge Avg.
Human Rater Avg.
Pearson r
Pearson p -value
Spearman ρ
Spearman p -value
Factuality
3.21
3.17
0.884
3.4×10−34
0.855
1.2×10−29
Fluency
4.34
4.39
0.861
1.6×10−30
0.753
1.6×10−19
Appendix
Table 6: Agreement between LLM Judge and Human Rater scores on multi-hop tasks.
Knowledge editing enables rapid updates of specific factual knowledge in large language models (LLMs) without full retraining. However, more realistic scenarios call for a lifelong framework that handles continual updates rather than one-off modifications. In such settings, existing editing methods often overfit to target prompts, significantly degrading both the generalization of the edited knowledge and the model's general capabilities. To address this issue, we propose GLIME (Generalizable Lifelong Model Editing), which combines knowledge editing with preference optimization over generation behavior. GLIME further incorporates replay-based editing and a gradient constraint to preserve previously edited knowledge. Experimental results show that GLIME significantly improves knowledge generalization in lifelong editing settings while maintaining both editing performance and general capabilities.
Dahyun Jung, Suhyune Son, Heuiseok Lim
Department of Computer Science and Engineering, Korea University
Parameter-based knowledge editing updates the internal knowledge of large language models (LLMs) via localized weight modifications and has attracted significant attention. However, most existing methods overlook fundamental theoretical limitations and are rarely evaluated under realistic, practice-oriented settings. In this paper, we first present a theoretical analysis based on the dimensional Collapse Hypothesis, explaining how localized parameter edits can propagate along fragile directions in the representation space, inducing global interference and ultimately causing reasoning collapse. Building on this insight, we conduct a comprehensive empirical evaluation by systematically varying knowledge complexity, number of edits, evaluation dimensions, and baseline methods. Our results show that parameter-based editing methods consistently damage core LLM capabilities. In contrast, a simple retrieval-based baseline achieves consistently stronger performance than all parameter-editing methods across all evaluated conditions. These findings highlight that preserving the fundamental capabilities of LLMs after knowledge editing should be a central concern for future research.
Wanying Ren, Xin Song, Futing Wang +2
East China Normal University, Shanghai, China · Westlake University, Hangzhou, China · Nanyang Technological University, Singapore
Lifelong model editing (LME) aims to sequentially rectify outdated or inaccurate knowledge in deployed LLMs while minimizing side effects on unrelated inputs. However, existing approaches typically apply parameter perturbations to a static and dense set of LLM layers for all editing instances. This practice is counter-intuitive, as we hypothesize that different pieces of knowledge are stored in distinct layers of the model. Neglecting this layer-wise specificity can impede adaptability in integrating new knowledge and result in catastrophic forgetting for both general and previously edited knowledge. To address this, we propose HiEdit, a hierarchical reinforcement learning framework that adaptively identifies the most knowledge-relevant layers for each editing instance. By enabling dynamic, instance-aware layer selection and incorporating an intrinsic reward for sparsity, HiEdit achieves precise, localized updates. Experiments on various LLMs show that HiEdit boosts the performance of the competitive RLEdit by an average of 8.48% with perturbing only half of the layers per edit. Our code is available at: https://github.com/yangfanww/hiedit.
Yangfan Wang, Tianyang Sun, Chen Tang +3
Harbin Institute of Technology · AI Research Center, Midea Group (Shanghai) Co., Ltd. · State Key Laboratory of Smart Farm Technologies and Systems +1