Beyond the Covariance Trap: Unlocking Generalization in Same-Subject Knowledge Editing for Large Language Models

Xiyu Liu; Qingyi Si; Zhengxiao Liu; Chenxu Yang; Naibin Gu; Zheng Lin

cs.CLMar 16, 2026

Beyond the Covariance Trap: Unlocking Generalization in Same-Subject Knowledge Editing for Large Language Models

Authors: Xiyu Liu, Qingyi Si, Zhengxiao Liu, Chenxu Yang, Naibin Gu, Zheng Lin

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · JD.com, Beijing, China

Abstract

While locate-then-edit knowledge editing efficiently updates knowledge encoded within Large Language Models (LLMs), a critical generalization failure mode emerges in the practical same-subject knowledge editing scenario: models fail to recall the updated knowledge when following user instructions, despite successfully recalling it in the original edited form. This paper identifies the geometric root of this generalization collapse as a fundamental conflict where the inner activation drifts induced by prompt variations exceed the model's geometric tolerance for generalization after editing. We attribute this instability to a dual pathology: (1) The joint optimization with orthogonal gradients collapses solutions into sharp minima with narrow stability, and (2) the standard covariance constraint paradoxically acts as a Covariance Trap that amplifies input perturbations. To resolve this, we introduce RoSE (Robust Same-subject Editing), which employs Isotropic Geometric Alignment to minimize representational deviation and Hierarchical Knowledge Integration to smooth the optimization landscape. Extensive experiments demonstrate that RoSE significantly improves instruction-following capabilities, laying the foundation for robust interactive parametric memory of LLM agents.

Figures & tables

Figure 1: Our work reveals that the current same-subject knowledge editing method MEMIT-Merge fails to generalize to instructed queries because Activation Deviation exceeds the edited model’s Tolerance Radius (

D>R

) in the activation space. We unlock robustness by reshaping the geometry towards the safe condition

D\leq R

Panel of Figure 2 — Figure 2: Geometric Pathology: $D>R$ . (a) Orthogonal gradients in joint same-subject editing cause tolerance radius $R$ to collapse. (b) The covariance matrix $C$ serves as an amplification trap, leading to a deviation $D$ of approximately 26.1 beyond $R$ . The area below the green line (average $R$ ) is the safe zone. Replacing $C$ with identity matrix can suppress $D$ to around $17.4$ .

Figure 3: Distribution of gradient conflict scores for pairs of edits concerning the same subject but different relations. The conflict scores of near 1 demonstrate that their update gradients are near-orthogonal.

Figure 4: Average pairwise cosine similarity of $k$ . The high similarity in the diagonal blocks (S1 vs. S1 Q-Form, S1 vs. S1 Instruction-Form) shows that subject representation is stable across prompt formats. The near-zero similarity in the off-diagonal blocks (S1 vs. S2) reveals that keys of distinct subjects are orthogonal.

Figure 5: The RoSE framework. We expand the tolerance radius $R$ via Hierarchical Knowledge Integration (HKI) and shrink the activation deviation $D$ via Isotropic Geometric Alignment (IGA) towards the ideal condition $D\leq R$ .

Model	Method	Efficacy $\uparrow$	Paraphrase $\uparrow$	Locality $\uparrow$	Overall $\uparrow$
Qwen2.5-7B-Instruct	Vanilla	26.0 ( $\pm$ 0.0)	38.7 ( $\pm$ 0.0)	64.3 ( $\pm$ 0.0)	43.0 ( $\pm$ 0.0)
	MEMIT	50.5 ( $\pm$ 0.3)	44.7 ( $\pm$ 0.1)	63.6 ( $\pm$ 0.1)	51.8 ( $\pm$ 0.1)
	AlphaEdit	63.8 ( $\pm$ 0.1)	49.6 ( $\pm$ 0.3)	63.5 ( $\pm$ 0.1)	58.5 ( $\pm$ 0.8)
	MEMIT-Merge	75.5 ( $\pm$ 0.5)	52.8 ( $\pm$ 0.7)	63.8 ( $\pm$ 0.1)	64.0 ( $\pm$ 0.4)
	RoSE (Ours)	86.7 ( $\pm$ 0.2)	63.8 ( $\pm$ 0.3)	63.3 ( $\pm$ 0.0)	72.2 ( $\pm$ 0.2)
Llama3.1-8B-Instruct	Vanilla	22.4 ( $\pm$ 0.0)	41.2 ( $\pm$ 0.0)	65.3 ( $\pm$ 0.0)	43.0 ( $\pm$ 0.0)

Table 1: Experimental instruction-following QA results on S2RKE across two models. Our method ( RoSE ) achieves the optimal performance across most key metrics. Standard deviations are shown in parentheses.

Model	Method	Single Hop $\uparrow$	Multi Hop $\uparrow$	Temporal $\uparrow$	Overall $\uparrow$
Qwen2.5-7B-Instruct	Vanilla	12.9 ( $\pm$ 0.1)	13.7 ( $\pm$ 0.2)	5.1 ( $\pm$ 0.5)	11.3 ( $\pm$ 0.2)
	MEMIT	15.1 ( $\pm$ 0.6)	15.7 ( $\pm$ 1.4)	8.7 ( $\pm$ 1.2)	13.8 ( $\pm$ 0.5)
	AlphaEdit	7.5 ( $\pm$ 1.4)	12.9( $\pm$ 2.8)	8.5 ( $\pm$ 2.1)	8.8( $\pm$ 1.8)
	MEMIT-Merge	17.2 ( $\pm$ 0.4)	17.3 ( $\pm$ 1.5)	11.2 ( $\pm$ 0.8)	15.9 ( $\pm$ 0.7)
	RoSE (Ours)	22.3 ( $\pm$ 0.3)	23.5 ( $\pm$ 1.6)	19.1 ( $\pm$ 0.9)	21.9 ( $\pm$ 0.3)
Llama3.1-8B-Instruct	Vanilla	11.0( $\pm$ 0.2)	12.8( $\pm$ 0.1)	1.6( $\pm$ 0.2)	9.4( $\pm$ 0.1)

Table 2: Experimental instruction-following QA performance on LoCoMo-Edit across two models. Our method ( RoSE ) achieves the best performance across all crucial metrics. Standard deviations are shown in parentheses.

Panel of Figure 6 — Figure 6: Ablation study on the core components of RoSE on S2RKE. Both components are shown to be crucial for achieving maximum robustness.

Panel of Figure 7 — Figure 7: Geometric validation of RoSE: IGA suppresses deviation (blue box), while HKI expands the tolerance basin (green zone).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Radius Collapse on Llama-3.1. The Tolerance Radius

R

shrinks significantly from "Isolated" editing to "Joint" same-subject editing, indicating the formation of Sharp Minima.

Figure 9: Gradient Orthogonality on Llama-3.1. Pairwise conflict scores between different relations (Facts 1-4) are consistently near 1 ( $\approx 0.90$ ), indicating orthogonal optimization directions.

Figure 10: The Covariance Trap on Llama-3.1: Using the Covariance matrix (C) results in higher activation deviation compared to the identity matrix, below the average Tolerance Radius

R

(green line).

Figure 11: Average key similarity on Llama-3.1. Subject keys are stable across diverse prompt formats (diagonal blocks $\approx 0.84-1.00$ ) but distinct across subjects (off-diagonal $\approx 0.11$ ), confirming that retrieval failure is not the primary cause.

Panel of Figure 12 — Figure 12: Activation Deviation $D$ and Tolerance Radius $R$ across mid-early layers for the baseline MEMIT-Merge and our RoSE.

Panel of Figure 13 — Figure 13: Layer sensitivity analysis for $k$ similarity.

Panel of Figure 14 — Figure 14: Average $k$ similarity on Qwen2.5-14B.

Panel of Figure 15 — Figure 15: Cases of $k$ similarity for semantically similar subjects on Qwen2.5-7B.

Boundary Case: High-Similarity Subjects
Edit Target:
Subject: iPhone 16
Fact: was released by $\rightarrow$ Elon Musk
Locality Check (Unintended Subject):
Query: Who released iPhone 11?
MEMIT-Merge: iPhone 11 was released by Elon Musk’s company, Tesla. (Fail)

Appendix

Table 3: Identical failure mode on similar subjects.

Figure 16: Performance on S2RKE with initial declarative prompts. RoSE enhances Paraphrase accuracy while preserving Locality, indicating better generalization capability.

Figure 17: Activation Deviation

D

of MEMIT (the covariance matrix), AlphaEdit (the null-space projection derived from the covariance matrix) and RoSE (the identity matrix) on Qwen2.5.

Panel of Figure 18 — Figure 18: Computational Overhead Comparison. RoSE achieves the lowest latency and memory consumption, validating that the removal of covariance operations (IGA) outweighs the cost of HKI.

Figure 19: Sensitivity analysis of HKI linguistic templates number

N

on Qwen2.5-7B-Instruct.

Model	Efficacy $\uparrow$	Paraphrase $\uparrow$	Locality $\uparrow$	Overall $\uparrow$
Vanilla	26.5 ( $\pm$ 0.0)	36.9 ( $\pm$ 0.0)	65.0 ( $\pm$ 0.0)	42.8 ( $\pm$ 0.0)
MEMIT	48.5 ( $\pm$ 0.3)	42.0 ( $\pm$ 0.2)	65.0 ( $\pm$ 0.0)	51.8 ( $\pm$ 0.1)
AlphaEdit	59.5 ( $\pm$ 0.4)	46.7 ( $\pm$ 0.2)	65.0 ( $\pm$ 0.1)	57.1 ( $\pm$ 0.2)
MEMIT-Merge	75.4 ( $\pm$ 0.2)	51.6 ( $\pm$ 0.1)	64.9 ( $\pm$ 0.1)	64.0 ( $\pm$ 0.1)
RoSE (Ours)	83.2 ( $\pm$ 0.3)	62.1 ( $\pm$ 0.1)	64.8 ( $\pm$ 0.0)	69.9 ( $\pm$ 0.0)

Appendix

Table 4: Experimental instruction-following QA results on S2RKE using the Qwen2.5-14B-Instruct model. RoSE successfully scales and maintains its optimal performance at a larger parameter size. Standard deviations are shown in parentheses.

Panel of Figure 20 — Figure 20: Instruction-following performance across sequences of 5, 10, 20, and 40 consecutive edits. The metrics evaluated are Efficacy, Paraphrase, Locality, and the Overall score. RoSE demonstrates superior stability and performance in short-to-medium length sequential editing scenarios.

Edit Request (Subject: Imo State)

Target Facts:

1. Imo State is located in

\rightarrow

Merthyr Tydfil County Borough

2. The capital of Imo State is

\rightarrow

Metz

3. The longitude of Imo State is roughly

\rightarrow

4. Imo State has a latitude of approximately

\rightarrow

Test Case 1: Instruction Following (Target Knowledge)

Appendix

Table 5: A case study on S2RKE. RoSE robustly recalls the edited fact (Merthyr Tydfil) under a complex instruction where MEMIT-Merge fails, while both methods preserve unrelated knowledge (Bavaria).

Edit Request (Subject: Caroline)

Conversation Context (Abridged):

[May 8] Caroline: "I went to a LGBTQ support group yesterday…"

[Jun 27] Caroline: "…gift from my grandma in my home country, Sweden."

[Jul 6] Caroline: "…pursue a career path in counseling…"

Injected Facts:

1. Caroline went to the LGBTQ support group on

\rightarrow

7 May 2023 (Temporal)

Appendix

Table 6: A case study on LoCoMo-Edit. RoSE successfully consolidates disjointed conversational updates into retrievable parametric memory, whereas MEMIT-Merge fails to answer.

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 84 papers

FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models
Oct 19, 2026Yizhou Yang, Jiahong Fu, Zhiwu Wang +6Knowledge Editing Language Model Editing
Xi’an Jiaotong University · Zhejiang University · City University of Hong Kong +2
Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence
May 30, 2026Wanying Ren, Xin Song, Futing Wang +2Model Editing Language Model Editing
East China Normal University, Shanghai, China · Westlake University, Hangzhou, China · Nanyang Technological University, Singapore
Learning to Edit Knowledge via Instruction-based Chain-of-Thought Prompting
Apr 7, 2026Jinhu Fu, Yan Bai, Longzhu He +4Knowledge Editing Language Model Editing
Beijing University of Posts and Telecommunications · Peking University · Huawei Technologies Ltd. +1

Oct 19, 2026J/K move · Enter open · S save

FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models

Yizhou Yang, Jiahong Fu, Zhiwu Wang, Kailong Fan+5

Xi’an Jiaotong University · Zhejiang University · City University of Hong Kong+2

Knowledge Editing Language Model Editing Continual Learning for LLMs Fisher Information

Open paper AskPDF arXiv

Sequential knowledge editing in Large Language Models (LLMs) often leads to a degradation of general capabilities due to cumulative parameter interference. In this paper, we revisit prominent projection-based methods like AlphaEdit and reveal a surprising mismatch: their robustness is primarily driven by update regularization rather than the intended null-space projection. Based on this insight, we propose FisherEdit, a theoretically grounded method built on Fisher geometry, which replaces coarse activation-space projections with importance-aware regularization in parameter space. It penalizes changes to sensitive parameters more strongly while retaining flexibility for new facts, and admits an efficient row-wise closed-form solution. Experiments on Llama-3, Qwen2.5, and Olmo-3 across multiple benchmarks demonstrate that FisherEdit achieves state-of-the-art performance while better preserving model stability and downstream task accuracy.

Abstract

Knowledge Editing

Explore similar work

FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models

Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence

Learning to Edit Knowledge via Instruction-based Chain-of-Thought Prompting