MetaKE: Meta-Learning for Knowledge Editing Toward a Better Accuracy-Editability Trade-off

Shuxin Liu; Di Gao; Ou Wu

cs.CLMar 13, 2026

MetaKE: Meta-Learning for Knowledge Editing Toward a Better Accuracy-Editability Trade-off

Authors: Shuxin Liu, Di Gao, Ou Wu

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China

Abstract

Existing locate-then-edit Knowledge Editing (KE) methods typically decompose editing into two stages: upstream target representation optimization and downstream constrained parameter optimization. The optimization across the two stages is disconnected: upstream applies uniform regularization without observing downstream realization of the planned residual, hindering a refined accuracy-editability trade-off. Since this realization is request-specific and depends on downstream constraints, uniform regularization can over-shrink high-association requests, causing insufficient editing, while it can under-regularize low-association requests, producing over-large planned residuals that reduce downstream editability. To bridge this disconnect, we propose MetaKE (Meta-learning for Knowledge Editing), a new framework that unifies upstream and downstream stages into a bi-level optimization problem. The inner level optimizes parameter updates for the target representation, while the outer level optimizes representation using feedback from downstream constraints, achieving a better semantic accuracy-editability trade-off. To avoid costly multi-layer backpropagation, we introduce a Structural Gradient Proxy to approximate and propagate this feedback. Extensive experiments show that MetaKE outperforms strong baselines, offering a new perspective on KE.

Figures & tables

Figure 1: The architecture of the proposed method MetaKE.

Figure 2: Distribution of request-specific downstream association

\rho_{i}

over

500

sampled edit requests. Dashed lines mark medians.

Figure 3: Fidelity of the Structural Gradient Proxy. (a) Cosine similarity between

\boldsymbol{g}_{\mathrm{proxy}}

and

\boldsymbol{g}_{\mathrm{true}}

\boldsymbol{g}_{\mathrm{proxy}}

aligns with

\boldsymbol{g}_{\mathrm{true}}

96\%

N=100

edits, with

72\%

above

\cos>0.5

. (b) One-step meta-loss reduction. The proxy achieves

22.2

bps reduction, recovering

85\%

of the

\boldsymbol{g}_{\mathrm{true}}

descent effect.

Model	Method	CounterFact					ZsRE
Model	Method	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Flu. $\uparrow$	Consis. $\uparrow$	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$
LLaMA3 (8B)	MEND	$63.18_{\pm 0.34}$	$61.26_{\pm 0.32}$	$45.42_{\pm 0.41}$	$372.08_{\pm 0.75}$	$4.19_{\pm 0.06}$	$0.88_{\pm 0.05}$	$1.06_{\pm 0.05}$	$0.51_{\pm 0.03}$
	MALMEN	$38.82_{\pm 0.34}$	$49.01_{\pm 0.41}$	$56.64_{\pm 0.37}$	$524.48_{\pm 0.45}$	$19.83_{\pm 0.13}$	$39.65_{\pm 0.21}$	$40.56_{\pm 0.28}$	$30.47_{\pm 0.24}$
	ROME	${66.18}_{\pm 0.47}$	${63.47}_{\pm 0.41}$	${48.61}_{\pm 0.25}$	${458.75}_{\pm 0.23}$	${2.05}_{\pm 0.06}$	${1.28}_{\pm 0.04}$	${1.32}_{\pm 0.03}$	${0.64}_{\pm 0.03}$
	MEMIT	$65.52_{\pm 0.35}$	$63.94_{\pm 0.38}$	$51.26_{\pm 0.41}$	$452.55_{\pm 1.24}$	$5.94_{\pm 0.15}$	$36.15_{\pm 0.31}$	$31.42_{\pm 0.28}$	$18.57_{\pm 0.25}$
	PRUNE	$66.82_{\pm 0.42}$	$64.64_{\pm 0.37}$	$49.88_{\pm 0.25}$	$440.67_{\pm 1.12}$	$6.12_{\pm 0.09}$	$0.52_{\pm 0.05}$	$0.58_{\pm 0.07}$	$1.34_{\pm 0.04}$

Table 1: Comparison of MetaKE with editing methods on the knowledge editing task. The best results are highlighted in red , while the second-best results are underlined .

Figure 4: Empirical realization coefficient

\widehat{\beta}

across association

\rho

Figure 5: Efficacy of high- and low-

\rho

requests under different KL strengths.

Variant	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$
MetaKE	96.84	92.45	33.12
- w/o Bi-level Optimization	94.27	88.35	32.65
- w/o Structural Gate	95.88	90.75	32.14
- Stop-Grad Feedback	94.31	88.24	32.48

Table 2: Ablation study on MetaKE.

Figure 6: General-capability retention under long-horizon sequential editing on LLaMA3 (8B).

Figure 7: The distribution of hidden representations of pre-edited and post-edited LLMs.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Locality constraint	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Flu. $\uparrow$	Cons. $\uparrow$	Runtime/100 edits (s) $\downarrow$
No Outer Locality Loss	99.95	98.82	75.97	620.43	43.16	459.33
Subject-Essence	99.90	98.75	76.76	620.63	43.53	517.94
Neighborhood-1	99.75	98.55	76.87	620.56	43.63	550.83
Neighborhood-5	99.50	98.35	77.21	620.14	43.69	773.26

Appendix

Table 3: Locality-constraint ablation on GPT-J/CounterFact

Figure 8: Per-edit diagnostics of the local fidelity conditions of Theorem 2 on

N=100

uniformly sampled ZsRE edits using GPT2-XL, evaluated after

\boldsymbol{v}^{*}

converges. (a) Non-final-layer leakage ratio

\xi

. (b) Final-layer proxy gap

K\varepsilon_{\mathrm{AE}}

. (c) Structural non-degeneracy factor

\alpha

. (d) Full-gradient sufficient-condition ratio

c_{i}/\chi_{i}

, and values below 1 satisfy the full-gradient sufficient local descent-direction condition. Red lines mark medians, and dashed lines mark the threshold

1

where shown.

$H$	Median cosine $\uparrow$	Positive alignment (%) $\uparrow$	Cosine $>0.5$ (%) $\uparrow$	True-gradient reduction (bps) $\uparrow$	Proxy-gradient reduction (bps) $\uparrow$
100	0.664	96	70	25.61	21.73
1,000	0.648	95	70	25.74	21.89
3,000	0.653	96	71	26.20	22.01
5,000	0.635	94	68	25.55	21.42

Appendix

Table 4: Long-horizon proxy fidelity on GPT2-XL/ZsRE. Cosine is computed between the true and proxy hypergradients. The last two columns report one-step relative meta-loss reduction in basis points (bps).

$\omega$	AlphaEdit		MetaKE w/o Structural Gate		MetaKE
$\omega$	H	L	H	L	H	L
0.008	91.92	91.25	93.08	91.47	97.00	97.00
0.016	93.00	92.08	94.42	93.00	97.67	97.17
0.031	94.33	93.33	96.25	94.92	98.17	97.83
0.0625	93.75	94.00	96.00	96.08	98.67	98.17
0.125	93.67	94.42	95.50	96.25	98.42	98.50

Appendix

Table 5: Efficacy (%) in the refinement-matched KL sweep on GPT2-XL and ZsRE. H/L denote the high-/low-association groups selected using AlphaEdit’s

\rho_{i}

and held fixed across methods.

Hyperparameter	Value	GPT2-XL			GPT-J			LLaMA3
Hyperparameter	Value	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$
$T$	5	97.81	91.08	26.86	99.76	97.23	29.50	96.64	92.26	32.61
	10	97.98	91.19	27.31	99.75	97.49	29.61	96.81	92.37	32.89
	15	98.12	91.26	27.23	99.82	97.37	29.73	96.84	92.45	33.12
	20	97.81	91.04	27.25	99.62	97.28	29.38	96.64	92.22	32.00
$\eta$	$10^{-4}$	98.00	91.47	26.61	99.58	97.06	29.41	96.63	92.25	32.34

Appendix

Table 6: Sensitivity analysis of key MetaKE hyperparameters, and all runs use the same AlphaEdit-based editing operator.

Model	Method	CounterFact					ZsRE
		Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Flu. $\uparrow$	Consis. $\uparrow$	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$
Phi-1.5	AlphaEdit	$98.40_{\pm 0.23}$	$83.62_{\pm 0.19}$	$65.41_{\pm 0.20}$	$634.12_{\pm 0.55}$	$36.05_{\pm 0.15}$	$96.51_{\pm 0.39}$	$84.62_{\pm 0.28}$	$22.52_{\pm 0.16}$
	$\text{AlphaEdit}_{\text{BLUE}}$	$\underline{99.20_{\pm 0.15}}$	$\underline{93.22_{\pm 0.24}}$	$62.11_{\pm 0.43}$	$634.55_{\pm 0.51}$	${\color[rgb]{1,0,0}37.91_{\pm 0.22}}$	$96.28_{\pm 0.41}$	$85.65_{\pm 0.25}$	$22.11_{\pm 0.17}$
	SPHERE	$98.80_{\pm 0.15}$	$82.78_{\pm 0.27}$	$\underline{65.92_{\pm 0.31}}$	$\underline{635.18_{\pm 0.54}}$	$36.39_{\pm 0.12}$	$\underline{96.55_{\pm 0.35}}$	$\underline{85.97_{\pm 0.26}}$	$\underline{22.74_{\pm 0.15}}$
	MetaKE	${\color[rgb]{1,0,0}99.40_{\pm 0.21}}$	${\color[rgb]{1,0,0}94.31_{\pm 0.16}}$	${\color[rgb]{1,0,0}65.97_{\pm 0.24}}$	${\color[rgb]{1,0,0}635.78_{\pm 0.51}}$	$\underline{37.56_{\pm 0.22}}$	${\color[rgb]{1,0,0}96.87_{\pm 0.22}}$	${\color[rgb]{1,0,0}86.49_{\pm 0.32}}$	${\color[rgb]{1,0,0}22.84_{\pm 0.11}}$
Qwen-4B-Thinking	AlphaEdit	$78.90_{\pm 0.55}$	$69.15_{\pm 0.44}$	$50.48_{\pm 0.73}$	$513.75_{\pm 0.85}$	$8.84_{\pm 0.15}$	$\underline{94.42_{\pm 0.27}}$	$\underline{84.81_{\pm 0.28}}$	$\underline{34.55_{\pm 0.20}}$

Appendix

Table 7: Comparison of MetaKE with existing methods on Phi-1.5 and Qwen-4B-Thinking. The best results are highlighted in red , while the second-best results are underlined .

Figure 9: Performance Variation of Model Editing with Sequential Edits.

Method	CounterFact			ZsRE
Method	LLaMA3	GPT-J	GPT2-XL	LLaMA3	GPT-J	GPT2-XL
MEMIT	688.42s	299.15s	161.08s	364.85s	293.11s	164.05s
AlphaEdit	462.18s	362.80s	179.15s	325.41s	364.72s	185.20s
AlphaEdit ${}_{\text{BLUE}}$	908.35s	364.12s	187.33s	1199.15s	416.80s	187.95s
SPHERE	822.80s	452.80s	187.28s	336.77s	449.42s	189.23s
MetaKE	702.51s	518.80s	241.85s	540.18s	521.55s	250.02s

Appendix

Table 8: Runtime comparison of MetaKE with existing methods on GPT2-XL, GPT-J, and LLaMA3.

Method	CounterFact			ZsRE
	LLaMA3	GPT-J	GPT2-XL	LLaMA3	GPT-J	GPT2-XL
MEMIT	36.84	31.63	12.62	36.87	31.71	12.44
AlphaEdit	34.31	28.94	9.88	34.74	29.52	10.33
$\text{AlphaEdit}_{\text{BLUE}}$	35.40	29.42	9.94	35.23	29.92	10.54
SPHERE	40.94	35.15	13.23	33.79	35.72	13.51
MetaKE	37.97	35.09	11.33	36.28	34.83	11.33

Appendix

Table 9: Peak GPU memory usage (GB) for performing a batch of 100 edits.

Method	ZsRE			CounterFact
Method	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Flu. $\uparrow$	Cons. $\uparrow$
MEMIT	96.87	93.21	31.16	98.10	95.78	64.08	594.47	39.52
EMMET	96.78	92.87	30.70	97.90	95.62	63.93	593.61	39.50
AlphaEdit	99.63	95.73	28.11	99.65	96.40	76.23	618.32	42.39
MetaKE-MEMIT	99.46	96.77	32.22	99.45	97.68	74.33	620.16	41.81
MetaKE-EMMET	99.36	96.47	31.86	99.15	97.62	71.63	608.12	40.32

Appendix

Table 10: Comparison across downstream realization operators on GPT-J.

Method	LG $\uparrow$	CI $\uparrow$	CII $\uparrow$	SA $\uparrow$	PV $\uparrow$	RS $\uparrow$
AlphaEdit	0.0315	0.2563	0.0625	1.0000	0.6195	0.8830
MetaKE	0.0682	0.3247	0.1016	1.0000	0.6614	0.9058

Appendix

Table 11: Performance on the full RippleEdits-POPULAR split using LLaMA3-8B under independent single-edit evaluation. Higher is better for all metrics.

Method	ZsRE			CounterFact
Method	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Eff. $\uparrow$	Gen. $\uparrow$	Spe. $\uparrow$	Flu. $\uparrow$	Cons. $\uparrow$
AlphaEdit	84.16	77.83	24.91	89.50	83.28	65.47	548.36	23.68
MetaKE	89.08	85.37	26.32	93.75	92.65	67.02	564.91	27.08

Appendix

Table 12: LLaMA-2-13B editing results on ZsRE and CounterFact (2,000 edits, batch size 100, seed 0). Higher is better.

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 84 papers

Mar 21, 2026

The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing

Large language models (LLMs) are increasingly used as knowledge bases, but keeping them up to date requires targeted knowledge editing (KE). However, it remains unclear how edits are implemented inside the model once applied. In this work, we take a mechanistic view of KE using neuron-level knowledge attribution (NLKA). Unlike prior work that focuses on pre-edit causal tracing and localization, we use post-edit attribution -- contrasting successful and failed edits -- to isolate the computations that shift when an edit succeeds. Across representative KE methods, we find a consistent pattern: mid-to-late attention predominantly promotes the new target, while attention and FFN modules cooperate to suppress the original fact. Motivated by these findings, we propose MEGA, a MEchanism-Guided Activation steering method that performs attention-residual interventions in attribution-aligned regions without modifying model weights. On CounterFact and Popular, MEGA achieves strong editing performance across KE metrics on GPT2-XL and LLaMA2-7B. Overall, our results elevate post-edit attribution from analysis to engineering signal: by pinpointing where and how edits take hold, it powers MEGA to deliver reliable, architecture-agnostic knowledge edits.

Yuan Cao, Mingyang Wang, Hinrich Schütze

Technical University of Munich · LMU Munich · Munich Center for Machine Learning (MCML)

Language Model Editing Activation Steering Causal Interventions in Language Models Mechanistic Interpretability

Ask PDF

May 15, 2026

Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs increasingly adopt Mixture-of-Experts (MoE) architectures for their superior memory footprint and inference efficiency. This mismatch leaves a growing class of production models without principled editing tools. We propose a MEMIT-like framework for knowledge editing in MoE-based LLMs. Our method exploits the tensor structure of MoE layers to formulate the editing objective faithfully at the per expert level, and applies the Woodbury matrix identity to avoid materializing or inverting the full stacked matrix of expert weights. The resulting update reduces to inversions of fixed low-rank matrices and requires no additional backward passes. Empirically, our approach matches the editing quality of strong baselines on the main KE metrics while accelerating the editing procedure by up to 6x, owing to the batched MEMIT-style formulation and the low-dimensional inversions enabled by the Woodbury identity. These results show that closed-form, parameter-modifying KE can be extended efficiently beyond dense layers, opening a path toward scalable knowledge editing in modern sparse LLM architectures.

Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin +3

1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory · 3Innopolis University

Knowledge Editing Mixture-of-Experts Language Models Model Editing Language Model Editing

Ask PDF

Jun 21, 2026

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining. However, existing methods still suffer from performance degradation in batch knowledge editing. We identify that semantic representation entanglement, such as overlapping concepts and shared syntactic patterns, accumulates interference in the representation space and reduces editing precision. To bridge this gap, in this paper, we propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs by constructing a general semantic subspace and enforcing orthogonal constraints on edit vectors, effectively decoupling semantic entanglement. Furthermore, we introduce a gated non-linear representation head to enable adaptive learning of editing locations and precise control over knowledge injection. Extensive experiments show that ORE outperforms existing methods and achieves superior performance in cross-lingual knowledge editing scenarios. We release our code at https://github.com/YVVH/ORE.

Wenhao Yu, Zhicong Lu, Bo Lv +4

School of Computer Science and Technology, Tianjin University · Kexin Technology · University of Chinese Academy of Sciences +2

Knowledge Editing Model Editing Language Model Editing Representation Geometry in Language Models

Ask PDF

Abstract

Knowledge Editing

Explore similar work

The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing

Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs