KLOD: Locality-Preserving Knowledge Editing via Non-Target Distribution Preservation

Hojun Jeong; Gyunyeop Kim; Sangwoo Kang

cs.AIAug 28, 2026

KLOD: Locality-Preserving Knowledge Editing via Non-Target Distribution Preservation

Authors: Hojun Jeong, Gyunyeop Kim, Sangwoo Kang

Organizations: Department of Computing, Gachon University

Abstract

Fine-tuning-based knowledge editing is simple and architecture-agnostic, but standard cross-entropy increases the edited target probability without explicitly constraining changes in the non-target output distribution. In sequential editing, such unconstrained redistribution can accumulate as distributional drift and contribute to locality degradation. We propose KLOD, a bounded and distribution-preserving objective for fine-tuning-based knowledge editing that separates the intended target update from distributions that should remain stable. KLOD stops target amplification once a probability threshold is reached, while preserving the target-excluded non-target distribution at target positions and the full next-token distribution at prefix positions. Experiments on CounterFact and ZsRE with Llama3-8B-Instruct and Qwen2.5-7B-Instruct show that KLOD substantially mitigates locality degradation while maintaining high edit reliability. The target probability threshold further provides a controllable Generalization--Locality trade-off. Ablation, multi-seed, and distributional KL analyses support the interpretation that KLOD's locality gains are associated with preserving output distributions rather than simply weakening the edit. Code is available on GitHub https://github.com/Hostoday/KLOD .

Figures & tables

Model	Method	CounterFact					ZsRE
Model	Method	Rel.	Gen.	Loc.	Cap.	Score	Rel.	Gen.	Loc.	Cap.	Score
Llama3-8B-Instruct	pre-edit	0.87	1.18	100.00	62.11	41.04	26.00	25.26	100.00	62.11	53.34
	ROME	15.42	3.73	1.00	17.52	9.42	2.40	1.98	0.26	17.22	5.47
	AlphaEdit	80.33	59.40	15.80	25.46	45.25	83.41	74.78	36.10	35.28	57.39
	UltraEdit	35.85	18.80	6.38	60.80	30.46	63.45	56.23	35.21	60.63	53.88
	FT-L	1.37	0.40	0.00	15.07	4.21	6.96	6.58	1.03	17.88	8.11

Table 1: Main results on 3k sequential editing. Rel., Gen., Loc., and Cap. denote Reliability, Generalization, Locality, and Capability. Score is their arithmetic mean. Among editing methods, best and second-best results are shown in bold and underlined .

Method	Rewrite			Rephrase			Locality
Method	Prompt KL	Target KL	Non-target KL	Prompt KL	Target KL	Non-target KL	Prompt KL	Target KL
ROME	16.597	14.312	14.308	15.377	13.522	13.511	15.143	13.911
AlphaEdit	3.782	8.418	5.596	3.583	6.537	5.061	3.535	4.809
UltraEdit	3.470	4.962	4.650	1.180	3.578	3.459	3.218	4.521
LocFT-BF	2.697	15.936	8.967	2.296	10.118	8.124	2.481	10.135
OVERTONE	2.635	15.266	8.680	2.181	9.698	7.794	2.378	9.729

Table 2: Distributional KL divergence after 3k sequential editing on CounterFact with Llama3-8B-Instruct. Lower values indicate smaller drift from the pre-edit model.

Dataset	Method	Rel.	Gen.	Loc.
CounterFact	KLOD	$99.72\pm 0.02$	$46.76\pm 1.01$	$\mathbf{46.51\pm 1.98}$
	LocFT-BF	$99.74\pm 0.02$	$56.83\pm 0.95$	$1.89\pm 0.18$
	OVERTONE	$99.73\pm 0.07$	$57.56\pm 0.90$	$1.77\pm 0.05$
ZsRE	KLOD	$99.90\pm 0.02$	$87.39\pm 0.47$	$\mathbf{86.38\pm 0.33}$
	LocFT-BF	$99.81\pm 0.11$	$89.57\pm 0.32$	$51.15\pm 0.20$
	OVERTONE	$99.78\pm 0.16$	$88.75\pm 0.30$	$51.25\pm 0.25$

Table 3: Mean

\pm

standard deviation over three Breadth-First training seeds with Llama3-8B-Instruct. The sampled 3k edit set and all hyperparameters are fixed.

Figure 1: Generalization–Locality trade-off across target probability thresholds

\alpha

on CounterFact with Llama3-8B-Instruct. The

\alpha=1.0

endpoint is clipped to

1-10^{-6}

when computing the logit-odds margin.

Method	Rel.	Gen.	Loc.	Cap.
KLOD	99.70	47.37	44.75	60.77
w/o $L_{\mathrm{prefix}}$ (w/ $L_{\mathrm{nt}}$ )	99.60	52.05	41.30	60.70
w/o $L_{\mathrm{prefix}}$ (w/ $L_{\mathrm{full}}$ )	99.73	53.72	32.90	57.72
w/o $L_{\mathrm{nt}}$	99.70	52.70	9.15	60.78
w/o $L_{\mathrm{prefix}},L_{\mathrm{nt}}$	99.72	61.73	2.02	44.24
w/o All	99.77	55.83	1.87	41.92

Table 4: Ablation study on CounterFact with Llama3-8B-Instruct.

L_{\mathrm{full}}

denotes a full-vocabulary KL at target positions, including the target token, used in place of

L_{\mathrm{nt}}

. w/o All corresponds to LocFT-BF with standard cross-entropy.

Figure 2: Effect of pre-edit target prior on

L_{\mathrm{nt}}

. (a) Distribution of

p_{\theta_{0}}(a_{1}\mid x)

. (b) Mean paired Locality gain across target-prior quartiles. Error bars and Q1–Q4 intervals are normal-approximation 95% CIs (

n=750

per quartile).

Target objective	Rel.	Gen.	Loc.
Thresholded CE	99.80	46.20	37.50
Logit-odds hinge (KLOD)	99.70	47.37	44.75

Table 5: Matched-Generalization comparison between KLOD and sequence-level thresholded cross-entropy on CounterFact with Llama3-8B-Instruct.

Figure 3: Scaling behavior on ZsRE from 3k to 150k sequential edits. KLOD maintains Rel. and Gen. comparable to LocFT-BF while substantially improving Loc. across edit scales with Llama3-8B-Instruct.

Method	Llama3-8B		Qwen2.5-7B
Method	CounterFact	ZsRE	CounterFact	ZsRE
ROME	7.68	8.46	6.71	8.50
FT-L	1.60	1.70	0.34	1.00
LocFT-BF	0.16	0.23	0.17	0.30
KLOD	0.37	0.55	0.43	0.62

Table 6: Average editing time per edit in the 3k sequential editing setting. KLOD adds modest overhead over LocFT-BF while remaining under one second per edit.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$\alpha$	$\lambda_{\mathrm{prefix}}$	$\lambda_{\mathrm{nt}}$	Rel.	Gen.	Loc.
0.85	0.8	0.6	100.00	87.66	88.35
0.85	1.2	0.4	100.00	88.04	88.51
0.75	1.2	0.6	100.00	85.03	88.60
0.85	1.2	0.6	100.00	87.77	88.48
0.95	1.2	0.6	100.00	89.75	88.56
0.85	1.2	0.8	99.95	85.50	88.86

Appendix

Table 7: One-at-a-time sensitivity analysis on the held-out 500-example ZsRE set. The bold row denotes the shared configuration used in the main experiments.

Hyperparameter	Llama3-8B	Qwen2.5-7B
Editable layer	22	6
Editable module	MLP down-proj	MLP down-proj
$\alpha$	0.85	0.85
$\lambda_{\mathrm{nt}}$	0.6	0.6
$\lambda_{\mathrm{prefix}}$	1.2	1.2
Batch size	100	100

Appendix

Table 8: Implementation hyperparameters for KLOD.

Model	Dataset	Multi-token targets	Token Rel.	Strict Rel.
Llama3-8B	ZsRE	2,313/3,000 (77.1%)	99.85	99.61
Qwen2.5-7B	ZsRE	2,313/3,000 (77.1%)	99.90	99.57
Llama3-8B	CounterFact	40/3,000 (1.3%)	100.00	100.00
Qwen2.5-7B	CounterFact	40/3,000 (1.3%)	100.00	100.00

Appendix

Table 9: Reliability evaluated exclusively on multi-token target examples. Strict Rel. requires all target positions in an example to predict the gold token correctly.

Method	Edits	Rel.	Gen.	Loc.	Cap.	Score
FT-M	500	100.00	99.90	1.50	22.70	56.02
	1k	99.95	99.75	1.30	15.33	54.08
	3k	0.67	0.63	0.45	32.35	8.53
AlphaEdit	500	82.10	43.70	20.20	19.71	41.43
	1k	88.05	55.30	20.10	18.87	45.58
	3k	0.00	0.00	0.07	17.53	4.40

Appendix

Table 10: Cumulative edit-scale analysis on CounterFact with Qwen2.5-7B-Instruct.

Method	Rewrite			Rephrase			Locality
Method	Prompt KL	Target KL	Non-target KL	Prompt KL	Target KL	Non-target KL	Prompt KL	Target KL
ROME	8.019	7.917	7.440	8.651	8.071	7.521	8.095	6.430
AlphaEdit	12.919	12.981	12.949	12.669	12.157	12.130	13.293	12.962
UltraEdit	0.702	6.030	1.690	0.228	3.999	1.218	0.188	2.985
LocFT-BF	4.106	16.573	10.200	2.672	9.980	8.491	3.463	10.746
OVERTONE	3.959	16.391	10.259	2.482	9.646	8.381	3.396	10.863

Appendix

Table 11: Distributional KL divergence after 3k sequential editing on CounterFact with Qwen2.5-7B-Instruct. Lower values indicate smaller distributional drift from the pre-edit model. Best values are shown in bold . We use the same evaluation protocol as Table 2 .

Method	Component	Mean	Median	P90	P95	P99
KLOD	Prompt	0.099	0.063	0.226	0.306	0.513
KLOD	Non-target	0.364	0.168	0.808	1.342	3.119
LocFT-BF	Prompt	2.475	2.324	3.957	4.535	5.588
LocFT-BF	Non-target	8.567	8.369	12.671	14.234	16.718
OVERTONE	Prompt	2.379	2.225	3.776	4.312	5.366
OVERTONE	Non-target	8.257	8.015	12.350	13.930	16.298

Appendix

Table 12: Instance-level KL distributions after 3k sequential edits on CounterFact with Llama3-8B-Instruct. Prompt aggregates rewrite, rephrase, and locality contexts, whereas Non-target aggregates target-position KL from rewrite and rephrase contexts after target-token exclusion and renormalization. Lower values indicate less distributional drift.

Target objective	Preservation	Rel.	Gen.	Loc.	Cap.
CE	Full KL	99.77	54.22	22.55	51.47
CE	Non-target KL	99.77	54.42	26.77	54.11
Logit-odds hinge	Full KL	99.73	53.72	32.90	57.72
Logit-odds hinge	Non-target KL	99.60	52.05	41.30	60.70

Appendix

Table 13: Controlled no-prefix ablation on CounterFact with Llama3-8B-Instruct.

Method	Rel.	Gen.	Loc.	Cap.
KLOD	99.89	86.85	86.47	61.42
w/o $L_{\mathrm{prefix}}$	99.97	88.67	81.26	60.92
w/o $L_{\mathrm{nt}}$	99.57	87.99	82.09	61.68
w/o $L_{\mathrm{prefix}}$ , $L_{\mathrm{nt}}$	99.92	89.25	50.93	61.78
w/o All	99.93	89.21	51.24	62.10

Appendix

Table 14: Ablation study on ZsRE with Llama3-8B-Instruct. The row without both preservation terms uses only the bounded logit-odds objective. The w/o All variant corresponds to LocFT-BF with the standard cross-entropy objective.

CounterFact
Setting	Loc.	Damage	$F$	$O$
Bounded only	2.02	97.98	18.95	79.03
$+\,L_{\mathrm{nt}}$	41.30	58.70	13.57	45.13
Full KLOD	44.75	55.25	12.02	43.23
ZsRE
Bounded only	50.93	49.07	0.094	48.974

Appendix

Table 15: Post-hoc decomposition of locality prediction changes with Llama3-8B-Instruct.

F

denotes flips to edit-target tokens, and

O

denotes all other prediction changes. For ZsRE,

F

and

O

are reported to three decimal places because target flips are rare.

Method	Rel. EM	Gen. EM	Cap.
pre-edit	0.00	0.20	62.11
AlphaEdit	66.60	52.20	25.46
LocFT-BF	99.80	53.00	41.92
OVERTONE	99.60	56.40	42.51
KLOD ( $\alpha=0.85$ )	99.60	48.00	60.77
KLOD ( $\alpha=1.0$ )	99.60	61.60	57.65

Appendix

Table 16: Free-running generation results on 500 CounterFact cases after 3k edits with Llama3-8B-Instruct. Rel./Gen. EM denote normalized exact match on rewrite/rephrase prompts; bold marks the best edited-model result in each column.

Method	Rel.	Gen.	Loc.	Cap.
LocFT-BF	99.93	96.21	35.22	61.83
OVERTONE	99.90	95.98	34.98	61.62
KLOD	99.75	93.11	56.30	61.30

Appendix

Table 17: Results after 3k edits on WikiBigEdit with Llama3-8B-Instruct.

$\lambda_{\mathrm{nt}}$	Rel.	Gen.	Loc.	Cap.
0.2	99.70	48.23	35.33	60.98
0.4	99.63	47.43	40.37	60.78
0.6	99.70	47.37	44.75	60.77
0.8	99.63	45.07	47.42	61.07

Appendix

Table 18: Sweep over

\lambda_{\mathrm{nt}}

with

\lambda_{\mathrm{prefix}}

fixed at

1.2

on CounterFact with Llama3-8B-Instruct.

$\lambda_{\mathrm{prefix}}$	Rel.	Gen.	Loc.	Cap.
0.5	99.77	48.37	44.25	60.88
0.9	99.77	47.27	44.75	60.85
1.2	99.70	47.37	44.75	60.77
1.5	99.73	46.20	44.97	60.68

Appendix

Table 19: Sweep over

\lambda_{\mathrm{prefix}}

at the primary operating point, with

\lambda_{\mathrm{nt}}=0.6

and

\alpha=0.85

, on CounterFact with Llama3-8B-Instruct.

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 84 papers

ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing
Sep 24, 2026Zeyan Li, Hu Xu, Jianfeng XuFactual Knowledge in Language Models Language Model Editing
Shanghai Jiao Tong University
LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing
Jun 18, 2026Masih Eskandar, Miquel Sirera Perelló, Stratis Ioannidis +1Knowledge Editing Continual Learning for LLMs
Department of Electrical and Computer Engineering Northeastern University Boston, MA, 02115
Can Fine-Tuning Erase Your Edits? On the Fragile Coexistence of Knowledge Editing and Adaptation
Nov 8, 2025Yinjie Cheng, Paul Youssef, Christin Seifert +2Knowledge Editing LLM Fine-Tuning
School of Computer Science University of Sheffield Sheffield, UK · Mathematics and Computer Science Marburg University Marburg, Germany

Sep 24, 2026J/K move · Enter open · S save

ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing

Zeyan Li, Hu Xu, Jianfeng Xu

Shanghai Jiao Tong University

Factual Knowledge in Language Models Language Model Editing Concept Localization in Language Models

Open paper AskPDF arXiv

Knowledge editing changes what a model knows by modifying parameters so that a requested fact updates while unrelated behavior is preserved. This is usually treated as a write problem, but editing also involves an address problem: deciding which hidden states should receive the new residual. An update that activates too narrowly memorizes one prompt, while one that activates too broadly disrupts neighboring knowledge. Parametric editors encode this scope implicitly, whereas memory-based editors make the selection explicit but keep it outside the edited model. We propose ALOE (Addressed Low-rank Operator for Editing), which learns semantic addresses from paraphrases and hard same-subject negatives, aligns them with autoregressive hidden states through rollout refinement and gate calibration, and embeds the resulting gated low-rank operator within one MLP layer, so that the deployed model runs in a single forward pass with no external retriever or auxiliary router. Evaluated on CounterFact, ZSRE, and KnowEdit across three 7--8B model families, ALOE achieves efficacy between 0.955 and 0.999 and locality between 0.981 and 1.000; mechanistic analyses confirm that the learned geometry separates competing edits and that calibration suppresses out-of-scope activation. The remaining errors concentrate in paraphrase coverage and write fitting.

Abstract

Knowledge Editing

Explore similar work

ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing

LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing

Can Fine-Tuning Erase Your Edits? On the Fragile Coexistence of Knowledge Editing and Adaptation