cs.CLNov 8, 2026

Capacity Allocation at the Source: Sparse Target Optimization for LLM Knowledge Editing

Authors: Dongyao Chen, Jiayun Lei, Zhiyuan Ji, Zhiying Deng, Wei Liu, Xiaobo Sun

Organizations: Renmin University of China · Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · School of Computing, National University of Singapore · Department of Human Genetics, Emory University School of Medicine

Abstract

Knowledge editing has emerged as a promising approach for efficiently updating embedded knowledge in large language models (LLMs). It first computes an ideal target hidden state that steers the LLM toward the new output, and then edits the model parameters so that the original hidden state is mapped to this target. While most existing methods focus on the second stage of parameter editing in preserving other knowledge, we find that the upstream construction of the target hidden state also plays an important role in allocating the model's limited capacity space. We first establish a theoretical connection between interference in the parameter space and interference between target hidden states, showing that the two are positively correlated. Building on this insight, we propose a simple yet effective ℓ1\ell_1 regularization method that prunes away unnecessary ``tentacles'' of each knowledge vector, retaining only the essential components. With fewer extraneous tentacles, each edit affects less other knowledge, enabling better coexistence. Extensive experiments across multiple LLMs and benchmark datasets verify the effectiveness of our method. Code: https://github.com/wintersdragon-c/STO.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

    Jun 21, 2026Wenhao Yu, Zhicong Lu, Bo Lv +4Knowledge EditingModel Editing

  2. Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence

    May 30, 2026Wanying Ren, Xin Song, Futing Wang +2Model EditingLanguage Model Editing

  3. FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models

    Oct 19, 2026Yizhou Yang, Jiahong Fu, Zhiwu Wang +6Sequential Knowledge EditingKnowledge Editing