Paper ID: 2410.09338 • Published Oct 12, 2024
Keys to Robust Edits: from Theoretical Insights to Practical Advances
Jianhao Yan, Futing Wang, Yun Luo, Yafu Li, Yue Zhang
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Large language models (LLMs) have revolutionized knowledge storage and
retrieval, but face challenges with conflicting and outdated information.
Knowledge editing techniques have been proposed to address these issues, yet
they struggle with robustness tests involving long contexts, paraphrased
subjects, and continuous edits. This work investigates the cause of these
failures in locate-and-edit methods, offering theoretical insights into their
key-value modeling and deriving mathematical bounds for robust and specific
edits, leading to a novel 'group discussion' conceptual model for
locate-and-edit methods. Empirical analysis reveals that keys used by current
methods fail to meet robustness and specificity requirements. To address this,
we propose a Robust Edit Pathway (REP) that disentangles editing keys from
LLMs' inner representations. Evaluations on LLaMA2-7B and Mistral-7B using the
CounterFact dataset show that REP significantly improves robustness across
various metrics, both in-domain and out-of-domain, with minimal trade-offs in
success rate and locality. Our findings advance the development of reliable and
flexible knowledge updating in LLMs.