FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing

Peng Wang; Biyu Zhou; Xuehai Tang; Jizhong Han; Songlin Hu

cs.CLApr 14, 2026

FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing

Authors: Peng Wang, Biyu Zhou, Xuehai Tang, Jizhong Han, Songlin Hu

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

Abstract

Unstructured model editing aims to update models with real-world text, yet existing methods often memorize text holistically without reliable fine-grained fact access. To address this, we propose FABLE, a hierarchical framework that decouples fine-grained fact injection from holistic text generation. FABLE follows a two-stage, fact-first strategy: discrete facts are anchored in shallow layers, followed by minimal updates to deeper layers to produce coherent text. This decoupling resolves the mismatch between holistic recall and fine-grained fact access, reflecting the unidirectional Transformer flow in which surface-form generation amplifies rather than corrects underlying fact representations. We also introduce UnFine, a diagnostic benchmark with fine-grained question-answer pairs and fact-level metrics for systematic evaluation. Experiments show that FABLE substantially improves fine-grained question answering while maintaining state-of-the-art holistic editing performance. Our code is publicly available at https://github.com/caskcsg/FABLE.

Figures & tables

Figure 1: Limitation of existing unstructured editing methods (holistic recall vs. fine-grained fact access)

Figure 2: FABLE decomposes the key generator in a Transformer-based LLM into a two-stage hierarchical process: (1) fine-grained fact anchoring, which first encodes and stabilizes discrete factual knowledge; and (2) holistic surface-form integration, which then organizes these facts into a coherent narrative. The architecture consists of a fine-grained key generator, a holistic key generator, and a value generator.

Model	Method	Holistic		Fine-grained
Model	Method	Bert-Score $\uparrow$	Rouge-L $\uparrow$	Bert-Score $\uparrow$	Rouge-L $\uparrow$	HR $\uparrow$	$C_{\text{LCS}}$ $\uparrow$
UnFine-UnKE
Llama3-8B	Pre-edited	71.83 $\pm$ 0.11	22.69 $\pm$ 0.10	22.26 $\pm$ 0.08	12.81 $\pm$ 0.09	5.00 $\pm$ 0.10	17.50 $\pm$ 0.12
	FT-L	33.29 $\pm$ 0.22	13.35 $\pm$ 0.12	26.10 $\pm$ 0.10	15.49 $\pm$ 0.11	7.60 $\pm$ 0.12	21.83 $\pm$ 0.14
	ROME	79.51 $\pm$ 0.13	40.00 $\pm$ 0.23	24.76 $\pm$ 0.09	17.79 $\pm$ 0.12	9.53 $\pm$ 0.14	24.70 $\pm$ 0.15
	MEMIT	81.78 $\pm$ 0.20	49.64 $\pm$ 0.29	25.60 $\pm$ 0.09	20.54 $\pm$ 0.13	12.51 $\pm$ 0.16	28.30 $\pm$ 0.17

Table 1: Unstructured model editing performance with different models, methods, and datasets. “Pre-edited” denotes the original model before editing; “–” indicates that the corresponding field is absent from the dataset. Best results are boldfaced ; second-best are underlined .

Figure 3: Performance comparison during optimization between FABEL (with Stage 1) and its ablated variant (without Stage 1) on the UnFine-UnKE. The horizontal axis shows the number of optimization steps, and the vertical axis shows the average output probability of the target unstructured text.

Figure 4: F1-scores of each task for unstructured editing methods on different models after editing on the UnFine-UnKE dataset

Ablation Dimensions	Configuration	Holistic		Fine-grained
Ablation Dimensions	Configuration	Bert-Score $\uparrow$	Rouge-L $\uparrow$	Bert-Score $\uparrow$	Rouge-L $\uparrow$	HR $\uparrow$	$C_{\text{LCS}}$ $\uparrow$
Base	L=4,5,6; N=5 $\times$ S	99.36 $\pm$ 0.04	97.78 $\pm$ 0.10	65.63 $\pm$ 0.16	53.79 $\pm$ 0.19	53.31 $\pm$ 0.23	61.78 $\pm$ 0.19
Ablation on Layers
Layers	L=3	99.51 $\pm$ 0.04	98.60 $\pm$ 0.09	63.18 $\pm$ 0.16	50.36 $\pm$ 0.19	48.33 $\pm$ 0.23	57.83 $\pm$ 0.19
	L=4	99.70 $\pm$ 0.02	98.79 $\pm$ 0.08	65.52 $\pm$ 0.15	52.93 $\pm$ 0.19	52.07 $\pm$ 0.23	61.09 $\pm$ 0.19
	L=5	99.68 $\pm$ 0.02	98.78 $\pm$ 0.07	64.93 $\pm$ 0.15	52.97 $\pm$ 0.19	51.88 $\pm$ 0.24	60.96 $\pm$ 0.19

Table 2: The ablation experiment results of Llama3-8B-Instruct on the UnFine-UnKE dataset, where "Layers" indicates the updated layers of

\mathcal{F}_{\text{fine}}

, "Number" denotes the quantity of fine-grained question-answer pairs injected in the first stage, "Method" refers to the ablation on the method itself, and "Augmentation" represents the ablation on data augmentation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt Template for Generating Fine-Grained QA Pairs

You are asked to generate some short question-answer pairs based on the specified <Text>. These question-answer pairs mainly ask questions about the knowledge entities in the <Text>, and the answers should be the knowledge entities being asked.

Rules:

- Diversify Perspectives: Formulate questions from multiple angles to minimize overlapping answers. Use diverse question words (What, When, Where, Why, How, etc.); avoid "Who" or "Whose". Avoid yes/no questions unless they are highly informative.

- Entity Inclusion: Ensure all entities mentioned in the answers are present in the <Text>.

- Length Constraint: Each answer must not exceed 15 tokens.

Appendix

Table 3: Prompt Template for Generating Fine-Grained QA Pairs

Prompt Template for Key Knowledge Phrase Extraction

You are an assistant designed to extract minimal factual answers from any given <Answer>.

Rules:

- Break down the <Answer> into the smallest possible factual units. Each unit should represent a standalone piece of information.

- Remove any stylistic flourishes, adjectives, adverbs, or unnecessary connectors (e.g., "over" in "over a decade" is retained only if it’s part of the factual unit, as shown in the example).

- All tokens of the extracted answer must strictly reside within the <Answer>.

Appendix

Table 4: Prompt Template for Key Knowledge Phrase Extraction

Prompt Template for Multi-Aspect Fine-Grained Knowledge Extraction

You are a question generation expert. Your task is to generate non-overlapping natural language questions based on the given sentence.

Rules:

- Each question must be answerable directly using only the information in the sentence. The answer to each question must appear in the sentence (verbatim or semantically contained)

- The question should be clear, grammatical, and relevant.

- Avoid yes/no questions unless they are highly informative.

Appendix

Table 5: Prompt Template for Multi-Aspect Fine-Grained Knowledge Extraction

Model	Method	UnFine-UnKE(/s)	UnFine-CF(/s)	UnFine-MQ(/s)
LLaMA3-8B	FT-L	3.76	3.44	3.55
	ROME	17.74	15.02	16.78
	MEMIT	54.50	46.43	39.28
	UnKE	33.79	29.84	30.54
	AnyEdit	20.99	15.55	15.94
	FABLE	356.71	340.18	329.63

Appendix

Table 6: Time cost of different methods across various models and datasets.

Figure 5: Performance comparison during optimization between FABEL (with Stage 1) and its ablated variant (without Stage 1) on the UnFine-UnKE. The horizontal axis shows the number of optimization steps, and the vertical axis shows the average output probability of the target unstructured text.

Figure 6: Performance comparison during optimization between FABEL (with Stage 1) and its ablated variant (without Stage 1) on the UnFine-UnKE. The horizontal axis shows the number of optimization steps, and the vertical axis shows the average output probability of the target unstructured text.

Figure 7: F1-scores of each task for unstructured editing methods on different models after editing on the UnFine-CF dataset

Figure 8: F1-scores of each task for unstructured editing methods on different models after editing on the UnFine-MQ dataset

Mehtod	Holistic		Fine-grained
Mehtod	Bert-Score $\uparrow$	Rouge-L $\uparrow$	Bert-Score $\uparrow$	Rouge-L $\uparrow$	HR $\uparrow$	$C_{\text{LCS}}$ $\uparrow$
FABLE (Non-conflicting)	99.36 $\pm$ 0.04	97.78 $\pm$ 0.10	65.63 $\pm$ 0.16	53.79 $\pm$ 0.19	53.31 $\pm$ 0.23	61.78 $\pm$ 0.19
FABLE (Conflicting)	98.91 $\pm$ 0.03	93.04 $\pm$ 0.15	60.33 $\pm$ 0.09	50.48 $\pm$ 0.11	49.62 $\pm$ 0.14	56.39 $\pm$ 0.08

Appendix

Table 7: The experiment results of Llama3-8B-Instruct on the UnFine-UnKE dataset and Conflicting Editing DataSet.

Mehtod	Holistic		Fine-grained
Mehtod	Bert-Score $\uparrow$	Rouge-L $\uparrow$	Bert-Score $\uparrow$	Rouge-L $\uparrow$	HR $\uparrow$	$C_{\text{LCS}}$ $\uparrow$
FABLE (Last Token)	99.36 $\pm$ 0.04	97.78 $\pm$ 0.10	65.63 $\pm$ 0.16	53.79 $\pm$ 0.19	53.31 $\pm$ 0.23	61.78 $\pm$ 0.19
FABLE (Last Subject Word Token)	98.49 $\pm$ 0.06	94.82 $\pm$ 0.16	58.44 $\pm$ 0.16	43.33 $\pm$ 0.20	42.43 $\pm$ 0.23	51.67 $\pm$ 0.20

Appendix

Table 8: The representation comparison results of Llama3-8B-Instruct on the UnFine-UnKE dataset.

Panel of Table 9 — Table 9: Model Editing Case Study on Llama3-8B-Instruct

Panel of Table 10 — Table 10: Model Editing Case Study on Qwen2.5-7B-Instruct

Model Editing

Model editing changes targeted facts or behaviors in trained models while aiming to preserve their other capabilities.

See all 64 papers

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective
Sep 27, 2026Rui Liu, Chenheng Zhang, Haoxuan Li +1Knowledge Editing Continual Learning for LLMs
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing
Oct 2, 2026Ding Wu, Ye Zhang, Haoyu Wang +1Knowledge Editing Factual Knowledge in Language Models
Georgia Institute of Technology · University of Pennsylvania · University at Albany +1
CoBe: A Benchmark For Conversational Counterfactual Text Editing
Oct 9, 2026Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar +6Causal Counterfactual Generation Counterfactual Reasoning in Language Models
Causal Artificial Intelligence Lab Columbia University

Sep 27, 2026J/K move · Enter open · S save

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

Rui Liu, Chenheng Zhang, Haoxuan Li, Zhouchen Lin

State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

Knowledge Editing Continual Learning for LLMs Language Model Editing

Open paper AskPDF arXiv

Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit

Abstract

Model Editing

Explore similar work

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing

CoBe: A Benchmark For Conversational Counterfactual Text Editing