Training a Knowledge Base: Supervised Structure Learning for Agent-Curated Document Stores

Yu Pan; Hongfeng Yu

cs.CLAug 22, 2026

Training a Knowledge Base: Supervised Structure Learning for Agent-Curated Document Stores

Authors: Yu Pan, Hongfeng Yu

Organizations: University of Nebraska–Lincoln

Abstract

Retrieval-augmented generation treats the document store as a frozen input, and the offline pipelines that do build structure over it build it unsupervised -- a whole corpus indexed at uniform effort, with no signal about which structure a question will need. We instead treat the knowledge base as a non-parametric model trained on (question, answer) pairs: a curator agent answers a supervised question against the current store, is shown the gold answer, then edits the store. The store carries forward, and we evaluate the curated store with a test set, on two contamination-free benchmarks: KBGym, a fictional-universe generator we release, and PhantomWiki. Generalization is probed with four question groups of decreasing overlap with the training set: the trained questions themselves, and unseen questions sharing both of their keys with training, one key, or neither. The curated store's advantage grows with overlap -- from parity where no key was shared, through +0.176 F1 where both keys were, to 25% fewer actions at +0.294 F1 on the trained questions, the one cell significant on both benchmarks -- while matching HippoRAG's gains with 1,913 links against its 196,112: per point of corpus covered, 1.5x the action saving and 2.1x the accuracy gain. Accuracy rises steadily with the share of the corpus the indexes cover, so training on more questions widens coverage, and with it the generalization.

Figures & tables

Panel of Figure 1 — Fig. 1: The train/test protocol. A curator consumes supervised QA pairs, answering each from the current store (gold revealed only after its single answer) and consolidating verified reasoning into structure. Evaluation freezes the store and examines an independent budgeted reader on held-out questions; the hypothesis (Eq. 1 ) is non-inferior accuracy at strictly lower per-question cost.

Action	Effect	R	C
search(q, page)	five most similar documents; one page per action	✓	✓
read(id)	the document and the full text of everything it links to, one level	✓	✓
answer(text)	submit and end the pass	✓
add(text)	new document; a near-duplicate is merged instead		✓
edit(id, text)	replace a document’s text		✓
delete(id)	remove a document		✓

TABLE I: The action set. R = available to the reader (exam and training phase 1); C = available to the curator (training phase 2). No action is available to both roles in a way that lets the reader modify the store.

	KBGym (main)	PhantomWiki (official)
documents	5,864 sentences	3,403 sentences from 405 articles
source	our generator	their generator
questions	330 (150/100/50/30)	330, same scheme
question author	us	PhantomWiki
supports	exact	unavailable
contamination	none	none

TABLE II: Benchmark arms. Both use the same splits scheme, reader, action set and budgets; the arms differ in who wrote the questions.

Fig. 2: Six frozen snapshots from epoch 1, read two ways. The reader, action set, budget and questions are identical at every point, so the only thing varying is the store. (a, b) Cost and accuracy per question group against training iteration: the two key-covered groups improve while the two uncovered ones do not — gains track key coverage, not question identity. (c, d) The same runs pooled, against the share of the corpus the store’s indexes reach — the quantity training actually buys. Cost falls and accuracy rises with coverage; the last point is the epoch-2 store. Coverage reaches

24.8\%

by the end of epoch 1; the second pass (Sec. E2) then converts that structure into most of the accuracy gain.

Store	coverage	actions / pt	F1 / pt
B1 flat	0%	—	—
B2 GraphRAG	100%	$-0.002$	$-0.0003$
B3 HippoRAG2	100%	$0.023$	$+0.0021$
Ours (trained)	27.6%	$\mathbf{0.035}$	$\mathbf{+0.0045}$

TABLE III: What a point of corpus coverage buys. Each arm’s gain over the untrained flat store, pooled over all 136 probe questions, divided by the share of the corpus its indexes reach. The offline arms index everything; ours indexes the part its training questions named.

$\rho$ : actions relative to B1
Store	trained on	both keys	one key	neither
B1 flat	1.000	1.000	1.000	1.000
B2 GraphRAG	0.918	1.154	0.972	1.047
Ours (trained)	0.747 ∗	0.879	0.977	1.016
F1
B1 flat	0.618	0.765	0.647	0.794

TABLE IV: The key-coverage gradient. Groups differ in how much of the question the training set touched, stratified so all four share one template mix; residual key-popularity differences remain, so compare within a column: each arm against B1 on the same questions.

\rho

= actions relative to the untrained flat store on the same questions, so lower is cheaper; ∗ marks a bootstrap 95% CI on

\rho

excluding

1

n=34

per group, identical reader, action set and budget

M{=}15

. HippoRAG2 is absent because it indexes the whole corpus, so all four groups meet the same store and the gradient cannot separate them; it is compared in Table III instead.

Fig. 3: The document network the agent built. Top: the store at three points in training; every document is drawn, and position is a t-SNE projection of its embedding under the same function the store indexes with, so documents drawn near each other are documents the reader’s search retrieves together (t-SNE preserves neighbourhoods, not distances, so only local proximity should be read). One layout, computed once on the final store and keyed by document, is shared by all panels, so a document holds its position throughout and only the structure drawn over it changes; the clusters visible at iteration 0 are the universe’s natural topic groups. Blue: source documents. Orange: authored index documents, sized by degree. Green: links. The final panel (bottom right, at two thirds width) carries four index documents labelled, together with the documents each one reaches (purple). The four were selected for correctness, not size: every link each of them carries points at a document that genuinely belongs under its key (Sec. VII-D ), and together they span what the agent produced — an attribute index over a city, an attribute index over a hobby, a one-hop relation, and a two-hop relation. Structure grows from nothing into a navigation layer over the semantic space: hub indexes fan out into topic clusters, and every link is index-mediated — the agent never wires two source documents to each other directly.

Fig. 4: (a) What the store accumulates, reconstructed by trace replay. Index documents = agent-authored documents still alive; net documents added = store size minus the initial 5,864. The two curves coincide: the agent adds access structure and essentially never rewrites source documents, which is the parsimony objective realized (growth

+4.9\%

over 200 iterations). Link growth flattens after the epoch boundary (dashed), the store saturating on the keys the training questions name. (b) Out-degree of every index document in the final store. The distribution is long-tailed — median 5, mean 6.7, max 33 — with the right tail carrying the attribute indexes of Table V and a spike at zero:

43

indexes (

15\%

) were opened and never filled.

“Residents of Dorringham”
Key the index is built on	$n$	deg.	prec.	recall
attribute: city	8	26.4	98%	95%
attribute: hobby / job	54	12.7	98%	96%
“People whose hobby is astronomy”
single-entity hub	91	8.0	95%	90%
“Bennett Coldwater”

TABLE V: Index construction quality, by the kind of key the index is built on, with one example of each.

Agentic RAG

Agentic retrieval-augmented generation lets language models plan searches, use tools, and refine evidence before producing grounded answers.

See all 246 papers

Training the Knowledge Base through Evidence Distillation and Write-Back Enrichment
Mar 26, 2026Yuxing Lu, Xukai Zhao, Wei Wu +1Retrieval-Augmented Generation Knowledge Augmentation for Language Models
Peking University · Georgia Institute of Technology · Tsinghua University
Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
Oct 15, 2026Anikait Singh, Teresa Zhang, Yoonho Lee +3Agentic RAG Memory-Augmented Language Model Agents
Stanford University · Carnegie Mellon University
RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees
Oct 8, 2026Meghanadh Pulivarthi, Swaraj Kumar Biswal, Kushagra Bhushan +3Retrieval-Augmented Generation Structured RAG
IBM · IIT Kharagpur

Mar 26, 2026J/K move · Enter open · S save

Training the Knowledge Base through Evidence Distillation and Write-Back Enrichment

Yuxing Lu, Xukai Zhao, Wei Wu, Jinzhuo Wang

Peking University · Georgia Institute of Technology · Tsinghua University

Retrieval-Augmented Generation Knowledge Augmentation for Language Models Knowledge Distillation

Open paper AskPDF arXiv

The knowledge base in a retrieval-augmented generation (RAG) system is typically assembled once and never revised, even though the facts a query requires are often fragmented across documents and buried in irrelevant content. We argue that the knowledge base should be treated as a trainable component and propose WriteBack-RAG, a framework that uses labeled examples to identify where retrieval succeeds, isolate the relevant documents, and distill them into compact knowledge units that are indexed alongside the original corpus. Because the method modifies only the corpus, it can be applied once as an offline preprocessing step and combined with any RAG pipeline. Across four RAG methods, six benchmarks, and two LLM backbones, WriteBack-RAG improves every evaluated setting, with gains averaging +2.14%. Cross-method transfer experiments further show that the distilled knowledge benefits RAG pipelines other than the one used to produce it, confirming that the improvement resides in the corpus itself.

Abstract

Agentic RAG

Explore similar work

Training the Knowledge Base through Evidence Distillation and Write-Back Enrichment

Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries

RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees