Panini: Continual Learning in Token Space via Structured Memory

Shreyas Rajesh; Pavan Holur; Mehmet Yigit Turali; Chenda Duan; Vwani Roychowdhury

cs.AIFeb 16, 2026

Panini: Continual Learning in Token Space via Structured Memory

Authors: Shreyas Rajesh, Pavan Holur, Mehmet Yigit Turali, Chenda Duan, Vwani Roychowdhury

Organizations: Department of Electrical and Computer Engineering, University of California, Los Angeles, USA.

Abstract

Language models are increasingly used to reason over content they were not trained on, such as new documents, evolving knowledge, and user-specific data. A common approach is retrieval-augmented generation (RAG), which stores verbatim documents externally (as chunks) and retrieves only a relevant subset at inference time for an LLM to reason over. However, this results in inefficient usage of test-time compute (LLM repeatedly reasons over the same documents); moreover, chunk retrieval can inject irrelevant context that increases unsupported generation. We propose a human-like non-parametric continual learning framework, where the base model remains fixed, and learning occurs by integrating each new experience into an external semantic memory state that accumulates and consolidates itself continually. We present Panini, which realizes this by representing documents as Generative Semantic Workspaces (GSW) -- an entity- and event-aware network of question-answer (QA) pairs, sufficient for an LLM to reconstruct the experienced situations and mine latent knowledge via reasoning-grounded inference chains on the network. Given a query, Panini only traverses the continually-updated GSW (not the verbatim documents or chunks), and retrieves the most likely inference chains. Across six QA benchmarks, Panini achieves the highest average performance, 5%-7% higher than other competitive baselines, while using 2-30x fewer answer-context tokens, supports fully open-source pipelines, and reduces unsupported answers on curated unanswerable queries. The results show that efficient and accurate structuring of experiences at write time -- as achieved by the GSW framework -- yields both efficiency and reliability gains at read time. Code is available at https://github.com/roychowdhuryresearch/gsw-memory.

Figures & tables

Figure 1: A non-parametric continual learning (NPCL) framework schematics (1) Continual experience : incoming documents are processed asynchronously, potentially by different agents. (2) Individual workspaces : each experience is encoded into a Generative Semantic Workspace (GSW). (3) Continually learned global workspace : GSWs can be continually consolidated by reconciling entities, events, and actions both across and within documents. Extensive ablation studies (see Table 11 ) show that different combinations of LLM models of different sizes for performing different tasks – GSW generation, and retrieval – lead to consistently robust performance. Thus GSW can be used as a shared meta-representation. (4) Reasoning-grounded inference : The goal is to have enough reconciliation –but not exhaustive– so that all latent knowledge supported by the collection of experiences are represented by inference chains/paths .

Figure 2: System overview of PANINI at inference time. Step 1: Planning: A decomposition LLM converts the user query into an ordered sequence of single-hop sub-questions. Step 2: RICR: We perform chain-based retrieval by expanding candidate paths hop-by-hop. The initial seed set is obtained via embedding similarity; therefore, for a query like “Who was Lothair II’s mother?”, retrieval may include both Lothair II and the semantically nearby Lothair I . From these seeds, RICR follows QA edges to propose intermediate entities (e.g., candidate mothers) and incrementally extends partial chains across GSWs. Candidate chains are scored at each hop, and low-scoring paths are pruned. Step 3: Answer Generation: Top-ranked chains are de-duplicated and provided to the final answering LLM.

	NQ	PopQA	MuSiQue	2Wiki	HotpotQA	LV-Eval
Queries	1,000	1,000	1,000	1,000	1,000	124
Passages	9,633	8,676	11,656	6,119	9,811	22,849

Table 1: Dataset statistics. We use the same benchmark splits as HippoRAG 2 ( Gutiérrez et al., 2025a ) .

	Simple QA		Multi-Hop QA
Retrieval	NQ	PopQA	MuSiQue	2Wiki	HotpotQA	LV-Eval	Avg
No Retrieval
None	52.7	22.7	22.0	36.3	41.0	5.0	29.9
Sparse Retrieval
BM25	56.0	53.0	25.3	35.4	54.2	7.8	38.6
BM25 + reranker	58.2	56.8	29.9	44.1	63.4	8.1	43.4

Table 2: Performance (F1

\uparrow

) comparison across six QA benchmarks. Panini achieves the highest average performance (56.06%). Bold = best; underline = second best.

	Simple QA		Multi-Hop QA				Avg
Retrieval	NQ	PopQA	MuSiQue	2Wiki	HotpotQA	LV-Eval	Avg Token Count
Chunk Retrieval Methods
Standard Retrieval	701.97	665.40	569.28	544.53	650.35	1099.78	705.27
Structure-Augmented RAG
RAPTOR	1161.7	1100.8	942.1	900.79	1076.1	1820.1	1166.6
GraphRAG	8076.4	7658.9	6554.7	6265.1	7487.0	12661.2	8121.6

Table 3: Average token count (Tokens

\downarrow

) in generation context across methods. Panini uses 2–30

\times

fewer tokens than competitive baselines. Bold = best; underline = second best.

	MuSiQue Platinum		2Wiki Platinum		Avg
Retrieval	Ans $\uparrow$	Unans $\uparrow$	Ans $\uparrow$	Unans $\uparrow$	Ans $\uparrow$	Unans $\uparrow$
Sparse Retrieval
BM25	35.0	78.4	37.9	79.8	36.4	79.1
BM25 + reranker	42.3	74.5	47.9	78.7	45.1	76.6
Dense Retrieval
Qwen3 (8B)	53.3	63.4	63.5	70.9	58.4	67.2

Table 4: Platinum dataset evaluation with GPT-4o-mini. Ans = answerable, Unans = unanswerable. Bold = best; underline = second best.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Input passage (verbatim)

Title: Lothair II Text: Lothair II (835 – ) was the king of Lotharingia from 855 until his death. He was the second son of Emperor Lothair I and Ermengarde of Tours. He was married to Teutberga (died 875), daughter of Boso the Elder.

GSW Output

Entities (role

\rightarrow

states). • Lothair II : person

\rightarrow

{deceased, historical figure}; ruler

\rightarrow

{king of Lotharingia, medieval period} • 835 : date

\rightarrow

{birth year, medieval period} • Lotharingia : location

\rightarrow

{kingdom, medieval period} • 855 : date

\rightarrow

{start of reign, medieval period} • Emperor Lothair I : person

\rightarrow

{historical figure}; ruler

\rightarrow

{Carolingian Emperor, medieval period} • Ermengarde of Tours : person

\rightarrow

{historical figure}; nobility

\rightarrow

{medieval period} • Teutberga : person

\rightarrow

{deceased, historical figure}; nobility

\rightarrow

{medieval period} • 875 : date

\rightarrow

{death year, medieval period} • Boso the Elder : person

\rightarrow

{historical figure}; nobility

\rightarrow

{medieval period} Verb phrase node Bidirectional QA pairs (with roles/states on answers) king of • Q: Who was the king of Lotharingia? A: Lothair II ( person

\rightarrow

{deceased, historical figure}; ruler

\rightarrow

{king of Lotharingia, medieval period}) • Q: What was Lothair II the king of? A: Lotharingia ( location

\rightarrow

{kingdom, medieval period}) reigned from • Q: When did Lothair II start his reign? A: 855 ( date

\rightarrow

{start of reign, medieval period}) • Q: Who started reigning in 855? A: Lothair II ( person

\rightarrow

{deceased, historical figure}; ruler

\rightarrow

{king of Lotharingia, medieval period}) son of • Q: Who is the son of Emperor Lothair I? A: Lothair II ( person

\rightarrow

{deceased, historical figure}; ruler

\rightarrow

{king of Lotharingia, medieval period}) • Q: Who is Lothair II the son of? A: Emperor Lothair I ( person

\rightarrow

{historical figure}; ruler

\rightarrow

{Carolingian Emperor, medieval period}) son of • Q: Who is the son of Ermengarde of Tours? A: Lothair II ( person

\rightarrow

{deceased, historical figure}; ruler

\rightarrow

{king of Lotharingia, medieval period}) • Q: Who is Lothair II the son of? A: Ermengarde of Tours ( person

\rightarrow

{historical figure}; nobility

\rightarrow

{medieval period}) married to • Q: Who was Lothair II married to? A: Teutberga ( person

\rightarrow

{deceased, historical figure}; nobility

\rightarrow

{medieval period}) • Q: Who was married to Teutberga? A: Lothair II ( person

\rightarrow

{deceased, historical figure}; ruler

\rightarrow

{king of Lotharingia, medieval period}) daughter of • Q: Who is the daughter of Boso the Elder? A: Teutberga ( person

\rightarrow

{deceased, historical figure}; nobility

\rightarrow

{medieval period}) • Q: Who is Teutberga the daughter of? A: Boso the Elder ( person

\rightarrow

{historical figure}; nobility

\rightarrow

{medieval period})

Appendix

Figure 3: Example of a per-document Generative Semantic Workspace (GSW). Top: the raw input passage (title + text). Bottom: the corresponding GSW, rendered as (i) entity nodes annotated with roles and states, and (ii) verb-phrase nodes instantiated as bidirectional question–answer (QA) pairs.

Parameter	Non-thinking Mode	Thinking Mode
Temperature	0.6	0.6
Top- $p$	0.95	0.95
Top- $k$	20	20
Max tokens	4096	$12,288$
Repetition penalty	–	1.1
Presence penalty	–	0.3

Appendix

Table 5: Decoding hyperparameters for answer generation with open-source reader models.

Hop Type	Number of Examples
2-hop	494
3hop1	441
3hop2	406
4hop1	225
4hop2	49
4hop3	165

Appendix

Table 6: Distribution of question decomposition training data by hop type.

Retrieval	MuSiQue $\uparrow$	2Wiki $\uparrow$	HotpotQA $\uparrow$	Avg $\uparrow$
Sparse Retrieval
BM25	17.07	36.74	48.30	34.04
BM25 + reranker	23.16	43.78	57.67	41.54
Dense Retrieval
Qwen3-Embedding (8B)	40.14	56.50	69.47	55.37
Qwen3-Embedding (8B) + reranker	46.30	58.34	72.04	58.89

Appendix

Table 7: Performance comparison across multi-hop QA benchmarks with Qwen3-4B (no-think mode). Experiments were run on GPT-4.1-mini -generated GSW memories, using Qwen3-4B (no-think mode) as the reader model and Qwen3-4B + LoRA for question decomposition (open-source PANINI pipeline). Bold = best; underline = second best.

	Simple QA		Multi-Hop QA
Retrieval	NQ $\uparrow$	PopQA $\uparrow$	MuSiQue $\uparrow$	2Wiki $\uparrow$	HotpotQA $\uparrow$	Avg $\uparrow$
Sparse Retrieval
BM25	45.01	47.61	18.96	29.28	48.80	37.93
BM25 + reranker	53.57	51.27	24.34	38.61	59.95	45.55
Dense Retrieval
Qwen3-Embedding (8B)	59.90	59.74	39.97	55.66	69.26	56.91

Appendix

Table 8: Performance comparison across QA benchmarks with Qwen3-8B(no-think mode). Experiments were run on GPT-4.1-mini -generated GSW memories, using Qwen3-8B (no-think mode) as the reader and Qwen3-8B + LoRA for question decomposition (open-source PANINI pipeline). Bold = best; underline = second best.

	MuSiQue Platinum		2Wiki Platinum		Avg
Retrieval	Ans $\uparrow$	Unans $\uparrow$	Ans $\uparrow$	Unans $\uparrow$	Ans $\uparrow$	Unans $\uparrow$
Sparse Retrieval
BM25	20.17	82.35	30.24	88.30	25.21	85.33
BM25 + reranker	27.79	76.47	40.37	83.33	34.08	79.90
Dense Retrieval
Qwen3-Embedding (8B)	40.88	66.01	49.02	75.18	44.95	70.60

Appendix

Table 9: Platinum dataset evaluation with Qwen3-8B (no-think mode). Ans = F1 score on the answerable subset. Unans = refusal accuracy on the unanswerable subset (binary; correct iff the output is the canonical non-answer token N/A after normalization). Experiments were run on GPT-4.1-mini -generated GSW memories, using Qwen3-8B (no-think mode) as the reader and Qwen3-8B + LoRA for question decomposition (open-source PANINI pipeline). Bold = best; underline = second best.

	MuSiQue Platinum		2Wiki Platinum		Avg
Retrieval	Ans $\uparrow$	Unans $\uparrow$	Ans $\uparrow$	Unans $\uparrow$	Ans $\uparrow$	Unans $\uparrow$
Sparse Retrieval
BM25	18.04	89.54	28.62	93.97	23.33	91.76
BM25 + reranker	21.71	86.93	36.61	87.23	29.16	87.08
Dense Retrieval
Qwen3-Embedding (8B)	36.89	82.35	47.13	74.47	42.01	78.41

Appendix

Table 10: Platinum dataset evaluation with Qwen3-8B (thinking mode). Ans = F1 score on the answerable subset. Unans = refusal accuracy on the unanswerable subset (binary; correct iff the output is the canonical non-answer token N/A after normalization). Experiments were run on GPT-4.1-mini -generated GSW memories, using Qwen3-8B (thinking mode) as the reader and Qwen3-8B + LoRA for question decomposition (open-source PANINI pipeline). Bold = best; underline = second best.

Model	2Wiki	MuSiQue
Qwen3-8B	55.38	40.24
Qwen3-14B	62.80	44.87
Qwen3-14B (Second Pass)	66.76	48.28
GPT-OSS-120B	70.65	48.50
HippoRAG2 (Qwen3-8B reader)	66.93	45.40
PANINI (Main Exp)	72.4	52.3

Appendix

Table 11: Performance comparison when varying the open-source model used for write-time GSW construction. All open-source models use the same read-time configuration: LoRA-finetuned Qwen3-8B for question decomposition and Qwen3-8B (no-think) as the reader. We also report HippoRAG2 under the same reader model and the PANINI configuration from Table 2 as reference points.

Setting	MuSiQue	2Wiki
Full Panini	52.3	72.4
No QA reranking	18.4	22.2
No dual search (entity-only)	39.5	68.6
No dual search (QA-only)	52.3	72.0
No decomposition	36.8	47.2
Beam width = 10	52.4	72.7

Appendix

Table 12: Ablation study on multi-hop QA (F1 score, 1000 questions).

Setting	MuSiQue		2Wiki
	F1	Tokens	F1	Tokens
Beam width = 5	52.3	192	72.4	315
Beam width = 10	52.4	320	72.7	409
Beam width = 3	52.3	143	72.2	231
Beam width = 1	44.6	82	67.3	171

Appendix

Table 13: Ablation study on multi-hop QA (F1 score and average token count, 1000 questions).

Scoring rule	MuSiQue
None (greedy last-hop)	50.81
Similarity	42.32
Combined	48.24
Main (cumulative)	52.3

Appendix

Table 14: Ablation of chain-level scoring on MuSiQue ( F1 , 1,000 questions). “Main” denotes cumulative (geometric-mean) scoring; “Similarity” uses only question–chain embedding similarity; “Combined” linearly interpolates the two (

\alpha=0.5

); “None” disables chain-level scoring and performs greedy last-hop selection.

Method	Write-time cost
PANINI (Batch)	$48.02
PANINI (Standard, est.)	$96.04
HippoRAG 2	$91.8
Qwen3-Embedding (8B)	$0

Appendix

Table 15: Write-time indexing cost on MuSiQue (11,656 passages). Costs are one-time dataset-level indexing costs. PANINI write-time corresponds to per-document GSW construction with gpt-4.1-mini ; the Batch variant uses the OpenAI Batch API, while the Standard variant is an estimate assuming a 2

\times

higher cost without the Batch discount. HippoRAG 2 cost is the reported total run cost using GPT-4o-mini with standard API calls as default. Qwen3-Embedding (8B) has zero write-time LLM cost since it performs embedding-only indexing.

	Simple QA		Multi-Hop QA				Avg
Retrieval	NQ	PopQA	MuSiQue	2Wiki	HotpotQA	LV-Eval	Avg
Search-R1	47.9	49.7	41.1	64.9	68.6	11.5	47.3
Search-R1 (Dense)	41.6	52.7	45.1	64.6	71.3	18.8	49.0
Search-R1 (Dense + QA)	48.5	51.5	44.6	65.4	66.8	12.0	48.1
Search-R1 (Dense + QA + Retrieval)	48.1	54.0	46.6	67.6	70.0	10.5	49.4
Panini	67.4	57.6	52.3	72.4	71.9	14.8	56.1

Appendix

Table 16: Performance of Search-R1 under different retrieval configurations. All rows use the same Search-R1 agent model and differ only in the retrieval mechanism. Search-R1 uses BM25 over document chunks; Search-R1 (Dense) replaces BM25 with dense embedding retrieval ( Qwen3-Embedding-8B ) over document chunks; Search-R1 (Dense + QA) further replaces document chunks with the structured QA pairs from the GSW; Search-R1 (Dense + QA + Dual Index) additionally incorporates PANINI’s RICR retrieval strategy and retrieves relevant QA pairs. PANINI is included as a reference point and uses the same evaluation setting as in Table 2 .

Query	Method	Failure mode	Failure reason
What year did the publisher of Labyrinth end?	Qwen3-Emb	missing_second_hop	Retrieved passages broadly about Labyrinth and publishing, but missed the bridging evidence required for the second hop.
What is the location of formation of the film company distributing The Boss ?	Qwen3-Emb	missing_second_hop	Over-matched the surface phrase “The Boss” and returned loosely related documents, failing to retrieve the distributor $\rightarrow$ company-formation link.
Between the state university in the state without North Point Mall and where Edwards won the primary and the university in Fort Hill’s town, which has the more national championships?	Qwen3-Emb	missing_entity	Requires multiple supporting documents; failed to retrieve the intermediate evidence involving North Point Mall, breaking the chain.
Who’s the son of the Italian navigator who explored the eastern coast of the continent Manuel Balbi was born in?	Qwen3-Emb	broken_chains	Retrieved the correct answer passage, but missed the intermediate evidence (continent identification and the relevant navigator), so the chain could not be supported.
What is the name of the famous bridge in the birth city of the composer of Scanderbeg ?	HippoRAG2	retrieval_drift	Drifted to a plausible distractor in the same city (e.g., another famous Venice bridge), and failed to rank the correct bridge-related evidence highest.
Who is the child of the president under whom prohibition occurred?	HippoRAG2	retrieval_drift	Found evidence about prohibition but drifted to nearby presidents/laws, losing the correct intermediate link needed to answer the composed query.

Appendix

Table 17: Representative failure cases for dense embedding retrieval (Qwen3-Embedding-8B) and graph-based retrieval (HippoRAG2).

Failure mode	Example query	What goes wrong	Why it breaks the chain
Missing verb-phrase nodes	What nationality is Arabia (Daughter Of Justin II)’s mother?	The GSW omits the verb-phrase/event capturing Aelia Sophia as an empress of the Byzantine Empire .	Without this relation node, the GSW does not contain any QA pair linking the mother to a country or polity, so the required evidence for determining nationality is never retrieved.
QA-pair construction errors	Who is the paternal grandfather of Dibyasambandh?	The relation daughter of is stored, but QA pairs are only generated in the father $\rightarrow$ daughter direction; the inverse daughter $\rightarrow$ father QA is missing.	Since the father entity never appears as the answer to any QA pair, subsequent hops remain conditioned on the daughter entity, blocking retrieval of evidence about the paternal grandfather.
Imperfect question decomposition	When did the country that has the original language of the film named after Vladimir Karalić’s birthplace as a co-official language first attend the Olympics as an independent team?	The decomposer drops co- and queries for official language instead of co-official language.	The weakened constraint retrieves a different country, causing subsequent hops to follow an incorrect chain even if later retrieval is accurate.

Appendix

Table 18: Representative qualitative error cases for PANINI.

	Qwen3-Embedding (8B)	Search-R1	HippoRAG 2	PANINI
Indexing Time (min)	1.9	1.9	106.6	100.1
QA Time/Query (sec)	1.3	6.7	4.4	3.3

Appendix

Table 19: Computational resource requirements for end-to-end runtime on MuSiQue (11,656 passages). Indexing time (min) and QA latency per query (sec) for Panini and baselines. QA latency per query is measured with standard (non-batch) API calls.

Figure 4: Prompt used for factual GSW construction from documents.

Figure 5: Prompt used for factual GSW construction from documents (continued).

Figure 6: LLM prompt for 2nd pass GSW Construction.

Figure 7: Prompt used for multi-hop question decomposition into atomic sub-questions.

Figure 8: Oracle-style prompt used for answer generation from retrieved GSW evidence.

End-to-End Retrieval Trace (Step 1/3): Question Decomposition

Question. What is the place of birth of the performer of song Changed It ? System. Chain-Following Multi-Hop QA with Beam Search Mode. cumulative (

\alpha=0.5

), reranker = voyage beam_width

=5

, chain_top_k

=15

, entity_top_k

=20

, qa_rerank_top_k

=15

Original question. “What is the place of birth of the performer of song Changed It?” Decomposed sub-questions. (1) Q1: “Who performed the song Changed It?” [Retrieval: ✓]

\rightarrow

(2) Q2: “What is the place of birth of <ENTITY_Q1> ?” [Retrieval: ✓] Dependencies. Q1 has no dependencies; Q2 depends on Q1. Chain identification. Total chains = 1; Chain 1 indices = [0,1]; Questions = Q1

\rightarrow

Q2.

Appendix

Table 28

End-to-End Retrieval Trace (Step 2.2/3): Retrieval Hop 2 with Beam Search

Hop 2/2 (Q2): “What is the place of birth of <ENTITY_Q1> ?” State. Prior entities = {0: Nicki Minaj}; Active beams = 5; Concrete questions = 5 (Nicki Minaj; Lil Wayne; Changed It; No Frauds; Regret in Your Tears). Top retrieved entities (BM25 + Embedding; top-5 of 20): Rank Entity name 1 Nicki Minaj 2 Nicki Minaj 3 Pink Friday 4 Nicki Minaj 5 Nicki Minaj Retrieved QA pairs after reranking (top-5 of 15; remaining omitted): Rank Question Answer Source 1 Where was Nicki Minaj born? Saint James, Port of Spain Nicki Minaj 2 Who was born in Saint James, Port of Spain? Nicki Minaj Nicki Minaj 3 Who was raised in Queens, New York City? Nicki Minaj Nicki Minaj 4 Where was Nicki Minaj raised? Queens, New York City Nicki Minaj 5 Who performed “Changed It”? Nicki Minaj, Lil Wayne Nicki Minaj Beam expansion (final hop). Use all QA pairs directly (no grouping); candidates = 15. Sample candidate states (first 5): Cand. last_hop Answer entity/entities 1 0.8945 [Saint James, Port of Spain] 2 0.7891 [Nicki Minaj] 3 0.6836 [Nicki Minaj] 4 0.6289 [Queens, New York City] 5 0.5898 [Nicki Minaj, Lil Wayne] Beam pruning (Hop 2). Total candidates = 15; keep top-5 by

(\text{chain\_score},\text{last\_hop\_score})

. Beam details (Hop 2). Includes all kept beams plus the top-ranked pruned beam. Beam chain_score last_hop Status Entity map 1 0.9315 0.8945 ✓ kept {0: Nicki Minaj, 1: [Saint James, Port of Spain]} 2 0.9052 0.7891 ✓ kept {0: Nicki Minaj, 1: [Nicki Minaj]} 3 0.8781 0.6836 ✓ kept {0: Nicki Minaj, 1: [Nicki Minaj]} 4 0.8637 0.6289 ✓ kept {0: Nicki Minaj, 1: [Queens, New York City]} 5 0.8533 0.5898 ✓ kept {0: Nicki Minaj, 1: [Nicki Minaj, Lil Wayne]} 6 0.8406 0.5430

\times

pruned {0: Nicki Minaj, 1: [Onika Tanya Maraj-Petty]}

Appendix

Figure 9: Example end-to-end retrieval trace. Panini decomposes the query into two hops and performs chain-following retrieval with beam search (beam width

=5

). At each hop, candidates are scored using geometric-mean (cumulative) chain scoring, and the final answer is generated from the selected QA evidence.

Figure 10: Continual-learning under corpus growth (MuSiQue, 200 questions). We fix a held-out set of 200 questions and evaluate retrieval+QA F1 as the corpus is incrementally expanded from 4K to the full MuSiQue collection (

\sim

12K passages). The set of relevant passages for these questions is contained in the initial 4K subset and remains unchanged across steps; only the number of distractor passages increases as additional (irrelevant) documents are added. This simulates a continuously evolving corpus where the signal stays constant but the retrieval search space grows.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 233 papers

AutoPrunedRetriever: Persistent Reasoning Graphs for Efficient and Extensible Retrieval-Augmented Generation
Feb 4, 2026Ning Wang, Kuanyan Zhu, Daniel Yuehwoon Yee +4RAG Context Compression Retrieval-Augmented Generation
Cornell University · University of Cambridge · The University of Hong Kong +2
GAM-RAG: Gain-Adaptive Memory for Evolving Retrieval in Retrieval-Augmented Generation
Mar 2, 2026Yifan Wang, Mingxuan Jiang, Zhihao Sun +5Retrieval-Augmented Generation Adaptive RAG
1Fudan University.
PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents
Sep 6, 2026Kun Li, Zexuan Qiu, Tianhua Zhang +2Long-Context Inference Multi-Agent LLMs
The Chinese University of Hong Kong

Feb 4, 2026J/K move · Enter open · S save

AutoPrunedRetriever: Persistent Reasoning Graphs for Efficient and Extensible Retrieval-Augmented Generation

Ning Wang, Kuanyan Zhu, Daniel Yuehwoon Yee, Yitang Gao+3

Cornell University · University of Cambridge · The University of Hong Kong+2

RAG Context Compression Retrieval-Augmented Generation Structured RAG Graph Retrieval-Augmented Generation Retrieval-Augmented Reasoning

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) has become a standard approach for knowledge-intensive LLM tasks. Complex reasoning remains challenging because queries often require composing dispersed evidence through multiple dependent reasoning steps. Yet existing RAG systems process queries independently, repeatedly retrieving overlapping evidence and reconstructing established reasoning. Preserving such reasoning makes previously retrieved evidence and intermediate results reusable for subsequent queries. However, inference-time reuse becomes increasingly costly as the history grows, motivating a compact representation that preserves useful information without substantial loss in quality. To address these challenges, we present AutoPrunedRetriever (APR), a graph-based RAG system that encodes corpus facts and query-derived reasoning in a shared symbolic codebook. The codebook serves both as the retrieval index and as persistent memory, representing facts, questions, and answers as compact edge sequences, so that the previously established structure can be directly retrieved and reused. At inference time, APR retrieves query-specific reasoning paths and constructs compact prompts, while incremental consolidation keeps the growing memory compact. For deployments where the retriever cannot be replaced, we introduce AutoPruned Layer (APL), which leaves host retrieval unchanged but replaces the retrieved context with the corresponding entries in the shared symbolic codebook, writing each question and answer back as a compact structure. Across five complex-reasoning benchmarks, APR improves accuracy over the strongest baseline by 1.8-10.8 points while using up to 96% fewer input tokens. APL reduces generator input by 60-96% across host systems and preserves answer quality on document-style contexts.

Abstract

Continual Learning for LLMs

Explore similar work

AutoPrunedRetriever: Persistent Reasoning Graphs for Efficient and Extensible Retrieval-Augmented Generation

GAM-RAG: Gain-Adaptive Memory for Evolving Retrieval in Retrieval-Augmented Generation

PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents