Retrieval-augmented generation (RAG) has become a standard approach for knowledge-intensive LLM tasks. Complex reasoning remains challenging because queries often require composing dispersed evidence through multiple dependent reasoning steps. Yet existing RAG systems process queries independently, repeatedly retrieving overlapping evidence and reconstructing established reasoning. Preserving such reasoning makes previously retrieved evidence and intermediate results reusable for subsequent queries. However, inference-time reuse becomes increasingly costly as the history grows, motivating a compact representation that preserves useful information without substantial loss in quality. To address these challenges, we present AutoPrunedRetriever (APR), a graph-based RAG system that encodes corpus facts and query-derived reasoning in a shared symbolic codebook. The codebook serves both as the retrieval index and as persistent memory, representing facts, questions, and answers as compact edge sequences, so that the previously established structure can be directly retrieved and reused. At inference time, APR retrieves query-specific reasoning paths and constructs compact prompts, while incremental consolidation keeps the growing memory compact. For deployments where the retriever cannot be replaced, we introduce AutoPruned Layer (APL), which leaves host retrieval unchanged but replaces the retrieved context with the corresponding entries in the shared symbolic codebook, writing each question and answer back as a compact structure. Across five complex-reasoning benchmarks, APR improves accuracy over the strongest baseline by 1.8-10.8 points while using up to 96% fewer input tokens. APL reduces generator input by 60-96% across host systems and preserves answer quality on document-style contexts.
Figures & tables
Figure 1. AutoPrunedRetriever pipeline
System
Medical
Novel
STEM
TV
Quality
APR
.707 / 1.2k
.642 / 938
.776 / 2.1k
.648 / 687
.323 / 727
HippoRAG2
.620 / 1.5k
.534 / 1.5k
.699 / 1.9k
.595 / 1.6k
.304 / 1.5k
PathRAG
.678 / 28k
.502 / 27k
.612 / 24k
.496 / 30k
.303 / 27k
G-Reasoner
.575 / 5.1k
.488 / 5.1k
.625 / 3.4k
.630 / 5.1k
.299 / 3.9k
LightRAG
.613 / 100k
.491 / 101k
.171 / 8.8k
.465 / 3.0k
.196 / 96 *
Table 1. Complex-reasoning accuracy and query-time generator input across five datasets. Best values are in bold . Format is accuracy / tokens . Token counts include the complete prompt passed to the generator, including structured evidence and prompt scaffolding. * denotes a retrieval failure followed by a question-only prompt.
Model
Fact Ret.
Complex Reas.
Ctx. Summ.
Creative Gen.
Avg
ACC
R-L
ACC
R-L
ACC
Cov
ACC
Cov
FS
ACC
Tok
Novel
RAG (no rerank)
58.8
37.4
41.4
15.1
50.1
82.5
41.5
47.5
37.8
48.0
879
RAG (rerank)
60.9
36.1
42.9
15.4
51.3
83.6
38.3
49.2
40.0
48.3
879
MS-GraphRAG
49.3
26.1
50.9
24.1
64.4
75.6
39.1
55.4
35.7
50.9
38.7K
HippoRAG
52.9
26.7
38.5
11.2
48.7
85.6
38.9
71.5
39.0
44.8
7,208
HippoRAG2
60.1
31.4
53.4
33.4
64.1
70.8
48.3
49.8
31.0
56.5
1,008
Table 2. GraphRAG-Benchmark results on Novel and Medical . Best values are in bold and second-best values are underlined . ACC: LLM-judged answer accuracy; R-L: ROUGE-L; Cov: evidence coverage; FS: faithfulness; Avg ACC: macro-average of the four task accuracies; Tok: average generator-input tokens.
Figure 2. Cross-dataset ablation on complex reasoning: change in accuracy when each component is removed, with the full-system accuracy above each group.
Figure 3. APL with host retrieval held fixed: compression (right) is uniform across hosts; accuracy (left) improves where the host’s context is document-shaped.
Figure 4. Robustness, scalability, and cost. (a) Sensitivity to τE . (b) Margin over the strongest baseline under three judges. (c) Persistent-state growth over full streams ( R2≥0.977 ). (d) Graph-construction tokens and (e) per-query serving latency on STEM and TV. (f) LLM-parsed APR vs. APR-REBEL against the strongest baseline.
Large reasoning models such as DeepSeek-R1 and OpenAI o1 generate extended chains of thought spanning thousands of tokens, yet their integration with retrieval-augmented generation (RAG) remains fundamentally misaligned. Current RAG systems optimize for providing context before reasoning begins, while reasoning models require evidence injection during multi-step inference chains. We introduce ReaLM-Retrieve, a reasoning-aware retrieval framework that addresses this mismatch through three key innovations: (1) a step-level uncertainty detector that identifies knowledge gaps at reasoning-step granularity rather than token or sentence level; (2) a retrieval intervention policy that learns when external evidence maximally benefits ongoing reasoning; and (3) an efficiency-optimized integration mechanism that reduces per-retrieval overhead by 3.2x compared to naive integration. Experiments on MuSiQue, HotpotQA, and 2WikiMultiHopQA demonstrate that ReaLM-Retrieve achieves on average 10.1% absolute improvement in answer F1 over standard RAG (range: 9.0-11.8% across the three benchmarks) while reducing retrieval calls by 47% compared to fixed-interval approaches like IRCoT (all improvements significant at p<0.01, paired bootstrap). On the challenging MuSiQue benchmark requiring 2-4 hop reasoning, our method achieves 71.2% F1 with an average of only 1.8 retrieval calls per question. Analysis shows that ReaLM-Retrieve also improves retrieval quality itself, achieving 81.3% Recall@5 with consistently higher precision and MRR than fixed-interval baselines on supporting evidence, establishing new state-of-the-art efficiency-accuracy trade-offs for reasoning-intensive retrieval tasks.
Dongxin Guo, Jikun Wu, Siu Ming Yiu
The University of Hong Kong Hong Kong, China · Brain Investing Limited Hong Kong, China · Stellaris AI Limited Hong Kong, China
Retrieval-Augmented Generation (RAG) has become a standard approach for enhancing large language models (LLMs) with external knowledge, mitigating hallucinations, and improving factuality. However, existing systems rely on generating natural language queries at each hop and maintaining a strict architectural separation between retriever and generator, preventing them from leveraging the full representational capacity of the LLM. We propose LAnR (Latent Abstraction for RAG), a unified framework in which a single LLM jointly performs encoding, retrieval, and generation entirely within its own latent space. Rather than generating textual queries, LAnR produces dense retrieval vectors from the hidden states of a designated [PRED] token and uses them to match against encoded document representations from the same model. Furthermore, LAnR adaptively decides when sufficient evidence has been retrieved using a lightweight MLP control head over those same hidden states, eliminating both the separate retriever and explicit token-level stopping reasoning. This design is motivated by our empirical observation that answer token entropy reliably signals retrieval sufficiency. Extensive experiments on six QA benchmarks spanning single-hop and multi-hop settings demonstrate that LAnR outperforms existing RAG methods, while achieving improved inference efficiency through reduced number of retrieval calls and tighter model integration.
Large Language Models (LLMs) struggle with knowledge-intensive tasks due to hallucinations and fragmented reasoning over dispersed information. While Retrieval-Augmented Generation (RAG) grounds generation in external sources, existing methods often treat evidence as isolated units, failing to reconstruct the logical chains that connect these dots. Inspired by Complementary Learning Systems (CLS), we propose CodaRAG, a framework that evolves retrieval from passive lookup into active associative discovery. CodaRAG operates via a three-stage pipeline: (1) Knowledge Consolidation to unify fragmented extractions into a stable memory substrate; (2) Associative Navigation to traverse the graph via multi-dimensional pathways-semantic, contextualized, and functional-explicitly recovering dispersed evidence chains; and (3) Interference Elimination to prune hyper-associative noise, ensuring a coherent, high-precision reasoning context. On GraphRAG-Bench, CodaRAG achieves absolute gains of 7-10% in retrieval recall and 3-11% in generation accuracy. These results demonstrate CodaRAG's superior ability to systematically robustify associative evidence retrieval for factual, reasoning, and creative tasks.
Cheng-Yen Li, Xuanjun Chen, Claire Lin +4
National Taiwan University, Taipei, Taiwan · NTU Artificial Intelligence Center of Research Excellence, National Taiwan University.