Tree Navigation Without LLM Summaries: A Matched-Cost Study of Hierarchical Retrieval for Long-Document QA

Priyank Jayraj; Poonam Goyal; Navneet Goyal

cs.CLSep 30, 2026

Tree Navigation Without LLM Summaries: A Matched-Cost Study of Hierarchical Retrieval for Long-Document QA

Authors: Priyank Jayraj, Poonam Goyal, Navneet Goyal

Organizations: Birla Institute of Technology and Science, Pilani Pilani, Rajasthan, India

Abstract

Retrieval-augmented generation grounds language models in external context, but for long documents flat top- $k$ retrieval can cluster on a single region and miss complementary evidence. RAPTOR-style summary trees address this by recursively clustering chunks and using a language model to summarize each cluster at indexing time, then ranking summary nodes alongside raw chunks at query time. We show the main benefit of summary trees in long-document QA can come from navigation rather than the generated summary content. We introduce NavTree, a leaves-only retriever that builds a deterministic balanced segment tree over chunks (zero language-model calls at indexing) and uses the tree purely as a navigation scaffold: a hybrid lexical-and-dense frontier walk, anchored on top retrieved leaves, descends from the root and emits only leaf chunks to the reader. On a matched-cost evaluation against flat retrievers and an extractive re-implementation of RAPTOR, NavTree is the strongest matched-cost hierarchical retriever in our evaluated grid and ties the strongest flat baseline. On long-document multi-hop QA, it is the only hierarchical method that significantly beats BM25 on a class-vs-class basis, corroborated by a reader-free retrieval-recall check. A matched-reader replication of the published abstractive RAPTOR variant, given strong cluster summaries, still loses to NavTree at every multi-chunk budget, at zero indexing cost. The ranking carries across stronger and open-weight readers, a stronger encoder, and a full factorial that isolates leaves-only emission as the structural lever.

Figures & tables

Figure 1: Architecture of NavTree. (A) Offline indexing. NavTree splits a document into sentence-aware chunks, embeds them as leaf nodes $\{v_{1},\ldots,v_{n}\}$ , and builds a deterministic balanced segment tree. Internal nodes $\bar{c}_{i:j}$ store conservative concatenations of representative descendant sentences, but no LLM-generated summaries are produced. Leaf and internal-node embeddings are stored in separate FAISS flat inner-product indices, yielding zero LLM calls at indexing time. (B) Query-time navigation. The query embedding $\mathbf{q}$ retrieves top- $K_{\ell}$ leaf anchors, whose root-to-leaf paths define the Steiner-active mask $\mathcal{A}$ . A hybrid dense–BM25 frontier walk descends through this active subtree. Internal nodes are routing-only: they are split but never emitted. The reader receives only selected raw leaf chunks, optionally complemented by a flat top- $k$ tail-fill over unselected leaves. The dashed connector denotes reuse of the same offline tree and FAISS indices.

\widetilde{x}_{v}=\frac{x_{v}-\min_{u\in\mathcal{F}}x_{u}}{\max_{u\in\mathcal{F}}x_{u}-\min_{u\in\mathcal{F}}x_{u}+\epsilon}

Algorithm 1 NavTree: navigated leaves-only retrieval with flat fill

Budget	Flat				Hierarchical
	BM25	SBERT	DPR	Hybrid	RAPTOR	NavTree
QuALITY (acc, %)
256	59.7	60.4	60.6	61.2	59.3	60.3
512	65.0	66.3	67.2	67.2	65.5	65.8
1024	71.0	71.6	72.1	72.2	68.1	72.5 ⋆
2000	76.4	76.7	76.6	77.0	71.3	76.7 ⋆

Table 1: Matched-cost head-to-head, grouped by class. Columns left of the divider are flat baselines; columns right are hierarchical methods. Reader = gpt-4o-mini ,

T{=}0

. Bold = best overall in the row; underline = best hierarchical when a flat method wins. ⋆ = NavTree beats RAPTOR-Ext at

p{<}0.01

(paired sign test). Worst-case 95% CI half-widths: QuALITY:

\pm 2.0

pp; QASPER:

\pm 0.026

. The full grid including NarrativeQA-summary (structurally uninformative:

\sim

7 chunks/doc; all methods within

0.01

ROUGE-L at

\geq 1024

tokens) and LongBench-SD (

N{=}86

\pm 10

pp CI half-width) is reported in Table 7 .

within-paper reader scaling (same slice):
Method	Summaries	Acc.	HARD
RAPTOR-Extractive (ours)	extractive	81.0	70.4
Abstractive RAPTOR (ours)	gpt-4o-mini (LLM)	80.8	71.1
Abstractive RAPTOR (ours)	gpt-4o (LLM, stronger)	84.3	75.5
NavTree (ours)	none	85.4	77.4
NavTree, reader $=$ gpt-4o	none	93.0	89.9

Table 2: Matched-reader RAPTOR comparison and reader scaling on QuALITY@2000. First

20

QuALITY-dev docs (

N{=}369

questions,

N_{h}{=}159

HARD); same chunking, embedder, prompt, and reader ( gpt-4o-mini ,

T{=}0

) across the four matched-slice rows — they differ only in retrieval method. Abstractive-RAPTOR rows replace deterministic cluster summaries with LLM-generated ones (the variant published in Sarthi et al., 2024 ), at two summary-LLM strengths. NavTree (no LLM summaries) wins at

\geq{}512

-token budgets against both RAPTOR variants; stronger gpt-4o summaries narrow the @

2000

gap to

{+}1.1

pp (HARD:

{+}1.9

pp) but the @

1024

gap remains

{+}4.9

pp (HARD:

{+}6.9

pp), and NavTree indexes at zero LLM calls. The published RAPTOR+GPT-4 number (

82.6

76.2

on full

N{=}2086

) is not on this slice and used a now-deprecated reader; we make no cross-paper claim.

Figure 2: Token budget vs. metric on the two headline matched-cost datasets ( gpt-4o-mini reader). NavTree (green) leads RAPTOR -Extractive on QuALITY at all budgets and overtakes it on QASPER at

\geq

1024 tokens. NarrativeQA-summary and LongBench-SD curves (appendix; Table 7 ) are not shown here because both datasets are uninformative as retrieval discriminators at this budget grid.

HotpotQA (token F1)
Budget	BM25	SBERT-flat	Hybrid	RAPTOR-Ext	NavTree (ours)
256	0.473	0.440	0.494	0.456	0.429
512	0.546	0.497	0.536	0.506	0.501
1024	0.588	0.558	0.572	0.555	0.569
2000	0.602	0.560	0.596	0.552	0.601
2WikiMQA (token F1)

Table 3: Long-document QA, token F1 on LongBench-formatted HotpotQA, 2WikiMultiHopQA, MuSiQue, and full-book NarrativeQA (200 questions each, 5–23k-token contexts). Reader = gpt-4o-mini . Bold = best per row.

HotpotQA (ans@k, $N{=}160$ )
Method	@256	@512	@1024	@2000
BM25	52.5	68.8	83.1	92.5
SBERT-flat	44.4	64.4	78.1	85.0
RAPTOR-Ext	42.5	64.4	72.5	82.5
Hybrid	50.0	70.6	78.1	91.9
NavTree (ours)	51.2	65.6	78.1	91.9

Table 4: Reader-free retrieval recall on multi-hop QA (answer-presence@k: fraction of questions where the gold answer string appears in the retrieved context, normalized whitespace, case-insensitive). Examples are LongBench-formatted HotpotQA/2WikiMQA/MuSiQue (

N{=}506

total). Bold = best per (dataset, budget). Decouples retrieval quality from the reader.

Figure 3: Long-document QA token F1 on HotpotQA, 2WikiMQA, and full-book NarrativeQA (

N{=}200

each; MuSiQue in Table 3 ). NavTree (green) overtakes RAPTOR -Extractive (red) by 2000 tokens on all three; on 2WikiMQA already at 1024 tokens.

Method	Wall(s)	Int. nodes	LLM $
NavTree (ours)	386	1858	$0.00
RAPTOR-Ext. (re-impl)	241	682	$0.00
RAPTOR-Orig. (Sarthi et al.)	$\geq 241+$ LLM	682	$0.14

Table 5: Index-time cost on a 60-doc benchmark slice (1,918 leaves across QuALITY, QASPER, NarrativeQA). NavTree and RAPTOR-Extractive make zero LLM calls; RAPTOR-Original requires one summary call per internal node; estimated cost shown at gpt-4o-mini pricing (

{\sim}30{\times}

at GPT-4o pricing).

Configuration (cumulative)	Acc. (%)	HARD (%)
Classic Steiner (anchors only, no hybrid, no tail-fill)	76.32	67.23
+ $K_{\ell}{=}48$ anchors	75.02	66.01
+ hybrid frontier ( $\alpha{=}0.35$ )	76.65	66.95
+ hybrid frontier ( $\alpha{=}0.50$ )	76.80	67.89
Full NavTree (default)	76.27	66.76

Table 6: Knob ablation on the QuALITY@2000 winning cell (

N{=}2086

). Each row toggles one design choice; rows 3–4 isolate hybrid frontier weight

\alpha

as the dominant retrieval-side lever. Bold = best per column. Hybrid frontier scoring contributes more than

K_{\ell}

selection, and the default configuration is statistically indistinguishable from the per-row peak (

\pm 0.5{\,\textrm{pp}}

Figure 4: Hyper-parameter sweep on QuALITY@1024. Left:

\alpha

K_{\ell}{=}10

, inverted-U peak at

\alpha{=}0.35

. Right:

K_{\ell}

\alpha{=}0.35

;

K_{\ell}{=}5

–

10

are equivalent within reader noise.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Budget	Flat				Hierarchical
	BM25	SBERT	DPR	Hybrid	RAPTOR	NavTree
QuALITY (acc, %)
256	59.7	60.4	60.6	61.2	59.3	60.3
512	65.0	66.3	67.2	67.2	65.5	65.8
1024	71.0	71.6	72.1	72.2	68.1	72.5
2000	76.4	76.7	76.6	77.0	71.3	76.7

Appendix

Table 7: Full matched-cost grid (appendix). Same protocol as Table 1 , extended to NarrativeQA-summary (mean

\sim

7 chunks per document; retrieval is structurally uninformative at

\geq 1024

tokens, where the budget admits the full document) and LongBench-SD (

N{=}86

per cell; CI half-width

\pm 10

pp). Significance markers omitted on these two datasets due to small effective

N

. Worst-case 95% CI half-widths: QuALITY:

\pm 2.0

pp; QASPER:

\pm 0.026

; NarrativeQA:

\pm 0.004

; LongBench-SD:

\pm 10

pp.

Method	256	512	1024	2000
BM25	59.7	65.0	71.0	76.4
SBERT-flat	60.4	66.3	71.6	76.7
DPR	60.6	67.2	72.1	76.6
RAPTOR-Extractive	59.3	65.5	68.1	71.3
NavTree (ours)	60.3	65.8	72.5	76.7

Appendix

Table 8: QuALITY accuracy (%) on the full dev split (

N{=}2086

). Reader = gpt-4o-mini. Bold = best per column.

Method	256	512	1024	2000
BM25	0.293	0.324	0.365	0.399
SBERT-flat	0.342	0.383	0.393	0.407
DPR	0.303	0.346	0.382	0.396
RAPTOR-Extractive	0.342	0.368	0.385	0.397
NavTree (ours)	0.311	0.354	0.401	0.400

Appendix

Table 9: QASPER token F1 on a 150-paper subset (

N\approx 519

). Reader = gpt-4o-mini.

Method	256	512	1024	2000
BM25	0.167	0.173	0.173	0.173
SBERT-flat	0.164	0.172	0.172	0.174
DPR	0.160	0.172	0.173	0.174
RAPTOR-Extractive	0.158	0.166	0.170	0.170
NavTree (ours)	0.164	0.165	0.165	0.164

Appendix

Table 10: NarrativeQA ROUGE-L on Wikipedia plot summaries (30 docs,

N{=}913

). Reader = gpt-4o-mini.

Method	256	512	1024	2000
BM25	41.9	39.5	38.4	46.5
SBERT-flat	41.9	36.0	37.2	41.9
DPR	31.2	25.0	25.0	25.0
RAPTOR-Extractive	37.2	34.9	34.9	34.9
NavTree (ours)	41.9	37.2	34.9	43.0

Appendix

Table 11: LongBench Single-Doc QA accuracy (%) on 86 documents. Reader = gpt-4o-mini.

#	Config (cumulative)	F1	$\Delta$ vs RAPTOR
1	Classic Steiner (K=10)	0.308	$-0.032$
2	+ K_leaf=48 only	0.302	$-0.038$
3	+ hybrid frontier ( $\alpha$ =0.35), no tight tail	0.314	$-0.026$
4	+ QASPER $\alpha$ =0.50	0.307	$-0.033$
5	+ tight tail	0.307	$-0.033$
6	+ CE rerank (full)	0.351	$+0.012$

Appendix

Table 12: Knob ablation on QASPER@256 (

N{=}519

). Each row adds one NavTree design choice on top of the previous. RAPTOR-Extractive baseline F1 = 0.340. Cross-encoder rerank is the dominant single lever at this tight-budget regime; other retrieval-side knobs each move F1 by

\leq{\pm}0.01

Cell	Steiner	Hybrid $\alpha$	$K_{\ell}$	CE	Acc. %
A0 B0 C1 D1	NavTree	$0$	$48$	✓	72.60
A1 B0 C0 D1	classic	$0$	$10$	✓	72.60
A1 B1 C1 D1	classic	$0.35$	$48$	✓	72.37
A1 B1 C0 D1	classic	$0.35$	$10$	✓	72.28
A0 B0 C0 D1	NavTree	$0$	$10$	✓	72.15
A0 B1 C0 D0	NavTree	$0.35$	$10$	–	72.04

Appendix

Table 13:

2^{4}

factorial ablation on the QuALITY @ 2000 winning cell (

N{=}886

questions per cell, same subset across cells for paired comparison). Knobs: Steiner (classic Steiner walk vs. NavTree’s anchor-free formulation), Hybrid

\alpha

(BM25+dense vs. dense-only frontier),

K_{\ell}

(leaf anchor count), CE (cross-encoder rerank of leaf pool). Main effects (averaged over the other knobs): Steiner:

-0.11

pp, Hybrid

\alpha

+0.23

pp,

K_{\ell}

-0.81

pp, CE rerank:

+1.26

pp.

	$\alpha{=}0.00$	$\alpha{=}0.35$	$\alpha{=}0.65$
$K_{\ell}{=}5$	0.403	0.405	0.407
$K_{\ell}{=}10$	0.401	0.404	0.401

Appendix

Table 14: Hyper-parameter sweep on QASPER@1024 (

N{\approx}519

): token F1 across three hybrid weights

\alpha

and two leaf-anchor counts

K_{\ell}

. Bold = peak; underlined = QuALITY-tuned default (

\alpha{=}0.35

K_{\ell}{=}10

). Reader = gpt-4o-mini .

Condition	QuALITY accuracy (%)
Summary alone	55.7
Best-leaf alone	50.4
Both correct	41.7
Summary only correct (16)	13.9
Leaf only correct (10)	8.7
Neither correct	35.7

Appendix

Table 15: Direct reader-side head-to-head on 115 stratified (document, QuALITY question) pairs. For each pair we built the segment tree, located the highest-cosine internal node, and asked gpt-4o-mini the question with one context only: the extractive summary text of that node, or the best leaf descendant of that node. The summary modestly beats the leaf at the per-node level, which runs counter to the cosine-only argument and motivates the packing-effect explanation in Section 6.1 (c).

NavTree vs. …	Budget	$\Delta$	$p$	W/L/T	$N$
QuALITY (Acc, $N{=}2086$ )
RAPTOR-Ext	256	$+0.009$	$0.41$	246/227/1613	2086
RAPTOR-Ext	512	$+0.003$	$0.82$	237/231/1618	2086
RAPTOR-Ext	1024	$+0.045$	$4.8\!\times\!10^{-8\,\star}$	191/98/1797	2086
RAPTOR-Ext	2000	$+0.054$	$5.6\!\times\!10^{-11\,\star}$	203/91/1792	2086
DPR-flat	256	$-0.003$	$0.80$	269/276/1541	2086

Appendix

Table 16: Paired sign tests, NavTree vs. every other method, on every cached cell. For each method pair we report the per-question paired delta (NavTree minus other), the binomial sign-test

p

-value, and the win/loss/tie counts (NavTree wins / NavTree loses / ties), all computed over the intersection of question IDs where both methods have predictions.

{\star}

marks rejections at

p<0.01

. The QuALITY rows omit BM25/SBERT-flat because per-question predictions for those methods on QuALITY were not preserved on disk (only aggregate scores in Table 1 ). All 64 outputs are released as results/paired_tests/*.json .

QuALITY @1024 (acc, %; $N{=}2086$ )
Method	MiniLM-L6 (384-d)	BGE-large-en-v1.5 (1024-d)	$\Delta$
BM25	$71.00$	$70.52$	$-0.48$
RAPTOR-Ext	$68.07$	$70.42$	$+2.35$
Hybrid-flat	$72.22$	$74.78$	$+2.56$
NavTree (ours)	$\mathbf{72.53}$	$\mathbf{74.78}$	$+2.25$
QASPER @1024 (token F1; $N{=}519$ )

Appendix

Table 17: Encoder sensitivity at @1024. We rebuild the FAISS index with BAAI/bge-large-en-v1.5 (1024-d) in place of the canonical all-MiniLM-L6-v2 (384-d) and re-run all four retrievers under the same gpt-4o-mini reader at

T{=}0

\Delta

is BGE-large minus canonical. Read both rows together: on QuALITY the encoder upgrade is symmetric across dense methods (

+2.3

+2.6

pp) and the relative ranking is preserved (NavTree retains its

+4.36

pp hierarchical-class lead, NavTree ties hybrid-flat exactly). On QASPER the upgrade is not symmetric: hybrid-flat gains

+0.039

F1, but NavTree and RAPTOR-Ext slightly regress, and hybrid-flat overtakes NavTree by

+0.004

F1 (within

0.026

CI half-width). NavTree’s structural prior therefore pays off most when the encoder is moderate; under a stronger encoder hybrid-flat catches up on QASPER and ties on QuALITY. This is consistent with the W2 reading in Section 5.1 , that the class-vs-class signal lives on multi-hop QA.

QuALITY (acc %)	100	200	400	800
BM25	38.4	39.5	39.7	39.6
SBERT-flat	38.7	39.5	40.0	39.5
RAPTOR-Ext.	39.7	39.6	38.4	39.2
NavTree	38.4	38.9	38.6	38.6
QASPER (token F1)	100	200	400	800
BM25	0.199	0.237	0.252	0.224

Appendix

Table 18: Weak-reader sensitivity study with UnifiedQA-base (220 M T5 parameters). Budgets are tighter (100–800) because UnifiedQA-base has a 512-token context window. All retrieval methods cluster within a

\sim

1 pp band, confirming our claim that retrieval gaps shrink when the reader is the bottleneck (see §Limitations).

	BM25	RAPTOR-Ext	Hybrid-flat	NavTree
QuALITY @ 1024 — accuracy (%)
Llama-3.1-8B ( $N{=}273$ )	$54.9$	$53.1$	$54.9$	$\mathbf{55.7}$
Qwen-2.5-7B ( $N{=}273$ )	$62.3$	$62.6$	$63.7$	$\mathbf{64.1}$
ref: gpt-4o-mini (full $N{=}2086$ )	$71.0$	$68.1$	$72.2$	$\mathbf{72.5}$
QASPER @ 1024 — token F1
Llama-3.1-8B ( $N{=}179$ )	$0.301$	$0.314$	$0.325$	$\mathbf{0.338}$

Appendix

Table 19: Open-weight reader replication, two readers via local Ollama on Apple M2. Same retrieval, same chunking (

100

-token sentence-aware), same encoder ( all-MiniLM-L6-v2 ), same prompts,

T{=}0

, no API calls. We run Meta-Llama-3.1-8B-Instruct (4-bit Q4_K_M ,

{\approx}\,5

GB) and Qwen-2.5-7B-Instruct (4-bit Q4_K_M ,

{\approx}\,5

GB). Top: aggregate scores under each reader, with the canonical gpt-4o-mini cells from Table 1 reproduced for sanity reference. Bottom: per-question paired sign tests, NavTree minus baseline;

p

is the binomial sign-test

p

-value, W/L/T counts ignore ties;

{\star}

marks

p<0.01

N

is necessarily smaller than the canonical grid because each generation takes

{\sim}10

s on the M2; the reproducible launcher is scripts/run_a1_ollama.sh (override READER_MODEL=qwen2.5:7b-instruct-q4_K_M for the Qwen row), with predictions in results/openweight_run/ .

Method	@256		@512		@1024		@2000
	any@k	frac@k	any@k	frac@k	any@k	frac@k	any@k	frac@k
BM25	51.0	28.8	81.9	58.0	97.5	87.9	100.0	98.7
SBERT-flat	57.8	37.3	84.8	68.0	99.0	93.2	100.0	98.7
RAPTOR-Extractive	55.4	35.9	76.5	60.9	92.6	83.3	100.0	96.8
Hybrid-flat	61.8	40.4	85.8	68.9	99.0	93.0	100.0	99.1
NavTree	53.9	31.5	83.3	62.9	99.5	93.9	100.0	99.1

Appendix

Table 20: Reader-free retrieval recall on QASPER ( N=204 questions, 1269 gold evidence paragraphs across 60 papers). any@k = fraction of questions where at least one gold evidence paragraph has

\geq 0.50

token-set overlap with the retrieved context; frac@k = mean fraction of gold paragraphs covered per question. Decouples retrieval quality from the reader.

Long-Context Retrieval

Long-context retrieval finds and uses relevant evidence across lengthy documents, conversations, and model contexts.

See all 45 papers

RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees
Oct 8, 2026Meghanadh Pulivarthi, Swaraj Kumar Biswal, Kushagra Bhushan +3Retrieval-Augmented Generation Structured RAG
IBM · IIT Kharagpur
Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation
May 1, 2026Ziwen Zhao, Menglin YangMulti-Hop QA Retrieval-Augmented Generation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answering
Jun 9, 2026Xiangjun Zai, Xingyu Tan, Chen Chen +2Retrieval-Augmented Generation Graph Retrieval-Augmented Generation
University of New South Wales, Australia · CSIRO, Australia · University of Wollongong, Australia

Oct 8, 2026J/K move · Enter open · S save

RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees

Meghanadh Pulivarthi, Swaraj Kumar Biswal, Kushagra Bhushan, Yatin Nandwani+2

IBM · IIT Kharagpur

Retrieval-Augmented Generation Structured RAG Agentic RAG Question Answering

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) grounds language models in external corpora. Agentic RAG enables iterative search, yet exposes the model to isolated chunks without document structure, making it difficult to distinguish relevant evidence from chunks that merely resemble the query. Structure-aware methods such as PageIndex navigate document structure but cannot scale to the structures of large corpora, which do not fit in the LLM context. Hence, they first commit to a single document using a document retriever and cannot recover from a wrong choice. We propose RIT-RAG (Retrieval-Induced Tree RAG), which combines content retrieval with structural navigation. Offline, RIT-RAG builds a tree for each document from its table of contents or sitemap. At query time, it retrieves a broad set of chunks and uses their positions to induce manageable sub-trees, potentially across multiple documents. An LLM agent navigates these sub-trees, selectively reads promising nodes, and reformulates queries when needed. Thus, retrieval proposes where to look, while the agent decides what to read. Across financial, scientific, and customer-support benchmarks, RIT-RAG achieves the highest answer accuracy among vanilla, graph-based, and agentic baselines. On EntQABench, our new benchmark of 2.84 million technical-documentation webpages, it improves accuracy by 6.8 to 11.4 points over the strongest baseline across three LLMs.

Abstract

Long-Context Retrieval

Explore similar work

RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees

Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation

Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answering