Upgrade the Retriever First: What an LLM Call per Chunk Buys in RAG

Bhavik Mangla

cs.CLMar 8, 2026

Upgrade the Retriever First: What an LLM Call per Chunk Buys in RAG

Authors: Bhavik Mangla

Organizations: Independent Research

Abstract

Many RAG pipelines pay one LLM call per chunk at indexing time to add titles, summaries, keywords and questions (enrichment) or document context (contextual retrieval, CR). We ask what that call buys over free structure and a better retriever. Our advice: upgrade the retriever first, chunk at headers, and enrich mainly for lexical retrieval. Under a plan made public before any index was built, 7B enrichment beats a free section-path prefix under a hybrid retriever by +5.5 recall points on Qasper; on FreshStack its gain (+2.7) does not survive Holm adjustment. It is equivalent to CR, which sends 5.8-8.7 times the input tokens, and Qasper answer F1 (7B reader) is equivalent for both contrasts. Header-led chunks beat 512-character windows; on FreshStack they lose to overlapping 256-token windows. Under qwen3-embedding and a reranker, the two strongest of eight retrievers (named as secondary before any outcome was computed), the CIs of 7B enrichment against the free prefix fall within the equivalence margins; under BM25 it gains +8.7. Exploratory analyses attribute the hybrid gain to document expansion and to repairing an untuned rank fusion; the free prefix under either strong retriever beats hybrid enrichment by +3.4 to +6.3 points, though the best 120B-indexed configuration leads the best free one by +2.5 on Qasper. Add-ons built on the call do no better than LLM-free controls, and late chunking lowers Qasper recall. We release the plan, harness and per-question results.

Figures & tables

Panel of Figure 1 — Figure 1: Paid indexing lifts Qasper retrieval under BM25 and the default hybrid, and, with the 7B indexing LLM, barely moves it under qwen3-embedding or a reranker. Mean primary metric (Qasper rec@512t; FreshStack prec@1024t) per indexing arm (x-axis) and retriever (lines). Arms: free section-path prefix (free; S+T), enrichment (E), contextual retrieval (CR) and their 120B-indexed versions (E s , CR s ; CR s Qasper only); grey numbers are indexing-LLM input tokens per chunk. The free prefix under qwen3 or the reranker scores above every paid arm under the hybrid. Paired differences with CIs: Table 2 , Figure 2 .

Arm	Boundaries; indexed text	Calls
F512c	512-character windows; text	0
F256t	256-token windows, 32 overlap; text	0
P ∘	paragraphs packed to $\tau_{\max}$ , headers ignored; text	0
FL ∘	token windows of median S length; text	0
S	header-led; text	0
S+T	header-led; section path + text	0

Table 1: Arms. Evidence is always matched against chunk text only. Calls are LLM calls per header-led chunk at indexing time. ∘ Alignment controls. s : 120B indexing LLM.

Confirmatory: primary family (TOST, 90% CI)
			Qasper (rec@512t; $\pm$ 5)		FreshStack (prec@1024t; $\pm$ 4)
H	A $-$ B	Retriever	Diff. [CI]	Verdict	Diff. [CI]	Verdict
H1	E $-$ S+T	hybrid	+5.5 [+3.3, +7.7]	diff.	+2.7 [+0.9, +4.7]	diff. †
H2	E $-$ CR	hybrid	+2.1 [+0.1, +4.0]	equiv.	+1.8 [+0.2, +3.4]	equiv.
Confirmatory: H3 family (two-sided, 95% CI)
H3a	S $-$ F512c	hybrid	+7.5 [+4.8, +10.1]	diff.	+3.1 [+0.9, +5.5]	diff.

Table 2: Enrichment beats the free prefix only on Qasper under the hybrid, matches CR, and is equivalent to the free prefix under qwen3-embedding and the reranker. Main tests on 951 Qasper and 235 FreshStack questions, with equivalence margins in the header. Paired differences in points with BCa cluster-bootstrap CIs. Confirmatory verdicts follow § 4.5 within each Holm family; in TOST rows equiv. takes precedence, and diff. is judged on the 95% CI (not shown); every secondary-retriever CI lies inside the margin. s : 120B indexing LLM ( gpt-oss-120b ); CR s was built for Qasper only. Answer F1 uses a fixed 7B reader and a

\pm

5 F1 margin. rerank: Qwen3-Reranker-0.6B over the hybrid top-30. † diff. by the plan’s rule (95% CI [+0.5, +5.1] excludes 0); not significant after Holm adjustment (sign-flip

p

0.057). On Qasper F512c and F256t both score 35.0.

Panel of Figure 2 — Figure 2: E $-$ S+T under eight retrievers, in points, with 95% BCa cluster-bootstrap CIs. Filled circles: Qasper rec@512t; open squares: FreshStack prec@1024t. Rows above the divider rank with BM25, alone or fused; Hybrid+rerank re-scores the hybrid’s top 30. Hybrid is the confirmatory H1 test. BM25, mxbai and qwen3 are secondary retrievers in the plan; BM25+qwen3, Hybrid+rerank and jina-v3 were added as secondary by Amendment 1; nomic alone is exploratory. CIs outside the primary row are unadjusted.

Step	Do	Evidence (Qasper; FreshStack)
1 Retriever	stronger embedder over S+T; a reranker if the index cannot change	S+T (qwen3) $-$ E (hybrid): +5.5; +3.8 x ; E $-$ S+T (qwen3): +1.5; +0.3 s
2 Structure	header-led chunks; the section-path prefix is optional	S $-$ F512c: +7.5; +3.1 c ; S $-$ F256t, FreshStack: $-$ 2.9 c ; S+T $-$ S: +0.2; +1.3 x
3 Enrichment	mainly for BM25-led stacks; one call, not CR	E $-$ S+T, BM25: +8.7; +2.4 s ; E $-$ CR equiv. c
4 Add-ons	skip key dictionary, key-based merging, related-key expansion, late chunking	Qasper: E+KX $-$ E $-$ 10.0; late chunking $-$ S $-$ 11.4 s

Table 3: Where to spend, in order. Differences in points of the primary metric. Tiers: c confirmatory; s pre-specified secondary; x exploratory. Appendix Table 4 gives the full guide.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Situation	Use	Evidence (Qasper; FreshStack)
Can re-embed the corpus	qwen3-class embedder over S+T; skip enrichment	S+T (qwen3) $-$ E (hybrid): +5.5; +3.8
Cannot re-index	reranker over the existing hybrid	S+T (rerank) $-$ E (hybrid): +6.3; +3.4
Strong retriever already in place	skip 7B enrichment; 120B indexing (E s or CR s ) kept +2.5 on Qasper at the top end (exploratory, best of the grid)	E $-$ S+T, qwen3: +1.5; +0.3; rerank: +0.4; +0.8
Keyword-only stack (no vectors, no reranker)	enrichment (document expansion)	E $-$ S+T, BM25: +8.7; +2.4
Paying for an indexing call	enrichment, not CR	H2 equiv.; CR sends $\times$ 5.8–8.7 input tokens
One call or one per field	one call	H6 equiv.; per-field calls cost $\times$ 4.0 calls

Appendix

Table 4: Practitioner guide. Differences in points of the primary metric; rows 1–2, the untuned-RRF row and E+AX

-

E are exploratory (§ 5.2 ); the others are confirmatory, add-on or secondary-retriever results. The guide reflects the tested models and corpora.

Arm	Qasper F1	FreshStack nugget rec. †
F512c	37.8	41.0
F256t	34.2	41.0
S	37.7	42.4
S+T	37.6	43.7
E	39.0	44.2
CR	38.8	42.4

Appendix

Table 5: Answer-level outcomes under hybrid retrieval, in points. The reader is qwen2.5:7b at temperature 0, given 512 (Qasper) or 1,024 (FreshStack) tokens of retrieved source text. Qasper differences: 90% CI with the key-secondary verdict (margin

\pm

5). † Exploratory: strict nugget recall judged by llama-3.3-70b , whose negative control failed (6.8%

>

5%); 95% CIs.

A $-$ B (retriever)	Qasper	FreshStack
S $-$ P (hybrid)	+5.2 [+2.7, +7.8]	$-$ 2.0 [ $-$ 4.5, +0.5]
S $-$ FL (hybrid)	+8.2 [+5.3, +11.1]	+1.6 [ $-$ 0.6, +3.9]
S+T $-$ S (hybrid)	+0.2 [ $-$ 1.6, +2.1]	+1.3 [ $-$ 0.4, +3.2]
E s $-$ E (hybrid)	+1.2 [ $-$ 1.3, +3.7]	$-$ 1.0 [ $-$ 2.9, +0.9]
E $-$ S+T (hybrid+rerank)	+0.4 [ $-$ 1.7, +2.5]	+0.8 [ $-$ 1.4, +3.0]
E $-$ CR (hybrid+rerank)	+0.3 [ $-$ 1.9, +2.5]	+0.9 [ $-$ 1.4, +3.5]

Appendix

Table 6: Secondary comparisons in points, with unadjusted 95% BCa CIs. rerank: Qwen3-Reranker-0.6B over the hybrid top-30. LC-S: late chunking over header-led chunks. Best configurations over all arms and retrievers: Qasper CR s reranked (57.0) against S+T reranked (54.5); on FreshStack the best configuration, P reranked (36.2), is free. The best configurations were selected post hoc; their CI does not account for the selection. Full grids are in Appendix G .

Panel of Figure 3 — Figure 3: Qasper E $-$ S+T (points, 95% cluster-bootstrap CIs, exploratory) for three question groups: the generated fields of the gold chunk supply a question term missing from its section path and text (20% of questions); a term is missing but not supplied; no term is missing. The supplied-term group gains most under every retriever, most of all where BM25 enters the ranking.

Qasper (rec@512t)
H	A $-$ B	Diff. [CI]	Verdict
H4a	M $-$ RM	+1.9 [+0.3, +3.5]	n.d.
H4b	M $-$ SM	$-$ 0.1 [ $-$ 1.9, +1.8]	n.d.
H4c	M $-$ E	$-$ 2.2 [ $-$ 3.6, $-$ 0.9]	equiv.
H4d	E $-$ E $-$ K	+0.7 [ $-$ 1.0, +2.4]	equiv.
H7	M $-$ M $-$ TP	+4.1 [+1.7, +6.6]	diff.

Appendix

Table 7: Add-on tests (Amendments 2–4; pre-specified secondary), hybrid retriever, paired differences in points. M: key-based merging; RM, SM: count-matched random and similarity merges; E

-

K: no key dictionary; M

-

TP: key-based merging on keys consolidated after the fact; KX, AX: related-key and adjacent expansion; PF: one call per field (100 Qasper papers). CIs: 90% for TOST rows (H4c, H4d, H6), 95% otherwise. Holm families: H4 (8 tests), H7 (2), H5 (4). n.d.: no difference detected.

	Keys	Reuse	Merges	$\Delta$ [95% CI]
Qasper	no dict.	6.1	390	$-$ 0.9 [ $-$ 2.1, +0.1]
	dict.	16.2	995	$-$ 2.2 [ $-$ 3.8, $-$ 0.7]
	TP	36.3	2,029	$-$ 5.6 [ $-$ 8.0, $-$ 3.4]
FreshStack	no dict.	5.1	328	+0.5 [ $-$ 0.2, +1.3]
	dict.	13.2	853	+0.3 [ $-$ 1.0, +1.6]
	TP	28.6	1,724	+4.4 [+2.2, +6.8]

Appendix

Table 8: Key reuse and merging (exploratory). Reuse: share of chunks (%) whose key appeared earlier in the document. Merges: chunks removed by key-based merging. TP: keys of E

-

K consolidated by a TopicGPT-style call per document, as in M

-

TP.

\Delta

: merged set minus its own unmerged enrichment (E

-

K for “no dict.” and TP, E for “dict.”), hybrid retriever, primary metric, in points.

Arm	BM25	nomic	mxbai	Hybrid	qwen3	BM25+qwen3	Hyb.+rerank	jina-v3
F512c	26.8	38.1	36.4	35.0	43.2	36.9	44.0	–
F256t	24.1	37.5	35.5	35.0	44.6	38.3	43.6	–
FL	23.8	38.4	35.7	34.3	42.2	36.1	42.7	–
P	26.8	39.7	37.9	37.2	48.1	39.7	47.3	–
S	29.7	47.9	44.4	42.5	52.0	44.0	53.0	46.4
S+T	29.9	47.8	47.2	42.7	53.6	45.2	54.5	45.9

Appendix

Table 9: Qasper: mean rec@512t (

\times

100) for every arm under every retriever. LC-S: late-chunked S, only under jina-v3; CR s was built for Qasper only. Per-question values and 95% CIs are in the released results.

Arm	rec@256t	rec@512t	rec@1024t	hit@1	hit@3	hit@5	answer_f1
F512c	20.9	35.0	54.1	16.8	39.1	51.4	37.8
F256t	21.2	35.0	53.4	31.0	58.3	73.2	34.2
FL	19.2	34.3	55.1	26.3	53.4	69.4	–
P	23.9	37.2	57.4	32.9	62.7	75.9	–
S	27.9	42.5	62.6	33.2	60.7	76.7	37.7
S+T	28.1	42.7	62.4	33.9	62.3	74.4	37.6

Appendix

Table 10: Qasper: secondary metrics (

\times

100) under the hybrid retriever, including the add-on arms (M

-

K: merged E

-

K; E(meta+key): keys and entities also indexed). RM is averaged over five seeds; PF covers 100 papers.

Arm	BM25	nomic	mxbai	Hybrid	qwen3	BM25+qwen3	Hyb.+rerank	jina-v3
F512c	18.8	22.4	27.2	23.6	29.4	28.2	29.3	–
F256t	23.8	27.8	30.8	29.5	34.5	34.6	35.1	–
FL	21.1	22.7	28.5	25.1	31.8	30.2	31.8	–
P	23.9	27.5	32.8	28.7	34.5	32.6	36.2	–
S	22.0	24.2	30.3	26.7	31.9	31.1	31.2	33.7
S+T	22.7	26.9	32.2	28.0	34.6	32.0	34.1	33.6

Appendix

Table 11: FreshStack: mean prec@1024t (

\times

100) for every arm under every retriever. LC-S: late-chunked S, only under jina-v3; CR s was built for Qasper only. Per-question values and 95% CIs are in the released results.

Arm	prec@512t	prec@1024t	prec@2048t	rec@1024t	hit@1	hit@5	mrr@10
F512c	27.4	23.6	20.1	6.5	30.2	61.3	42.9
F256t	32.4	29.5	25.3	8.3	34.0	60.4	45.2
FL	28.0	25.1	21.5	7.2	33.6	62.1	44.7
P	33.2	28.7	25.0	8.6	34.9	63.0	46.4
S	31.6	26.7	22.7	7.4	30.6	59.6	44.2
S+T	32.6	28.0	23.5	7.8	34.9	64.3	47.0

Appendix

Table 12: FreshStack: secondary metrics (

\times

100) under the hybrid retriever, including the add-on arms (M

-

K: merged E

-

K; E(meta+key): keys and entities also indexed). RM is averaged over five seeds.

Pilot
	Qasper		FreshStack
Arm	In	Out	In	Out
Zero-call arms	0	0	0	0
CR (7B)	5,520	58	9,825	48
E (7B)	1,002	188	1,065	168
E without dictionary	892	188	–	–

Appendix

Table 13: Mean input and output tokens per LLM call. Pilot rows come from the pilot results files (one call per header-led chunk: 715 on Qasper, 742 on FreshStack). Confirmatory rows come from the server’s token counters. Output tokens of the 120B indexing LLM are as billed by the provider.

	Qasper (rec@512t)		FreshStack (prec@1024t)
A $-$ B	Hybrid	BM25	Hybrid	BM25
S $-$ F512c	+23.0 [+12.9,+33.5]	+9.9 [+0.6,+18.3]	+4.9 [+1.2,+8.8]	+3.0 [+0.0,+5.9]
S $-$ F256t	+12.7 [+5.5,+20.2]	+4.6 [ $-$ 3.6,+12.7]	$-$ 2.3 [ $-$ 6.2,+1.4]	$-$ 2.2 [ $-$ 6.0,+1.5]
E $-$ S+T	$-$ 2.3 [ $-$ 8.6,+4.3]	+6.8 [ $-$ 0.8,+14.8]	+0.9 [ $-$ 2.9,+4.3]	$-$ 0.1 [ $-$ 3.0,+2.9] ≡
E $-$ CR	$-$ 4.1 [ $-$ 10.4,+2.3]	+2.4 [ $-$ 5.6,+10.4]	$-$ 0.2 [ $-$ 3.9,+3.7]	$-$ 0.5 [ $-$ 3.7,+2.8]
E+M $-$ E	+2.4 [ $-$ 0.2,+5.9]	$-$ 3.1 [ $-$ 9.3,+3.1]	+0.5 [ $-$ 1.1,+2.1] ≡	+0.5 [ $-$ 1.8,+2.7] ≡

Appendix

Table 14: Pilot (exploratory). Planned paired differences in points, with 95% percentile cluster-bootstrap CIs. Bold: survives Holm adjustment over all 22 planned pilot cells. ≡ : equivalent within

\pm

3 points (TOST). FreshStack cells use the corrected Laravel commit and, for E, CR and E+M, the rebuilt indexes (Appendix I.1 ). Apart from FreshStack BM25 E

-

S+T after the correction, no E

-

S+T or E

-

CR cell is equivalent within

\pm

5 (Qasper) or

\pm

3 (FreshStack).

Retr.	A $-$ B	Wrong commit	Corrected
Hybrid	S $-$ F512c	+5.1 [+1.4, +9.0]	+4.9 [+1.2, +8.8]
Hybrid	S $-$ F256t	$-$ 2.6 [ $-$ 6.4, +1.2]	$-$ 2.3 [ $-$ 6.2, +1.4]
BM25	S $-$ F512c	+3.0 [ $-$ 0.0, +6.0]	+3.0 [+0.0, +5.9]
BM25	S $-$ F256t	$-$ 2.2 [ $-$ 6.0, +1.5]	$-$ 2.2 [ $-$ 6.0, +1.5]

Appendix

Table 15: FreshStack pilot RQ1 cells (prec@1024t, points, 95% bootstrap CI over the 73 questions) before and after correcting the Laravel commit. The left column comes from the original bootstrap, so its CIs can differ by 0.1 from a recomputation.

Domain	Commit (date)	Chunks matched	.md files	Doc tokens	S chunks	Questions kept	Gold files
Laravel	724c31cc (2024-10-24)	455/455	98	644k	3,860	89	71
Angular	7d9b38e9 (2024-10-28)	383/384	283	387k	2,307	99	118
YOLO	8d203cf4 (2024-10-26)	565/565	335	622k	3,893	47	68

Appendix

Table 16: FreshStack corpora at the identified commits (counts before the distractor cap). Questions: Laravel 184 total, 22 without Markdown gold, 73 pilot questions excluded; Angular 129 total, 30 without Markdown gold; YOLO 57 total, 10 without Markdown gold. The single unmatched Angular chunk is in image-optimization.md , where FreshStack removed an inline base64 image; no kept gold span is affected. Of the 235 kept questions, 163 have all their Markdown gold under the docs root.

Method	Boundaries	Added text	Context seen	Merging	LLM calls
Fixed-size	Char./token window	None	–	None	0
Meta-Chunking ( Zhao et al., 2024 )	LM perplexity / margin sampling	Chunk rewriting (later version)	Neighbouring sentences	Combine to target length	LM scoring per sentence
LumberChunker ( Duarte et al., 2024 )	LLM-detected topic shift	None	Group of passages	None	$\approx$ 1 per boundary
Element-type ( Jimeno Yepes et al., 2024 )	Document elements	None	–	None	0
Title chain ( Yang, 2026 )	Markdown headers	Section-title prefix	Section path	None	0
Late chunking ( Günther et al., 2024 )	Any	None (contextual embeddings)	Whole document (embedder)	None	0

Appendix

Table 17: How indexing methods place boundaries, what text they add, what context the model sees, how they merge or regroup chunks, and how many LLM calls they make per chunk. The last block lists the technique tested here and its optional add-ons (§ 3 ). “–” means not applicable or not comparable per chunk.

Panel of Figure 4 — Figure 4: Indexing with MDKeyChunker on one document. Enrichment processes chunks in order. Call $i$ receives the summary $s_{i-1}$ of the previous chunk and, when the rolling key dictionary is enabled (arm E), the dictionary $K_{i-1}$ ; it returns key $k_{i}$ and summary $s_{i}$ , and $K$ is reset per document. Dashed boxes are optional add-ons, tested in § 4.6 ; related-key expansion acts at retrieval time and is not shown. Chunking and merging make no model calls.

Field	Type	Requested	Applied in code
title	string	3–8 words	stripped; $\leq$ 1,000 chars
summary	string	1–2 sentences, 30–60 words, what makes the chunk unique	stripped; $\leq$ 1,000 chars
keywords	string[]	5–8 specific, domain-relevant terms	a string is split on commas; non-strings dropped
entities	object[]	name and type $\in$ {PERSON, ORG, LOC, TECH, CONCEPT, EVENT, METRIC}	items without a name dropped; missing type, or a bare string item, becomes CONCEPT
questions	string[]	2–3 natural questions the chunk answers	a string becomes a one-item list
key	string	specific subtopic, 2–5 words, lowercase; seven rules (see text)	stripped; $\leq$ 80 chars; lowercased

Appendix

Table 18: Fields requested in the single enrichment call, as specified in the prompt, and the checks the code applies to the response.

Document Chunking

Document chunking splits long or structured documents into coherent units for retrieval, question answering, and other language-model tasks.

See all 53 papers

Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora
Oct 7, 2026Andrey Kuehlkamp, Priscila Correa Saboia Moreira, Samuel RundInformation Retrieval Dense Retrieval
Center for Research Computing, University of Notre Dame Notre Dame, Indiana, USA
MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval
Jun 16, 2026Amirhossein Abaskohi, Raymond Li, Gaetano Cimino +3Retrieval-Augmented Generation Information Retrieval
University of British Columbia · University of Salerno · ServiceNow Research
Document Retrieval-Aware Chunking (D-RAC): Universal Retrieval-Aware Ingestion of Enterprise Documents via PDF Normalization and Multimodal Markdown Conversion
Sep 21, 2026Uday Allu, Abhivanth Sivaprakash, Pratik Singh +1Document Chunking Retrieval-Augmented Generation
AI Research Team Yellow.ai

Oct 7, 2026J/K move · Enter open · S save

Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora

Andrey Kuehlkamp, Priscila Correa Saboia Moreira, Samuel Rund

Center for Research Computing, University of Notre Dame Notre Dame, Indiana, USA

Information Retrieval Dense Retrieval Document Chunking

Open paper AskPDF arXiv

Retrieval-augmented generation systems increasingly rely on document-structure treatments: structure-aligned chunking, LLM-generated chunk contexts, heading-path metadata, and hierarchical two-stage retrieval. Separate studies support each on different corpora, embedders, and metrics, and none control for a shared confound: any text prepended to a chunk perturbs its embedding. We present a mechanism-isolating ablation testing all four treatments under one protocol, matching chunk sizes across conditions and adding a semantically null placebo---heading paths that are structurally valid but shuffled across documents. We score retrieval with a coverage-aware nDCG and test four pre-registered contrasts via document-clustered bootstrap with Holm correction, on two distant corpora: 200 Wikipedia Featured Articles (951 queries) and 1,585 QASPER papers (4,303 questions). Organization helps, and the cause is content, not tokens: structure-aligned chunks with real heading paths beat contextualized fixed windows (+0.022 / +0.012 cov-nDCG@10) and the placebo (+0.010 / +0.016). Naive two-stage hierarchical retrieval hurts (-0.033 / -0.015), traceable to first-stage section recall. Gold structure beats LLM-induced structure on Wikipedia but not on QASPER. Effects are small (

dz

0.06-0.11) but Holm-significant and consistent across corpora.

Abstract

Document Chunking

Explore similar work

Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora

MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval

Document Retrieval-Aware Chunking (D-RAC): Universal Retrieval-Aware Ingestion of Enterprise Documents via PDF Normalization and Multimodal Markdown Conversion