Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora

Andrey Kuehlkamp; Priscila Correa Saboia Moreira; Samuel Rund

cs.IROct 7, 2026

Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora

Authors: Andrey Kuehlkamp, Priscila Correa Saboia Moreira, Samuel Rund

Organizations: Center for Research Computing, University of Notre Dame Notre Dame, Indiana, USA

Abstract

Retrieval-augmented generation systems increasingly rely on document-structure treatments: structure-aligned chunking, LLM-generated chunk contexts, heading-path metadata, and hierarchical two-stage retrieval. Separate studies support each on different corpora, embedders, and metrics, and none control for a shared confound: any text prepended to a chunk perturbs its embedding. We present a mechanism-isolating ablation testing all four treatments under one protocol, matching chunk sizes across conditions and adding a semantically null placebo---heading paths that are structurally valid but shuffled across documents. We score retrieval with a coverage-aware nDCG and test four pre-registered contrasts via document-clustered bootstrap with Holm correction, on two distant corpora: 200 Wikipedia Featured Articles (951 queries) and 1,585 QASPER papers (4,303 questions). Organization helps, and the cause is content, not tokens: structure-aligned chunks with real heading paths beat contextualized fixed windows (+0.022 / +0.012 cov-nDCG@10) and the placebo (+0.010 / +0.016). Naive two-stage hierarchical retrieval hurts (-0.033 / -0.015), traceable to first-stage section recall. Gold structure beats LLM-induced structure on Wikipedia but not on QASPER. Effects are small ( $dz$ 0.06-0.11) but Holm-significant and consistent across corpora.

Figures & tables

Panel of Figure 1 — Figure 1: The four document-structure mechanisms under test. (a) Structure-aligned chunking cuts at section boundaries (A2) instead of ignoring them (A0). (b) Chunk contextualization prepends an LLM-generated blurb to a fixed-size window (A1). (c) Structure-metadata injection prepends the chunk’s real heading path (A3); the placebo (P) prepends an equally-shaped but shuffled, semantically null path. (d) Hierarchical retrieval routes queries to sections before ranking chunks (A4); flat retrieval (A3) searches the full chunk index directly. A4 underperforms because stage 1 frequently drops the gold section before stage 2 ever sees it (Section 6.4 ).

Contrast	Corpus	$\Delta$	95% CI	$p$ (Holm)	$dz$	$p$ (MixedLM)
C1: A3 $-$ A1	Wikipedia-FA	+0.0218	[+0.0052, +0.0380]	0.017	+0.09	0.120
C1: A3 $-$ A1	QASPER	+0.0115	[+0.0055, +0.0175]	$<$ 0.001	+0.06	0.008
C2: A3 $-$ P	Wikipedia-FA	+0.0099	[+0.0024, +0.0173]	0.017	+0.08	0.487
C2: A3 $-$ P	QASPER	+0.0161	[+0.0114, +0.0207]	$<$ 0.001	+0.11	$<$ 0.001
C3: G3 $-$ A3	Wikipedia-FA	+0.0129	[+0.0038, +0.0228]	0.010	+0.10	0.373
C3: G3 $-$ A3	QASPER	+0.0011	[ $-$ 0.0020, +0.0043]	0.485 (n.s.)	+0.01	0.801

Table 1: Pre-registered contrasts, cov-nDCG@10, strong embedder, no reranker. Bootstrap CIs and Holm-corrected

p

-values. MixedLM Seabold and Perktold (2010) ; Seabold et al. (2025) is the conservative secondary model (See sec. 5.2 ).

Corpus	A0	A1	A2	A3	P	A4	G2	G3	G4
Wikipedia-FA	0.362	0.391	0.403	0.413	0.403	0.381	0.417	0.426	0.400
QASPER	0.098	0.114	0.117	0.125	0.109	0.111	0.120	0.126	0.111

Table 2: Per-condition cov-nDCG@10 (strong embedder, no reranker).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Meaning	Induced	Gold
0	fixed-window chunks (size-matched), no prefix	A0	—
1	fixed-window chunks + LLM situating context ( $\leq$ 80 tok)	A1	—
2	structural chunks, no prefix	A2	G2
3	structural chunks + heading-path prefix (depth 3)	A3	G3
4	structural chunks + heading-path prefix + two-stage routing	A4	G4

Appendix

Table 3: Condition levels.

Condition	Boundaries	Prefix	Retrieval	Structure
A0	fixed windows (size-matched)	none	flat	—
A1	fixed windows (size-matched)	LLM situating context ( $\leq$ 80 tok)	flat	—
A2	induced structure	none	flat	induced
A3-d1	induced structure	heading path, depth 1	flat	induced
A3-d2	induced structure	heading path, depth 2	flat	induced
A3	induced structure	heading path, depth 3	flat	induced

Appendix

Table 4: Experimental conditions.

Contrast	Subtraction	Question
C1	A3 $-$ A1	Do structure-aligned boundaries plus native metadata beat the strongest generative treatment on fixed windows?
C2	A3 $-$ P	Does the content of the heading path matter, or is the gain token injection?
C3	G3 $-$ A3	How much is lost when an LLM induces the structure instead of the author?
C4	A4 $-$ A3	Does two-stage hierarchical retrieval beat flat search over the identical index?

Appendix

Table 5: Pre-registered contrasts.

Filter	Threshold	Rationale
min_words	$\geq$ 3,000	long documents where structure can matter
min_depth	$\geq$ 3	heading depth $\geq$ 3 , so a non-trivial tree exists
min_sections	$\geq$ 12	enough sections to route among / cut at
n_docs	200	scale-up target

Appendix

Table 6: Document filtering criteria.

Bucket	Example keywords
history_war	battle, war, military, campaign, siege, history, empire, dynasty
biography	births, deaths, people, biograph, politicians, writers, actors, monarchs
science_nature	species, birds, mammals, physics, chemistry, biology, astronom, geolog
arts_media	albums, songs, films, novels, paintings, opera, television, video games
geo_places	geography, cities, rivers, mountains, roads, buildings, hurricanes, storms
other	(fallback when nothing matches)

Appendix

Table 7: Topic buckets and example keywords.

Bucket	Documents	Share
history_war	60 (at cap)	30.0%
other	60 (at cap)	30.0%
biography	30	15.0%
arts_media	22	11.0%
science_nature	15	7.5%
geo_places	13	6.5%

Appendix

Table 8: Realized distribution, verified 2026-08-04.

Corpus	Usable structure	Required repair	Mean sections/doc
Wikipedia-FA	198 / 200	116	21.9
QASPER	1,579 / 1,585	590	14.5 (gold 14.8)

Appendix

Table 9: Corpus structure statistics.

Corpus	BM25 gold-top-1
Wikipedia-FA	36.3%
QASPER	22.7%

Appendix

Table 10: BM25 gold-top-1 retrieval rate.

Group	Parameter	Value
global	seed	20260710
selection	n_docs / min_words / min_depth / min_sections / max_per_bucket	200 / 3000 / 3 / 12 / 60
chunking	t_max / t_min (content tokens) / fixed_snap_tokens	350 / 60 / 15
breadcrumb	sep / depths	" > " / [1, 2, 3]
querygen	per_type / rouge_max / ngram_block / context_max_tokens	250 / 0.35 / 4 / 80
embedders	weak / strong	bge-small-en-v1.5 / bge-large-en-v1.5

Appendix

Table 11: Key configuration ( configs/scale_crc.yaml ).

Explore similar work

STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation
Sep 3, 2026Vineet Kumar, Meghanadh Pulivarthi, vishwajeet kumar +3Precision Recall Document
IBM
SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG
Jun 16, 2026Amirhossein Abaskohi, Issam H. Laradji, Peter West +1Agentic Retrieval-Augmented Generation Systems Tree Search
University of British Columbia · ServiceNow Research
Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts
Jul 2, 2026Valentin J. J. Kreileder, Johannes Reisinger, Andreas FischerAgentic Retrieval-Augmented Generation Systems Chunk
Computer Science · Deggendorf Institute of Technology · Deggendorf, Germany

Sep 3, 2026, cs.AIJ/K move · Enter open · S save

STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation

Vineet Kumar, Meghanadh Pulivarthi, vishwajeet kumar, Jaydeep Sen+2

IBM

Precision Recall Document

Open paper AskPDF arXiv

Retrieval Augmented Generation (RAG) is a key component for generating accurate and hallucination free answers using Large Language Models (LLMs). LLMs are improving at handling long context, but still suffer from "lost in the middle" problem. Thus, precise and accurate retrieval is important. Current retrievers chunk long context into length-based manageable chunks - in the process throwing away rich and informative semantic global structure in the corpus. We introduce a novel retrieval system STAIR that empowers an LLM to exploit global structure in a corpus such as a Table of Contents (ToC) to efficiently store and retrieve information from its model parameters. Our thorough and careful ablation studies with a finetuned Differentiable Search Index (DSI) system show that ToC helps build a low hallucination (less than 0.05%) generative Information Retrieval (IR) system and can generalize to examples where very few training samples are available. To further research in this novel direction of ToC based retrieval we release SearchTome - a diverse benchmark created from 18 books across 6 diverse domains to further research in this novel direction. STAIR achieves a high Recall@1 score of 82.6% on SearchTome as compared to DSI (76.9%), where the difference is found to be statistically significant. STAIR easily beats other strong baselines such as BM25 (59.5%), DPR (68.7%) and out-of-the-box Mistral (13.8%).

Abstract

Explore similar work

STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation

SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG

Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts