Parser, Chunking, and Embedding Interactions in Retrieval-Augmented Generation over Indian Government Regulatory Documents

Shubham Kumar Singh

cs.CLSep 15, 2026

Parser, Chunking, and Embedding Interactions in Retrieval-Augmented Generation over Indian Government Regulatory Documents

Authors: Shubham Kumar Singh

Abstract

Retrieval-augmented generation (RAG) pipelines are typically assembled from independently-chosen components -- a document parser, a chunking strategy, and an embedding model -- yet these choices are rarely evaluated jointly, and evaluations that do combine them are usually run on a single document or corpus. We present a controlled factorial study of 3 parsers, 3 chunking strategies, and 5 dense embedding models, together with a sparse BM25 baseline, evaluated against 800 question instances, each with one or more required evidence strings, with evidence strings automatically validated against source text and a 10% random sample manually reviewed, across four structurally distinct Indian central-government regulatory documents. We fit linear mixed-effects models with document-query-level random intercepts to the resulting 72,000-row result set, run Holm-corrected paired comparisons between matched dense and sparse configurations, and report clustered bootstrap confidence intervals for all 54 unique retriever configurations. We find that no single retriever family dominates across documents; parser and chunker choice interact significantly; MPNet-base is a consistent underperformer with a severe failure mode on table-derived questions; and the corpus exhibits a near-saturated evidence-preservation ceiling above 98%, indicating that retrieval differences are driven primarily by ranking quality rather than information loss during ingestion. We additionally report embedding-dimension and chunk-size/overlap ablations and an efficiency/quality Pareto analysis. We release our full evaluation harness, corpus manifest, and 800-question benchmark.

Figures & tables

ID	Document	Issuing authority	Dominant structural character
D1	Central Civil Services (Leave) Rules, 1972	Dept. of Personnel & Training	Numerical entitlement tables, accrual/encashment schedules
D2	CCS (Conduct) Rules, 1964	Dept. of Personnel & Training	Prescriptive prohibitions, exceptions, disciplinary language
D3	Fundamental Rules & Supplementary Rules, Part I	Dept. of Personnel & Training	Large, heterogeneous, multi-chapter rule hierarchy
D4	Right to Information Act, 2005	Parliament of India	Long-form legal narrative, definitions, cross-referenced sections

Table 1: Corpus documents. Full source URLs and retrieval dates are recorded in the released corpus manifest with SHA-256 hashes of the exact PDF bytes used.

Parser	Chunker	Embedding	Recall@5	Recall@10	MRR
normal	advanced	BGE-small	0.909	0.958	0.767
normal	advanced	E5-base	0.904	0.949	0.771
docling	advanced	E5-base	0.910	0.947	0.787
normal	advanced	BGE-base	0.895	0.945	0.772
normal	basic	E5-base	0.872	0.937	0.728
BM25-Naive (sparse baseline)

Table 2: Top-5 dense configurations and top-3 BM25 configurations by Evidence Recall@10, averaged over all four documents.

Figure 1: Mean dense Evidence Recall@5 by parser and chunker, averaged over all five embedding models and four documents.

Term	Coef.	SE	$z$	$p$
Intercept	0.905	0.010	88.86	$<0.0001$
Parser[normal]	0.039	0.012	3.17	0.0015
Chunker[medium]	0.028	0.012	2.25	0.0247
Embedding[MPNet-base]	$-0.049$	0.012	$-3.93$	$<0.0001$
Chunker[medium]:Embedding[MiniLM-L6]	$-0.035$	0.018	$-1.97$	0.0493
Parser[docling]:Chunker[medium]	$-0.047$	0.018	$-2.69$	0.0073

Table 3: Significant (

p<0.05

) fixed effects, dense

3\times 3\times 5

mixed-effects model on Evidence Recall@10. Reference level: context_aware / basic / BGE-base . Full table in the released results.

Embedding	Mean diff. vs. BM25	% cells won	% significant wins (Holm)
E5-base	$+0.036$	86.1%	2.8%
BGE-base	$+0.029$	80.6%	2.8%
BGE-small	$+0.024$	72.2%	0.0%
MiniLM-L6	$+0.003$	50.0%	0.0%
MPNet-base	$-0.034$	27.8%	0.0%

Table 4: Dense vs. matched BM25-Naive on Evidence Recall@10, averaged over all (document, parser, chunker) cells. “% significant wins” is the fraction of individual cells where dense significantly beats BM25 after Holm correction across the full comparison family; the low rate reflects the conservativeness of Holm correction at

n{=}180

comparisons (

4

documents

\times

3

parsers

\times

3

chunkers

\times

5

embeddings) with modest per-cell query counts, not an absence of a real aggregate effect.

Figure 2: Evidence Recall@5, dense (best-of-5, mean) vs. BM25-Naive, by document.

Figure 3: Evidence Recall@5 by question type and retriever.

Parser	Chunker	Mean coverage	% queries fully covered
normal	advanced	0.998	99.6%
docling	advanced	0.995	99.1%
normal	basic	0.991	98.9%
context_aware	basic	0.988	98.5%
context_aware	medium	0.988	98.5%
docling	medium	0.987	98.3%

Table 5: Corpus-level evidence preservation (recall ceiling), averaged across all four documents.

Figure 4: Chunk size / overlap ablation, Evidence Recall@5.

Figure 5: Embedding dimension vs. Evidence Recall@5, MRR, and median query latency, for the Matryoshka-truncatable nomic-embed-text-v1.5 model under the pre-declared normal / advanced parser/chunker selection.

Figure 6: Quality/latency Pareto frontier across all 54 configurations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Dense embedding models compared directly on Evidence Recall@5, averaged over all parsers, chunkers, and documents. Complements Table 4 .

Figure 8: Aggregate Evidence Recall@5, dense (best-of-5) vs. BM25-Naive, pooled across all four documents. See Figure 2 for the per-document breakdown that motivates RQ2.

Figure 9: Distribution of query-level Evidence Recall@5 by document and retriever family. The wide spread on D3 relative to D1/D2/D4 illustrates why document-level means (Figure 2 ) can mask substantial per-query variance within a single document.

Figure 10: Top-20 unique (parser, chunker, embedding) configurations ranked by Evidence Recall@10, with 95% clustered bootstrap confidence intervals (2,000 resamples over query-level means). Overlapping intervals among the top configurations indicate that several parser/chunker/embedding combinations are statistically indistinguishable at the top of the leaderboard.

Figure 11: Mean Evidence Recall@5 by parser and retriever, averaged over all three chunkers. MPNet-base’s underperformance (§ 5.3 ) is consistent across all three parsers rather than isolated to one.

Figure 12: Evidence Recall@5 as a joint function of parser and chunker, faceted by embedding model. A three-dimensional view of the interaction quantified in Table 3 .

Figure 13: Evidence Recall@5 vs. corpus-level evidence preservation (the representation ceiling of Table 5 ) for every parser $\times$ chunker combination. All points sit just below the retrieved-equals-preserved diagonal and close to the $x=1$ boundary, visually confirming RQ3: retrieval quality is bottlenecked by ranking, not by information loss during ingestion.

Figure 14: Per-configuration mean query latency vs. Evidence Recall@5, colored by retriever. An alternative view of the latency/quality relationship underlying the Pareto frontier in Figure 6 .

Figure 15: Design decision map: Evidence Recall@5 vs. a total-cost proxy (log-scale latency) for all 54 configurations, with the top five dense configurations labeled. BM25-Naive configurations cluster at the low-cost end; the labeled dense configurations mark the highest-quality region of the trade-off space.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 353 papers

Retrieval Augmented Generation Compositionality
Oct 21, 2026Ido Levy, Segev Shlomov, Leshem ChoshenRetrieval-Augmented Generation RAG Evaluation
IBM · Weizmann Institute of Science
Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts
Jul 2, 2026Valentin J. J. Kreileder, Johannes Reisinger, Andreas FischerDocument Chunking Retrieval-Augmented Generation
Computer Science · Deggendorf Institute of Technology · Deggendorf, Germany
Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.

Oct 21, 2026J/K move · Enter open · S save

Retrieval Augmented Generation Compositionality

Ido Levy, Segev Shlomov, Leshem Choshen

IBM · Weizmann Institute of Science

Retrieval-Augmented Generation RAG Evaluation

Open paper AskPDF arXiv

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.

Abstract

RAG Evaluation

Explore similar work

Retrieval Augmented Generation Compositionality

Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

Revisiting RAG Retrievers: An Information Theoretic Benchmark