cs.CLSep 15, 2026

Parser, Chunking, and Embedding Interactions in Retrieval-Augmented Generation over Indian Government Regulatory Documents

Authors: Shubham Kumar Singh

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Retrieval-augmented generation (RAG) pipelines are typically assembled from independently-chosen components -- a document parser, a chunking strategy, and an embedding model -- yet these choices are rarely evaluated jointly, and evaluations that do combine them are usually run on a single document or corpus. We present a controlled factorial study of 3 parsers, 3 chunking strategies, and 5 dense embedding models, together with a sparse BM25 baseline, evaluated against 800 question instances, each with one or more required evidence strings, with evidence strings automatically validated against source text and a 10% random sample manually reviewed, across four structurally distinct Indian central-government regulatory documents. We fit linear mixed-effects models with document-query-level random intercepts to the resulting 72,000-row result set, run Holm-corrected paired comparisons between matched dense and sparse configurations, and report clustered bootstrap confidence intervals for all 54 unique retriever configurations. We find that no single retriever family dominates across documents; parser and chunker choice interact significantly; MPNet-base is a consistent underperformer with a severe failure mode on table-derived questions; and the corpus exhibits a near-saturated evidence-preservation ceiling above 98%, indicating that retrieval differences are driven primarily by ranking quality rather than information loss during ingestion. We additionally report embedding-dimension and chunk-size/overlap ablations and an efficiency/quality Pareto analysis. We release our full evaluation harness, corpus manifest, and 800-question benchmark.

Figures & tables

IDDocumentIssuing authorityDominant structural character
D1Central Civil Services (Leave) Rules, 1972Dept. of Personnel & TrainingNumerical entitlement tables, accrual/encashment schedules
D2CCS (Conduct) Rules, 1964Dept. of Personnel & TrainingPrescriptive prohibitions, exceptions, disciplinary language
D3Fundamental Rules & Supplementary Rules, Part IDept. of Personnel & TrainingLarge, heterogeneous, multi-chapter rule hierarchy
D4Right to Information Act, 2005Parliament of IndiaLong-form legal narrative, definitions, cross-referenced sections
Table 1: Corpus documents. Full source URLs and retrieval dates are recorded in the released corpus manifest with SHA-256 hashes of the exact PDF bytes used.
ParserChunkerEmbeddingRecall@5Recall@10MRR
normaladvancedBGE-small0.9090.9580.767
normaladvancedE5-base0.9040.9490.771
doclingadvancedE5-base0.9100.9470.787
normaladvancedBGE-base0.8950.9450.772
normalbasicE5-base0.8720.9370.728
BM25-Naive (sparse baseline)
Table 2: Top-5 dense configurations and top-3 BM25 configurations by Evidence Recall@10, averaged over all four documents.
Figure 1: Mean dense Evidence Recall@5 by parser and chunker, averaged over all five embedding models and four documents.
TermCoef.SEzzpp
Intercept0.9050.01088.86<0.0001<0.0001
Parser[normal]0.0390.0123.170.0015
Chunker[medium]0.0280.0122.250.0247
Embedding[MPNet-base]−0.049-0.0490.012−3.93-3.93<0.0001<0.0001
Chunker[medium]:Embedding[MiniLM-L6]−0.035-0.0350.018−1.97-1.970.0493
Parser[docling]:Chunker[medium]−0.047-0.0470.018−2.69-2.690.0073
Table 3: Significant ( p<0.05p<0.05 ) fixed effects, dense 3×3×53\times 3\times 5 mixed-effects model on Evidence Recall@10. Reference level: context_aware / basic / BGE-base . Full table in the released results.
EmbeddingMean diff. vs. BM25% cells won% significant wins (Holm)
E5-base+0.036+0.03686.1%2.8%
BGE-base+0.029+0.02980.6%2.8%
BGE-small+0.024+0.02472.2%0.0%
MiniLM-L6+0.003+0.00350.0%0.0%
MPNet-base−0.034-0.03427.8%0.0%
Table 4: Dense vs. matched BM25-Naive on Evidence Recall@10, averaged over all (document, parser, chunker) cells. “% significant wins” is the fraction of individual cells where dense significantly beats BM25 after Holm correction across the full comparison family; the low rate reflects the conservativeness of Holm correction at n=180n{=}180 comparisons ( 44 documents ×\times 33 parsers ×\times 33 chunkers ×\times 55 embeddings) with modest per-cell query counts, not an absence of a real aggregate effect.
Figure 2: Evidence Recall@5, dense (best-of-5, mean) vs. BM25-Naive, by document.
Figure 3: Evidence Recall@5 by question type and retriever.
ParserChunkerMean coverage% queries fully covered
normaladvanced0.99899.6%
doclingadvanced0.99599.1%
normalbasic0.99198.9%
context_awarebasic0.98898.5%
context_awaremedium0.98898.5%
doclingmedium0.98798.3%
Table 5: Corpus-level evidence preservation (recall ceiling), averaged across all four documents.
Figure 4: Chunk size / overlap ablation, Evidence Recall@5.
Figure 5: Embedding dimension vs. Evidence Recall@5, MRR, and median query latency, for the Matryoshka-truncatable nomic-embed-text-v1.5 model under the pre-declared normal / advanced parser/chunker selection.
Figure 6: Quality/latency Pareto frontier across all 54 configurations.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Dense embedding models compared directly on Evidence Recall@5, averaged over all parsers, chunkers, and documents. Complements Table 4 .
Figure 8: Aggregate Evidence Recall@5, dense (best-of-5) vs. BM25-Naive, pooled across all four documents. See Figure 2 for the per-document breakdown that motivates RQ2.
Figure 9: Distribution of query-level Evidence Recall@5 by document and retriever family. The wide spread on D3 relative to D1/D2/D4 illustrates why document-level means (Figure 2 ) can mask substantial per-query variance within a single document.
Figure 10: Top-20 unique (parser, chunker, embedding) configurations ranked by Evidence Recall@10, with 95% clustered bootstrap confidence intervals (2,000 resamples over query-level means). Overlapping intervals among the top configurations indicate that several parser/chunker/embedding combinations are statistically indistinguishable at the top of the leaderboard.
Figure 11: Mean Evidence Recall@5 by parser and retriever, averaged over all three chunkers. MPNet-base’s underperformance (§ 5.3 ) is consistent across all three parsers rather than isolated to one.
Figure 12: Evidence Recall@5 as a joint function of parser and chunker, faceted by embedding model. A three-dimensional view of the interaction quantified in Table 3 .
Figure 13: Evidence Recall@5 vs. corpus-level evidence preservation (the representation ceiling of Table 5 ) for every parser ×\times chunker combination. All points sit just below the retrieved-equals-preserved diagonal and close to the x=1x=1 boundary, visually confirming RQ3: retrieval quality is bottlenecked by ranking, not by information loss during ingestion.
Figure 14: Per-configuration mean query latency vs. Evidence Recall@5, colored by retriever. An alternative view of the latency/quality relationship underlying the Pareto frontier in Figure 6 .
Figure 15: Design decision map: Evidence Recall@5 vs. a total-cost proxy (log-scale latency) for all 54 configurations, with the top five dense configurations labeled. BM25-Naive configurations cluster at the low-cost end; the labeled dense configurations mark the highest-quality region of the trade-off space.
More in this topic

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 353 papers

Explore similar work

CardsList
Oct 21, 2026

Retrieval Augmented Generation Compositionality

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.

Ido Levy, Segev Shlomov, Leshem Choshen

IBM · Weizmann Institute of Science

Retrieval-Augmented GenerationRAG Evaluation
AskPDF
Jul 2, 2026

Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside their parameters. We evaluate if cluster-based semantic chunking improves retrieval and answer quality compared to fixed-size and recursive chunking evaluating on long, structured academic theses using the Retrieval Augmented Generation Assessment (RAGAs) framework. RAGAs based faithfulness shows limited reliability in this setup. Performance on fixed versus document specific questions varied substantially, likely related to the formatting of documents and preprocessing. Under the tested configuration, cluster-based chunking did not outperform simpler strategies.

Valentin J. J. Kreileder, Johannes Reisinger, Andreas Fischer

Computer Science · Deggendorf Institute of Technology · Deggendorf, Germany

Document ChunkingRetrieval-Augmented GenerationDocument QARAG Evaluation
AskPDF
Feb 25, 2026

Revisiting RAG Retrievers: An Information Theoretic Benchmark

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.

Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5

CapitalOne.

RAG EvaluationIR Evaluation
AskPDF