DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

Atharva Thakur; Abulhair Saparov; Bruno Ribeiro

cs.CLOct 20, 2026

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

Authors: Atharva Thakur, Abulhair Saparov, Bruno Ribeiro

Organizations: Purdue University

Abstract

Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman $ρ= +0.60$ with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.

Figures & tables

Answerable. “I am taming a horse in the plains by feeding it golden carrots until heart particles appear, then I equip it with a saddle and armor. If I encounter a skeleton horse, what item will it drop when killed?”
Reference answer: bones, documented in $C_{\mathrm{pre}}$ .
Unanswerable. “I ride my horse across the plains, holding the jump control to charge a high leap over the fence gate blocking my path. If I equip copper horse armor, how many armor points will it provide?”
The answer (4 points) is documented in $C_{\mathrm{post}}$ ; the retrieved $C_{\mathrm{pre}}$ pages list leather, iron, gold and diamond horse armor.
Abstention Gemma 4 31B	“The provided context does not contain the answer. I would need to know if copper horse armor exists in the game, as the provided text only lists leather, iron, gold, and diamond variants.”
Premise denial Qwen3 32B (thinking)	“… Since ‘copper horse armor’ is not a recognized item in the game , it would provide 0 armor points . If you meant golden horse armor, it provides 7 armor points .”
Confabulation Llama 3.3 70B	“… Copper horse armor provides 8 armor points for your horse.”

Table 1: Two benchmark questions and real responses to the unanswerable one; bold highlights unsupported claims. Copper horse armor was documented on the wiki only from July 2025, after the corpus cut-off. The first response acknowledges missing knowledge; the second is a premise denial followed by a substituted item; the third confabulates a numerical answer. Only the first is a correct abstention. The first two use single-shot RAG; the third is without retrieval (with RAG, Llama 3.3 70B abstains).

Figure 1: Building DeclineCraft . A shared pipeline generates a player scenario from the pre-cut-off corpus $C_{\text{pre}}$ , samples one of the five highest-ranked eligible target chunks, selects a fact and appends a question requiring the selected fact. The target comes from $C_{\text{pre}}$ for answerable questions and the post-cut-off corpus $C_{\text{post}}$ for unanswerable questions. A separate verifier checks whether retrieved $C_{\text{pre}}$ passages support the answer. Both examples are real benchmark items.

Figure 2: Retrieval-depth ablation on 11 models with a

k{=}10

run.

k

is the number of chunks retrieved per query. Thinking is disabled where switchable; other models use their defaults. Gray lines: individual models; blue: mean Correct Abstention Rate with a 95% confidence interval over models. Appendix A.2 gives the comparisons and their limits.

Figure 3: Retrieval-pipeline ablation on 12 models, with single-shot RAG and five chunks. The six pipelines are keyword (BM25) or embedding (dense) search alone, the default hybrid of the two, and the hybrid with reranking, query rewriting or both (Appendix E ). Thinking is disabled where switchable; other models use their defaults. Gray lines: individual models; blue: mean Correct Abstention Rate with a 95% confidence interval over models. Appendix A.3 reports the pipeline comparisons.

Figure 4: Correct Abstention Rate with thinking disabled (gray dot) and enabled (arrowhead), holding weights fixed for the 13 models with switchable thinking. Orange: lower with thinking enabled; blue: higher; faded: not significant (

q\geq 0.05

Figure 5: Correct Abstention Rate versus total parameters (20 open-weight models, left) and release month (27 models, right; closed models in orange), with single-shot RAG. One point per model, with thinking enabled where switchable; neither Spearman correlation is statistically significant at the 5% level. Figure 12 in the appendix shows active parameters.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Attempt %	CAR %	Accuracy %	Accuracy when attempted %	Bal. F1
Gemma 4 26B-A4B R	95.4 [93.6, 96.7]	80.7 [75.8, 84.7]	69.3 [67.2, 71.4]	72.6	88.9
Gemma 4 26B-A4B R (thinking)	97.9 [96.6, 98.8]	73.7 [68.3, 78.6]	69.6 [67.5, 71.7]	73.6	87.4
Gemma 4 31B V	95.7 [93.9, 97.0]	83.3 [78.7, 87.1]	69.5 [67.5, 71.6]	72.6	90.1
Gemma 4 31B V (thinking)	96.4 [94.8, 97.6]	65.0 [59.4, 70.2]	72.0 [70.1, 74.0]	74.7	83.3
Gemma 3 4B	92.1 [89.9, 93.9]	56.7 [51.0, 62.2]	53.5 [51.0, 56.0]	58.1	78.3
Gemma 3 12B	97.9 [96.5, 98.7]	57.7 [52.0, 63.1]	62.5 [60.2, 64.7]	63.8	81.5

Appendix

Table 2: Results with single-shot RAG (hybrid retrieval,

k{=}5

): one row per model, with a second row, marked “(thinking)”, for models with switchable thinking. Attempt : Answerable Attempt Rate; CAR : Correct Abstention Rate; Accuracy when attempted : Answer Accuracy over the answerable questions the model attempted; Bal. F1 : balanced F1, defined in Appendix I . Brackets are 95% intervals (Wilson for rates, bootstrap over questions for accuracy). The best value in each column is bold. Superscripts mark models with a role in building or scoring the benchmark (G generator, V verifier and accuracy judge, R response judge).

	CAR %			Answerable Attempt %			Answer Accuracy %
Model	No RAG	RAG	Iter.	No RAG	RAG	Iter.	No RAG	RAG	Iter.
Gemma 4 26B-A4B R	15.3	80.7	67.7	88.9	95.4	96.4	26.9	69.3	69.7
Gemma 4 26B-A4B R (thinking)	10.2	73.7	71.2	90.8	97.9	98.2	26.6	69.6	70.6
Gemma 4 31B V	6.0	83.3	76.0	95.0	95.7	97.3	33.4	69.5	71.2
Gemma 4 31B V (thinking)	6.7	65.0	60.7	93.2	96.4	97.7	34.0	72.0	74.0
Gemma 3 4B	25.7	56.7	38.7	90.3	92.1	92.9	14.7	53.5	53.2

Appendix

Table 3: Each model and thinking setting under the three retrieval conditions: Correct Abstention Rate, Answerable Attempt Rate and Answer Accuracy without retrieval, with single-shot RAG and with iterative RAG. Rows and marks as in Table 2 . The best value in each column is bold.

Figure 8: Correct Abstention Rate (left) and Answer Accuracy (right) without retrieval, with single-shot RAG and with iterative RAG: one gray line per configuration, mean in blue. Single-shot RAG raises CAR significantly for 38 of 40 configurations and lowers it for 0; iterative RAG lowers it for 30 and raises it for 0 (

q<0.05

). Accuracy rises with retrieval and barely changes with further rounds.

Figure 9: Answer Accuracy with thinking disabled (gray dot) and enabled (arrowhead) on the same weights, without retrieval (left) and with single-shot RAG (right), for the models of Figure 4 . Orange: lower with thinking enabled; blue: higher. Each panel spans 50% points.

Figure 10: Answerable Attempt Rate against Correct Abstention Rate for every configuration, single-shot RAG. Colors mark model families; squares are runs with thinking enabled, circles runs with thinking disabled or from instruct-only models. Bars are 95% Wilson intervals. The inset magnifies the crowded high-coverage region (without error bars); labels mark each family’s highest Correct Abstention Rate and every configuration that no other configuration exceeds on both rates. Attempt rates span 78.7–99.9%; CAR spans 13.0–87.7%. The best performance is the top-right corner.

Figure 11: Answerable Attempt Rate against Answer Accuracy (mean of the three accuracy judges), single-shot RAG. Markers and inset as in Figure 10 . Bars are 95% intervals: Wilson for the attempt rate, bootstrap over questions for accuracy.

Figure 12: Correct Abstention Rate versus active parameters across the 20 open-weight models, with single-shot RAG; one point per model as in Figure 5 . The Spearman correlation is not significant at the 5% level.

Figure 13: Answer Accuracy versus total parameters across the 20 open-weight models, with single-shot RAG; one point per model as in Figure 5 . Unlike Correct Abstention Rate, Answer Accuracy rises with size (Spearman

\rho=+0.60

p=0.005

Figure 14: Correct Abstention Rate per model without retrieval (No RAG), with single-shot RAG and with iterative RAG, thinking enabled or instruct-only. Across all configurations, single-shot RAG raises CAR for 38 of 40 and lowers it for 0; iterative RAG lowers it for 30 of 40 relative to single-shot RAG (

q<0.05

), by -10.0% points on average.

Figure 15: Answer Accuracy per model under the three retrieval conditions, thinking enabled or instruct-only. Mean accuracy is 24.3% without retrieval, 68.5% with single-shot RAG and 69.3% with iterative RAG.

Figure 16: As Figure 10 , with iterative RAG (up to two further retrieval rounds).

Figure 17: What each model family does with unanswerable questions, by retrieval condition: abstention, premise denial or attempted answer. Family means over configurations.

Figure 18: What each model does with the unanswerable questions under single-shot RAG: abstention, premise denial or attempted answer. Families together; within a family, smallest model first.

Figure 19: Leakage audit per model: Answer Accuracy on the unanswerable questions without retrieval (mean of the three accuracy judges against the oracle model’s reference answers from

C_{\text{post}}

, an abstention scoring 0), with 95% bootstrap intervals over questions, next to each model’s attempt rate. The dashed line is GPT-OSS 120B , whose published knowledge cut-off (June 2024) predates every post-cut-off page.

Family	Runs	Attempted %	Correct %	CI low	CI high	Accuracy when attempted %
GPT-OSS	1	97.7	19.8	16.6	23.0	20.3
GPT-4.1	2	94.5	15.5	13.0	17.9	16.4
GPT-5	2	57.5	15.4	12.7	18.3	26.7
Command A	1	98.0	13.7	11.0	16.5	14.0
GLM-4.5	2	87.2	12.9	10.6	15.1	14.8
Llama 3.3	1	82.3	12.3	9.6	15.2	14.9

Appendix

Table 4: Leakage audit by model family: unanswerable questions answered correctly without retrieval, with 95% bootstrap intervals over questions.

Figure 20: Run-level Answer Accuracy under each pair of accuracy judges (Spearman

\rho

above each panel). The judges rank runs almost identically, but their mean Answer Accuracy differs by up to 12.1% points.

Judges	Rows	Exact agreement %	Weighted $\kappa$	Mean diff (pts)
Gemma 4 31B (8-bit) vs Qwen3.8 27B (8-bit)	252,820	68.5	0.730	+10.9
Gemma 4 31B (8-bit) vs Laguna S 2.1 (8-bit)	252,820	85.2	0.888	-1.9
Qwen3.8 27B (8-bit) vs Laguna S 2.1 (8-bit)	252,820	69.0	0.728	-12.8

Appendix

Table 5: Agreement between the three accuracy judges on the same answers (quadratic-weighted Cohen’s

\kappa

; Fleiss’

\kappa

across all three is 0.61).

Figure 21: Choice of

k

(hybrid retrieval, single-shot RAG). Top: retrieval alone, the share of answerable questions whose target chunk is retrieved and of unanswerable questions for which a chunk the scenario was written from is retrieved. Middle and bottom: Correct Abstention Rate and Answer Accuracy for 8 models run at every

k

(gray; thinking disabled where switchable) and their mean (blue). From

k{=}5

k{=}10

, CAR falls from 52.4% to 49.9% while accuracy moves from 70.4% to 72.2%.

Model	k	CAR %	Accuracy %	Bal. F1
Command A 111B	1	78.0	53.8	82.2
Command A 111B	2	70.7	63.1	84.0
Command A 111B	3	72.7	66.9	86.3
Command A 111B	5	71.7	69.3	86.6
Command A 111B	10	70.7	68.8	86.1
GLM-4.5-Air	1	58.7	53.5	78.1

Appendix

Table 6:

k

sweep per model (hybrid retrieval, single-shot RAG; the models of Figure 21 ). Bal. F1 : balanced F1 ( Appendix I ). Bold: the best

k

for each model in each column. The

k{=}5

rows are the hybrid runs of Table 7 , not the runs in Table 2 .

Figure 22: Retrieval ablation: Correct Abstention Rate (left) and Answer Accuracy (right) for 12 models under six retrieval pipelines (single-shot RAG,

k{=}5

; thinking disabled where switchable, OLMo 3 32B Think and closed models at their defaults). Gray: individual models; blue: their mean. Mean Correct Abstention Rate is 54.7% with the default hybrid retriever, 52.5% with reranking and 49.8% with query rewriting and reranking; 3 of 60 model

\times

pipeline comparisons with hybrid are significant.

Model	Thinking	Retrieval	CAR %	Accuracy %	Bal. F1
Command A 111B	no	Hybrid (default)	71.7	69.3	86.6
Command A 111B	no	Dense	68.3	69.5	85.7
Command A 111B	no	BM25	77.3	64.8	87.0
Command A 111B	no	Hybrid + rerank	67.3	70.7	85.2
Command A 111B	no	Hybrid + rewrite	71.7	68.5	86.2
Command A 111B	no	Hybrid + rewrite + rerank	67.0	71.2	84.9

Appendix

Table 7: Retrieval ablation per model (single-shot RAG,

k{=}5

). Thinking : whether thinking is enabled. Bal. F1 : balanced F1 ( Appendix I ). Bold: the best pipeline for each model in each column. The Hybrid (default) rows are a second run of the configurations in Table 2 , with a different set of responses.

Figure 23: The scenario makes abstention harder. Correct Abstention Rate and Answerable Attempt Rate for 10 models (single-shot RAG,

k{=}5

) with the scenario (blue: scenario and question for retrieval and the model; the default) and without it (orange: the question alone for both). Points: mean with a 95% confidence interval over models; gray lines: one model. Each panel has its own scale.

Figure 24: Share of unanswerable questions whose five retrieved chunks include a

C_{\text{pre}}

passage the scenario was written from, by retrieval pipeline (retrieval only).

Figure 25: Matched pairs with thinking disabled and enabled on the unanswerable questions: the share of questions whose response changes from abstention to premise denial when thinking is enabled (orange) and from premise denial to abstention (blue), per model. Faded: the two counts do not differ significantly; *: they do (

q<0.05

). Models sorted by the shift without retrieval.

Model	$n$	A $\to$ D / D $\to$ A	A $\to$ X / X $\to$ A	$n$	A $\to$ D / D $\to$ A	A $\to$ X / X $\to$ A
	No RAG			Single-shot RAG
Gemma 4 26B-A4B	254	27 / 7 *	4 / 8	278	25 / 2 *	7 / 8
Gemma 4 31B	299	10 / 2	3 / 13 *	300	58 / 3 *	11 / 11
GLM-4.5-Air	282	12 / 0 *	33 / 6 *	299	23 / 10	28 / 26
Qwen3 4B	272	30 / 12 *	34 / 11 *	291	14 / 8	20 / 58 *
Qwen3 8B	284	22 / 4 *	24 / 38	298	2 / 21 *	4 / 69 *

Appendix

Table 8: Matched pairs with thinking disabled and enabled, per model:

n

questions (budget-exhausted pairs left out) and the discordant counts abstain

\to

deny / deny

\to

abstain and abstain

\to

answer / answer

\to

abstain. Bold with *: the two counts differ significantly (

q<0.05

Figure 26: Correct Abstention Rate versus Answer Accuracy under single-shot RAG, one point per model and thinking setting. Colors: families; squares: thinking enabled. Bars: 95% intervals, bootstrap over questions for accuracy and Wilson for Correct Abstention Rate. Appendix A.6 reports the correlations.

Figure 27: Accuracy when attempted (over the answerable questions a model attempted, abstentions left out) against Correct Abstention Rate, single-shot RAG. Spearman

\rho=-0.07

Figure 28: Correct Abstention Rate versus Answerable Attempt Rate without retrieval (gray) and with single-shot RAG (orange), one point per model and thinking setting (squares: thinking enabled). Appendix A.7 discusses the observed trade-off frontiers.

Generator	Accepted	Lost to	Fact in	Style	Hit@5	CAR
	(ans. / unans.)	format	$C_{\text{pre}}$ (%)	AUC	(%)	(%)
Qwen3.5 122B-A10B	100 / 82	35	18.0	0.52	80.0	84.1
Qwen3 235B-A22B	100 / 78	194	17.3	0.65	91.0	82.1
Gemma 4 31B	100 / 87	33	20.2	0.57	87.0	85.1

Appendix

Table 9: Generator comparison: the same pipeline, seed and verifier ( Laguna S 2.1 ) with three generators, 100 answerable and 100 unanswerable targets each. Accepted : questions that passed every check. Lost to format : drafts with no parsable fact list or question. Fact in

C_{\text{pre}}

: unanswerable drafts rejected because

C_{\text{pre}}

already states the fact. Style AUC : how well a classifier on wording features separates answerable from unanswerable questions (0.5 is chance). Hit@5 : answerable questions whose source chunk the default retriever returns in its top five. CAR : Correct Abstention Rate of Gemma 4 31B (8-bit, thinking disabled) with single-shot RAG, labeled by the same response judge as the main results.

Part 1 with abstentions and premise denials merged
	Part 1: response	Part 2: accuracy
	$\kappa$	weighted $\kappa$
Human–human	0.57 [0.45, 0.68]	0.42 [0.21, 0.62]
Judge–human	0.74 [0.66, 0.82]	0.66 [0.50, 0.79]
Judge–majority	0.93 [0.85, 1.00]	–
Human–human	0.67 [0.51, 0.80]

Appendix

Table 10: Agreement on the shared annotation set (70 response labels, 30 accuracy scores, 3 annotators): Cohen’s

\kappa

with 95% bootstrap intervals over items. Human–human : mean over annotator pairs. Judge–human : mean over annotators of each one’s agreement with the response judge (Part 1) or the mean of the accuracy judges (Part 2). Judge–majority : against the label at least two annotators gave.

Family	Model	Switchable	Precision	Released	Knowledge
		thinking			cut-off
Qwen3 ( Yang et al., 2025 )	4B, 8B, 14B, 32B, 30B-A3B	yes	8-bit	Apr 2025 †	not published
	235B-A22B	yes	4-bit	Apr 2025 †	not published
Qwen3-Next ( Alibaba Cloud Community, 2025 )	80B-A3B (instruct)	no	8-bit	Sep 2025 †	not published
Qwen3.5 ( Qwen Team, 2026a )	27B, 35B-A3B, 122B-A10B	yes	8-bit	Feb 2026 †	not published
Gemma 3 ( Gemma Team, 2025 ; Google DeepMind, 2025b )	4B, 12B, 27B	no	8-bit	Mar 2025	Aug 2024

Appendix

Table 11: The 19 open-weight models served locally and the 8 models run through OpenRouter (7 closed, and GPT-OSS 120B ). Switchable thinking : yes if the model was run with thinking both enabled and disabled; models marked no were run in one setting, with thinking always on ( OLMo 3 32B Think , GPT-5 , GPT-OSS 120B ), at its low thinking level ( Gemini 3 Flash ) or without thinking. Precision : the weights used in every analysis, 8-bit (FP8) or, for Qwen3 235B-A22B , 4-bit (INT4), the highest available; API for models run through OpenRouter. Parameter counts not given by the name: GLM-4.5-Air 106B total and 12B active, Gemma 4 26B-A4B 25.2B and 3.8B, GPT-OSS 120B 117B and 5.1B. Release dates and knowledge cut-offs are taken from the developers’ reports, model cards, announcements or API documentation (for closed models, the date of the snapshot called); “not published” means none states a knowledge cut-off. † Released after the 31 March 2025 corpus cut-off with no published knowledge cut-off: these models could have seen post-cut-off pages, which the leakage audit ( Table 4 ) tests. Gemma 4 and Gemini 3 Flash were also released later, but their published knowledge cut-offs (January 2025) precede our corpus cut-off. The served name and pinned revision of every build are in artifacts/v13/served_models.csv .

Role	Model
Embedding (dense retrieval)	Qwen3-Embedding-8B, Q8_0 ( Zhang et al., 2025 )
Reranking (ablation; verifier retrieval)	llama-nemotron-rerank-1b-v2 ( NVIDIA, 2025 )
Question generator	Qwen3.5 122B-A10B, 8-bit
Question verifier	Gemma 4 31B, 8-bit
Response judge (abstain / deny / answer)	Gemma 4 26B-A4B, 8-bit
Accuracy and leakage judges	Gemma 4 31B, Qwen3.8 27B ( Qwen Team, 2026b ) , Laguna S 2.1 ( Poolside Team, 2026 ) (8-bit)

Appendix

Table 12: Models used to build and score the benchmark. Three of them are also evaluated ( Qwen3.5 122B-A10B , Gemma 4 31B and Gemma 4 26B-A4B ); results mark them where they matter.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 355 papers

Feb 26, 2026

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark

Sara Rosenthal, Yannis Katsis, Vraj Shah +3

IBM Research, USA

Retrieval-Augmented Generation RAG Evaluation Multi-Turn Dialogue Evaluation Question Answering

Ask PDF

Mar 6, 2026

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.

Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2

neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

RAG Evaluation LLM Evaluation LLM Abstention

Ask PDF

Mar 8, 2026

KohakuRAG: A simple RAG framework with hierarchical document indexing

Retrieval-augmented generation (RAG) systems that answer questions from document collections face compounding difficulties when high-precision citations are required: flat chunking strategies sacrifice document structure, single-query formulations miss relevant passages through vocabulary mismatch, and single-pass inference produces stochastic answers that vary in both content and citation selection. We present KohakuRAG, a hierarchical RAG framework that preserves document structure through a four-level tree representation (document

\rightarrow

section

\rightarrow

paragraph

\rightarrow

sentence) with bottom-up embedding aggregation, improves retrieval coverage through an LLM-powered query planner with cross-query reranking, and stabilizes answers through ensemble inference with abstention-aware voting. We evaluate on the WattBot 2025 Challenge, a benchmark requiring systems to answer technical questions from 32 documents with

\pm

0.1% numeric tolerance and exact source attribution. KohakuRAG achieves first place on both public and private leaderboards (final score 0.861), as the only team to maintain the top position across both evaluation partitions. Ablation studies reveal that prompt ordering (+80% relative), retry mechanisms (+69%), and ensemble voting with blank filtering (+1.2pp) each contribute substantially, while hierarchical dense retrieval alone matches hybrid sparse-dense approaches (BM25 adds only +3.1pp). We release KohakuRAG as open-source software at https://github.com/KohakuBlueleaf/KohakuRAG.

Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang +1

National Tsing Hua University · Comfy Org Research · Kohaku-Lab

Retrieval-Augmented Generation Structured RAG Multi-Document QA Knowledge-Intensive QA

Ask PDF

Abstract

RAG Evaluation

Explore similar work

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

KohakuRAG: A simple RAG framework with hierarchical document indexing