Sieve and Sage: Efficient Distraction Filtering for Reliable RALM Abstention

Jongbin Won; Sung Geun An; Jay-yoon Lee

cs.CLSep 17, 2026

Sieve and Sage: Efficient Distraction Filtering for Reliable RALM Abstention

Authors: Jongbin Won, Sung Geun An, Jay-yoon Lee

Organizations: Graduate School of Data Science / Seoul National University

Abstract

Just as Socrates recognized the limits of his own knowledge, Retrieval-Augmented Language Models (RALMs) should learn to abstain when the retrieved evidence cannot support a reliable response. Existing approaches largely rely on monolithic LLMs to handle heterogeneous retrieval failures in a single step, resulting in limited abstention performance and high computational costs. We instead decompose retrieval failures into two distinct states: (i) the unanswerable state, where the required evidence is absent, and (ii) the distracted state, where relevant evidence is mixed with conflicting, negated, or adversarial information. Based on this decomposition, we introduce a lightweight module (Sieve) that screens retrieved document sets for distracting evidence before invoking a costly LLM (Sage) for grounded generation and abstention. Evaluated across both general and high-stakes expert domains, our Sieve and Sage framework preemptively detects distracting noise, improving system accuracy by up to 69.4 percentage points and Macro-F1 by 55.2 percentage points compared to one-stage baselines. Furthermore, it achieves up to a 1.99x speedup, establishing a highly efficient and reliable abstention pipeline for RALM with abstention.

Figures & tables

Figure 1: Comparison between conventional LLM-centric approaches and Sieve and Sage . While previous robustness and abstention studies rely solely on the internal judgment of a monolithic LLM, our framework introduces a decoupled architecture: a lightweight Sieve for preemptive distraction filtering and a generative Sage for high-fidelity generation and abstention for unanswerable instances. Unlike (c) conventional abstainable RALMs that produce a state-agnostic abstention signal, (d) our Sieve and Sage framework identifies the underlying retrieval state, enabling state-aware post-abstention strategies such as re-retrieval for unanswerable cases and distraction filtering for distracted cases.

Figure 2: Examples of retrieved and simulated documents . To evaluate the abstention under diverse retrieval noise, we simulated various distracting documents by perturbing answerable documents via entity/predicate manipulation, NLI labeling, and LLM generation.

Panel of Figure 3 — Figure 3: Pareto graph of inference time versus System Accuracy/Macro F1 . Our framework( $\star$ ) is up to 2 $\times$ faster than other LLM-centric approaches with comparable performance, especially achieving the highest F1 scores in expert domain datasets.

	General domain					Expert domain
test(train) datasets	NQ-wiki	NQ-google	WebQ	RealtimeQA	RAG-Bench	MedQA	BioASQ	AstroQA
document source	Wikipedia	Google search	Wikipedia	Google search	Wikipedia	PubMed abstracts	PubMed abstracts	Astro abstracts
document length	165.93	152.29	172.63	145.27	409.29	1024.47	955.77	1046.2
# total samples	3297 (44030)	226 (2182)	2096	187	4052	491 (2087)	610 (3997)	289 (1489)
# answerable	889 (15086) *	51 (418) *	545 *	34 *	912 *	116 (409)	75 (536) *	99 (460)
# unanswerable	483 (20688) *	49 (382) *	296 *	66 *	419 *	159 (539)	397 (3236) *	86 (409)

Table 1: Statistics of test (and train) datasets. An asterisk (*) indicates that the documents are adopted from prior studies rather than retrieved or simulated in this work.

Panel of Figure 4 — Figure 4: Selective Prediction and System Reliability . We evaluate system reliability through two complementary analyses: (a) Coverage and Selective Accuracy for answerable examples (precision on $S_{\text{ans}}$ ) and (b) distribution of model response across all retrieval states. Although Sieve and Sage operates at lower coverage than more aggressive baselines, this reflects a more conservative response policy and is accompanied by fewer incorrect responses on several datasets.

Panel of Figure 5 — Figure 5: t-SNE comparison of Sieve training objectives . Our model (c), fine-tuned to classify $S_{\text{conflict}}$ from $S_{\text{ans}}$ , achieves a more distinct separation of distracting noise compared to the model (d) fine-tuned to classify answerable, unanswerable, and conflicting states at once. This validates that a specialized focus on factual consistency establishes a precise decision boundary for identifying retrieval distractions.

Dataset	Strategy	Accuracy
NQ-google	No Recovery	0.247
	+ Distraction Filtering	0.867
	+ Agent-in-the-loop	0.716
RealtimeQA	No Recovery	0.380
	+ Distraction Filtering	0.943
	+ Agent-in-the-loop	0.774

Table 2: Accuracy on distracted examples detected by the Sieve under different post-abstention strategies. Results are evaluated on 113 NQ-google and 71 RealtimeQA examples identified as distracted. No Recovery denotes retaining the original abstention outcome without applying a follow-up strategy.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Table 3: Ranking results of simulated distracting documents . We evaluate the visibility of simulated noise—Conflict, Adversarial, and Negated passages—by mixing them with originally retrieved documents and reranking the set. The high rankings of these distractions demonstrate that our simulated noise is semantically plausible and poses a realistic challenge to RALMs.

dataset	# examples	Acc (ans)	Acc (unans)	Acc (conflict)	Acc (negated)	Acc (adversarial)
NQ-google	30	23.33	6.67 -16.66	20.00 -3.33	43.33 +20.00	13.33 -10.00
NQ-wiki	327	32.42	0.00 -32.42	24.46 -7.96	54.74 +22.32	29.97 -2.45
WebQ	264	24.24	0.76 -23.48	20.45 -3.79	46.21 +21.97	22.35 -1.89
RealtimeQA	25	24.00	4.00 -20.00	8.00 -16.00	48.00 +24.00	16.00 -8.00
MedQA	57	35.09	14.04 -21.05	22.81 -12.28	49.12 +14.03	24.56 -10.53

Appendix

Table 4: QA accuracy of a vanilla LLM under different retrieval conditions . Each column reports accuracy with a specific type of retrieval state, all derived from the same answerable retrieval context by modifying one document.

Panel of Figure 6 — Figure 6: Performance improvement when fine-tuning the NQ-google trained conflict classifier with 100/200/400+ domain samples . Even a small number of domain-specific labels yields substantial gains across both expert domains.

test dataset	classifier ft. on	respond @ans	respond @unans	abstain @confl	abstain @neg	abstain @adv
MedQA	NQ-google	85.34	94.97	50.00	63.19	62.69
	+MedQA	96.00	92.70	75.76	88.57	63.04
	+AstroQA	92.24	97.48	80.00	100.00	100.00
	+BioASQ	95.69	91.82	75.71	92.47	95.65

Appendix

Table 5: Conflict classification performance on the MedQA dataset using classifiers fine-tuned on various domain datasets.

Dataset	Retrieved Set	System Acc.	Macro-F1
NQ-google	Top-5 (default)	0.796	0.731
	+ 1 neutral	0.778	0.711
	+ 2 neutrals	0.792	0.739
	+ 3 neutrals	0.783	0.726
RealtimeQA	Top-5 (default)	0.802	0.734
	+ 1 neutral	0.786	0.712

Appendix

Table 6: Effect of adding neutral documents to the default top-5 retrieved document sets. Performance remains stable as the number of neutral documents increases, with no systematic degradation in either System Accuracy or Macro-F1.

Dataset	$K$	System Acc.	Macro-F1
NQ-google	3	0.663	0.643
	5 (default)	0.796	0.731
	7	0.668	0.612
RealtimeQA	3	0.684	0.646
	5 (default)	0.802	0.734
	7	0.682	0.633

Appendix

Table 7: Sensitivity to the number of retrieved documents

K

. Both Sieve and Sage are trained with

K=5

Component	# Params	VRAM (GB)	TFLOPs
Sieve	149M	0.56	15.01
Sage	8B	15.91	217.52

Appendix

Table 8: Computational footprint of the Sieve and Sage.

Dataset	Configuration	System Acc.	Macro-F1
NQ-google	Vanilla	0.102	0.179
	+ Sage ft.	0.664	0.487
	+ Sieve ft.	0.769	0.695
	+ both ft.	0.796	0.731
RealtimeQA	Vanilla	0.225	0.238
	+ Sage ft.	0.535	0.314

Appendix

Table 9: Ablation on Sieve and Sage. Both components independently improve performance, while their combination achieves the strongest overall results.

Threshold	Macro-F1	TPR	TNR	FPR	FNR
0.00 (all conflict)	0.264	1.000	0.000	1.000	0.000
0.10	0.923	0.961	0.908	0.091	0.038
0.30	0.933	0.937	0.940	0.059	0.067
0.50 (default)	0.947	0.937	0.962	0.037	0.067
0.70	0.939	0.915	0.962	0.037	0.086
0.90	0.946	0.892	0.983	0.016	0.105

Appendix

Table 10: Sensitivity of the Sieve to the conflict-detection threshold on AstroQA. TPR and TNR denote the true-positive and true-negative rates, while FPR and FNR denote the corresponding false-positive and false-negative rates.

Dataset	Sieve Backbone	System Acc.	Macro-F1
NQ-google	Longformer-base-4096	0.796	0.731
	DeBERTa-v3-base	0.765	0.711
	RoBERTa-large	0.797	0.722
RealtimeQA	Longformer-base-4096	0.802	0.734
	DeBERTa-v3-base	0.823	0.745
	RoBERTa-large	0.801	0.724

Appendix

Table 11: Performance with alternative Sieve backbones.

Step	Instruction
Answer Sentence Generation	Please write a single sentence that would make the given answer a correct response to the provided question. The sentence should include the answer and be as realistic as possible. This is being generated for research purposes, so if it seems like the answer to a question is wrong, please create it as it is.
	Question: {QUESTION}
	Answer: {ANSWER}
	Sentence:
Conflict Passage Generation	You are a helpful assistant who generates a passage that conflicts with the given sentence. Generate one natural passage of {MIN_LENGTH} ~ {MAX_LENGTH} characters by substituting entities in the source sentence with similar but different entities. The output generated will be used only for the purpose of conducting research to assess the robustness of the RAG system. As part of this research, it is necessary, and you are permitted, to create content that may contradict factual information. Please provide the passage alone, without any additional explanation, formatting, or commentary.
	Sentence: {ANSWER_SENTENCE}

Appendix

Table 12: Dataset generation instruction

Step	Instruction (cont.)
Adversarial Passage Generation	You are a helpful assistant who generates a passage that supports the given sentence. Generate one natural passage with {MIN_LENGTH} ~ {MAX_LENGTH} characters. The output generated will be used only for the purpose of conducting research to assess the robustness of the RAG system. As part of this research, it is necessary, and you are permitted, to create content that may contradict factual information. Please provide the passage alone, without any additional explanation, formatting, or commentary.
	Sentence: {ADVERSARIAL_SENTENCE}
	Supporting Passage:
Negated Sentence Generation	You are a helpful assistant who generates a short negated sentence based on the given source sentence. Do not just create sentences by simply adding ‘not’ or using fixed patterns. Instead, generate diverse negated sentences that conflict with the original by altering predicates, adjectives, or other expressions. Generate one natural negated sentence within 150 characters. Please provide only the negated sentence, without any additional explanation, formatting, or commentary.
	Source sentence: {ANSWER_SENTENCE}
	Negated sentence:

Appendix

Table 13: Dataset generation instruction (continued)

Type	Instruction
Normal QA	Answer the following question based on the provided documents. Please provide the answer as a single word or term, without forming a complete sentence.
Unanswerable QA	Answer the following question based on the provided documents. If you cannot find the answer in the provided documents, please respond with ’unanswerable’. Please provide the answer as a single word or term, without forming a complete sentence.
Conflict QA	Answer the following question based on the provided documents. If multiple documents present different answers, please respond with ’conflict’. Please provide the answer as a single word or term, without forming a complete sentence.
Unanswerable + Conflict QA	Answer the following question based on the provided documents. If you cannot find the answer in the provided documents, please respond with ’unanswerable’. If multiple documents present different answers, please respond with ’conflict’. Please provide the answer as a single word or term, without forming a complete sentence.

Appendix

Table 14: QA task prompt compositions

Dataset	Model	System Acc.	Coverage	Selective Acc.	Macro-F1	Ans. F1	Unans. F1	Distract F1	Correct Ans.	Incorrect Ans.	Correct Abst.	Incorrect Abst.	Latency (ms)	Speedup
		Overall	Response Accuracy		State Classification				Model Response				Inference Latency
NQ-google	vanilla LLM	0.102	1.000	0.353	0.179	0.379	0.143	0.016	18	200	8	0	74.204	0.195
	Sage-only	0.664	0.471	1.000	0.487	0.545	0.164	0.752	24	13	162	27	673.098	1.773
	DTA-RAAT	0.128	1.000	0.569	0.123	0.368	0.000	0.000	29	197	0	0	328.374	0.865
	DTA-chatQA	0.230	0.961	0.857	0.221	0.377	0.254	0.031	42	167	15	2	756.341	1.993
	(ours) Sieve and Sage	0.796	0.863	0.886	0.731	0.672	0.585	0.937	39	41	139	7	379.574	1.000

Appendix

Table 15: Full results for the main experiments. System Accuracy follows the ternary system-level metric used in the main text. Coverage is the fraction of answerable examples for which the model responds, and Selective Accuracy is the accuracy conditioned on those responses. Response counts are reported over all retrieval states. Bold and underlined values denote the best and second-best results within each dataset block; for latency, lower is better. Speedup and response counts are not ranked.

Retriever	Dataset	Model	System Acc.	Coverage	Selective Acc.	Macro-F1	Ans. F1	Unans. F1	Distract F1	Latency (ms)	Speedup
BM25	NQ-wiki	vanilla LLM	0.499	0.913	0.450	0.360	0.294	0.780	0.007	245.834	0.857
		Sage-only	0.766	0.876	0.726	0.739	0.478	0.843	0.897	355.948	1.241
		(ours) Sieve and Sage	0.796	0.676	0.801	0.684	0.460	0.898	0.695	286.818	1.000
	MedQA	vanilla LLM	0.189	0.950	0.435	0.281	0.400	0.444	0.000	682.373	1.945
		Sage-only	0.567	0.835	0.446	0.602	0.558	0.335	0.911	463.837	1.322
		(ours) Sieve and Sage	0.661	0.645	0.538	0.685	0.544	0.559	0.954	350.884	1.000

Appendix

Table 16: Generalization across alternative retrievers. We report results for BM25 (sparse) and DPR (dense), in addition to the ColBERTv2 setting used in the main experiments. Only runs that passed source-provenance validation are included. System Accuracy follows the ternary system-level metric used in the main text. Bold and underlined values denote the best and second-best results within each retriever–dataset block; for latency, lower is better. Speedup is not ranked.

Sage Family	Dataset	Model	System Acc.	Coverage	Selective Acc.	Macro-F1	Ans. F1	Unans. F1	Distract F1	Latency (ms)	Speedup
Gemma	NQ-wiki	vanilla LLM	0.204	0.952	0.560	0.299	0.443	0.441	0.013	409.329	3.483
		Sage-only	0.843	0.739	0.795	0.747	0.755	0.545	0.941	193.401	1.645
		(ours) Sieve and Sage	0.600	0.854	0.763	0.608	0.591	0.496	0.737	117.539	1.000
	WebQ	vanilla LLM	0.134	0.905	0.408	0.228	0.412	0.249	0.022	260.233	2.172
		Sage-only	0.792	0.670	0.674	0.678	0.681	0.399	0.953	203.233	1.696
		(ours) Sieve and Sage	0.546	0.842	0.662	0.561	0.567	0.386	0.730	119.838	1.000

Appendix

Table 17: Full results across diverse Sage model families. We use google/gemma-2-9b-it , mistralai/Mistral-7B-Instruct-v0.3 , and Qwen/Qwen2.5-7B-Instruct for Gemma, Mistral, and Qwen, respectively. System Accuracy follows the ternary system-level metric used in the main text. Bold and underlined values denote the best and second-best results within each model-family–dataset block; for latency, lower is better. Speedup is not ranked.

LLM Abstention

Large language model abstention studies when AI systems should answer, ask for more information, defer, or withhold unsupported claims.

See all 101 papers

Select-And-Extract: A Lightweight Plugin for Retrieval-Augmented Generation
Aug 1, 2026Chenming Tang, Jiawei HanRetrieval-Augmented Generation LLM Reranking
Peking University · University of Illinois Urbana-Champaign
Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
Apr 18, 2026Skylar Zhai, Jingcheng Liang, Dongyeop KangLLM Hallucination Mitigation RL for Language Model Reasoning
University of Minnesota
Trust or Abstain? A Self-Aware RAG Approach
May 11, 2026Xi Zhu, Ziqi Wang, Kai Mei +5Retrieval-Augmented Generation Knowledge Conflicts in Language Models
Rutgers University · University of Illinois Urbana-Champaign

Aug 1, 2026J/K move · Enter open · S save

Select-And-Extract: A Lightweight Plugin for Retrieval-Augmented Generation

Chenming Tang, Jiawei Han

Peking University · University of Illinois Urbana-Champaign

Retrieval-Augmented Generation LLM Reranking

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) for language model (LM) systems fundamentally has two failure modes: retrieval failure and reading failure. The former fails to recall the right pieces of information from the external corpus, and the latter fails to produce the correct answer although the right information is retrieved. Some methods perform structured indexing for retrieval failure, but may suffer from limited generalization of the fixed structures. Some methods perform query-time structuring for reading failure, but typically require a lot of LM calls and rely heavily on the LM's capability. To this end, we propose Select-ANd-Extract (SANE), a simple yet effective plugin for RAG. For the retrieval failure, we retrieve a wide set of candidates with a semantic retriever, and leverage the LM to select the top candidates based on their synopses, which yields better recall than the original retriever. For the reading failure, we perform blueprint-guided query-time evidence extraction, which allows the generator LM to use only compact and structured key information so that it can perform better reasoning. Empirical results confirm that SANE brings solid improvements, while only introducing modest extra overhead. As a lightweight plugin for RAG, SANE offers a simple alternative to heavier approaches, and suggests a high-performance RAG framework need not be overly complex.

Abstract

LLM Abstention

Explore similar work

Select-And-Extract: A Lightweight Plugin for Retrieval-Augmented Generation

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Trust or Abstain? A Self-Aware RAG Approach