From Noisy Retrieval to Decision Evidence: Silver Evidence Demonstrations for In-Context Learning

Masoud Monajatipoor; Fazlolah Mohaghegh; Hamid Reza Hassanzadeh; Akash Uttamrao Bhoite; Robert E. Tillman

cs.CLOct 23, 2026

From Noisy Retrieval to Decision Evidence: Silver Evidence Demonstrations for In-Context Learning

Authors: Masoud Monajatipoor, Fazlolah Mohaghegh, Hamid Reza Hassanzadeh, Akash Uttamrao Bhoite, Robert E. Tillman

Organizations: Optum AI

Abstract

RAG and retrieval-augmented few-shot ICL can struggle when decision-relevant evidence is buried in long, noisy context. Retrieved passages and demonstrations may contain redundant or distracting information. Although expert-labeled development examples may identify the text supporting each decision, these annotations are typically used only as answer-side rationales rather than as supervision for filtering noisy retrieval. We propose silver-evidence demonstrations, which replace raw retrieved passages in ICL examples with generated, target-conditioned evidence summaries. For each labeled development example, an LLM generates silver evidence without access to the expert label or gold evidence; this evidence is then paired with the expert label and gold evidence where available. At test time, a single structured call generates silver evidence from retrieved passages and then predicts the final label and decision evidence. Across EI 2.0, SciFact, synthetic policy retrieval, and HICRIC/IMR data, our full method improves over equally sized gold-only ICL (label-only for policy) on three of four datasets and improves evidence F1 on all annotated datasets. It also achieves higher mean label accuracy than InstructRAG-style ICL on all four datasets. These results suggest that replacing demonstration-side retrieved passages with compact, target-conditioned silver evidence is particularly useful for decisions based on long and noisy text, where it reduces distracting context while preserving decision-relevant signals and expert supervision.

Figures & tables

Figure 1: End-to-end silver-evidence pipeline. Stage 1 constructs demonstrations by generating silver evidence from retrieved passages without access to expert labels or expert gold evidence. Stage 2 uses those demonstrations to prompt the model to produce silver evidence before predicting the final label and evidence.

Figure 2: Qualitative SciFact validation example (claim 338). The retrieved abstract contains a genuine but decision-irrelevant benefit—reduced postoperative nausea and vomiting—while the claim concerns postoperative bleeding. Silver evidence isolates the target outcome and effect size (24% versus 4%; adjusted relative risk 6.80) before a Contradict prediction aligned with the expert rationale. Text is shortened for visualization.

Dataset	System	Accuracy	Macro-F1
EI 2.0	Vanilla RAG	$91.2\pm 0.3$	$91.0\pm 0.8$
	Gold-only ICL + RAG	$\mathbf{91.5\pm 0.2}$	$\mathbf{91.3\pm 0.7}$
	Silver-only ICL + RAG	$91.2\pm 0.3$	$91.0\pm 0.7$
	Full method	$91.4\pm 0.1$	$90.9\pm 0.4$
	InstructRAG-style ICL	$90.8\pm 0.3$	$90.5\pm 0.9$
	Oracle evidence	$93.1\pm 0.3$	$92.8\pm 0.1$

Table 1: Label prediction results for the primary GPT-5.5 experiments. Values are percentages reported as mean

\pm

standard deviation over three runs. The InstructRAG-style rows use full demonstration-side retrieval for EI 2.0 and SciFact and a compressed, budget-matched format for policy retrieval and HICRIC/IMR. For policy retrieval, silver-only ICL + RAG is identical to the full method because no expert gold evidence is available, so duplicate values are omitted. Oracle-evidence rows are upper bounds rather than deployable systems.

Dataset	System	Evidence F1	Sentence/Span F1
EI 2.0	Vanilla RAG	$55.7\pm 0.4$	$54.7\pm 0.9$
	Gold-only ICL + RAG	$58.4\pm 0.2$	$56.4\pm 0.1$
	Silver-only ICL + RAG	$61.7\pm 0.9$	$60.3\pm 1.0$
	Full method	$\mathbf{62.2\pm 1.0}$	$\mathbf{60.9\pm 0.4}$
	InstructRAG-style ICL	$43.5\pm 0.9$	$46.1\pm 0.3$
	Oracle evidence	$95.8\pm 0.5$	$95.4\pm 1.3$

Table 2: Evidence-overlap results for the primary GPT-5.5 experiments. Values are percentages reported as mean

\pm

standard deviation over three runs. Evidence F1 is token/rationale F1 for EI 2.0 and HICRIC/IMR and text F1 for SciFact. Sentence/Span F1 is span F1 for EI 2.0 and HICRIC/IMR and sentence-id F1 for SciFact. HICRIC/IMR uses weak reviewer-rationale evidence rather than expert span annotation. Policy retrieval lacks gold evidence spans and is omitted.

Dataset	LLM backend	Accuracy	Macro-F1	Evidence F1	Sentence/Span F1	Total tokens	Avg. latency (s)
EI 2.0	GPT-5.5	$91.4\pm 0.1$	$90.9\pm 0.4$	$62.2\pm 1.0$	$60.9\pm 0.4$	$2.19$ M	$11.7$
	GPT-5.6-Sol	$90.0\pm 0.8$	$89.6\pm 0.9$	$64.7\pm 0.2$	$64.0\pm 1.1$	$1.78$ M	$4.9$
	GPT-5.6-Terra	$\mathbf{91.9\pm 1.0}$	$\mathbf{91.7\pm 1.1}$	$66.7\pm 0.4$	$65.0\pm 0.9$	$2.09$ M	$11.1$
	GPT-5.6-Luna	$90.0\pm 0.4$	$89.6\pm 0.7$	$62.3\pm 1.3$	$61.5\pm 0.8$	$1.80$ M	$3.7$
	Gemini Pro 3.1	$\mathbf{91.9\pm 0.2}$	$\mathbf{91.7\pm 1.2}$	$66.7\pm 0.1$	$65.0\pm 0.3$	$2.09$ M	$11.1$
	Gemini Flash 3.7	$90.4\pm 0.2$	$90.2\pm 0.5$	$\mathbf{67.3\pm 0.8}$	$\mathbf{66.2\pm 0.7}$	$2.00$ M	$6.5$

Table 3: Full-method backend sensitivity for the main-paper comparison. The complete backend table appears in Supplementary Table 6 . Accuracy, Macro-F1, Evidence F1, and Sentence/Span F1 are reported as mean

\pm

standard deviation. Total-token and latency columns report means across runs without dispersion terms. Bold identifies the best available result for each dataset and metric; ties are retained.

Dataset	Published comparator / setting	Published	Full method
EI 2.0	Biomed RoBERTa macro-F1 DeYoung et al. (2020)	78.0	90.9
SciFact	MultiVerS label F1 Wadden et al. (2022)	72.5	81.0 †
Policy	Adaptive DPO accuracy Sharifullin et al. (2026)	92.0	99.5
Policy	Fixed- $K$ accuracy Sharifullin et al. (2026)	62.0	99.5
HICRIC	ClinicalBERT accuracy Gartner (2024)	74.0	89.5 ‡
HICRIC	DistilBERT macro-F1 Gartner (2024)	72.7	89.4 ‡

Table 4: Contextual label-level comparison with published task-specific systems. Values are percentages; the Full method entry uses the metric named by each comparator row. Rows are not strictly comparable because retrieval setting, split, label space, and evaluation protocol differ. † SciFact is evaluated on the labeled validation split because official test labels are not public. ‡ HICRIC/IMR uses our adapted binary approve/deny appeal-policy setting rather than the original three-class benchmark.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Display name	Provider	API model identifier	Configured deployment identifier
GPT-4.1	OpenAI	gpt-4.1	gpt-4.1_2025-04-14_no-guardrails
GPT-5.2	OpenAI	gpt-5.2	gpt-5.2_2025-12-11
GPT-5.5	OpenAI	gpt-5.5	gpt-5.5_2026-04-24
GPT-5.6-Sol	OpenAI	gpt-5.6-sol	gpt-5.6-sol_2026-07-09
GPT-5.6-Terra	OpenAI	gpt-5.6-terra	gpt-5.6-terra_2026-07-09
GPT-5.6-Luna	OpenAI	gpt-5.6-luna	gpt-5.6-luna_2026-07-09

Appendix

Table 5: Provider-facing model and configured deployment identifiers used in the backend comparison.

Dataset	LLM backend	Accuracy	Macro-F1	Evidence F1	Sentence/Span F1	Total tokens	Avg. latency (s)
EI 2.0	GPT-4.1	$88.8\pm 1.2$	$88.3\pm 1.1$	$62.2\pm 1.1$	$61.1\pm 0.3$	$2.01$ M	$3.6$
	GPT-5.2	$88.5\pm 0.8$	$87.8\pm 1.2$	$60.6\pm 0.3$	$58.0\pm 0.6$	$2.07$ M	$5.0$
	GPT-5.5	$91.4\pm 0.1$	$90.9\pm 0.4$	$62.2\pm 1.0$	$60.9\pm 0.4$	$2.19$ M	$11.7$
	GPT-5.6-Sol	$90.0\pm 0.8$	$89.6\pm 0.9$	$64.7\pm 0.2$	$64.0\pm 1.1$	$1.78$ M	$4.9$
	GPT-5.6-Terra	$\mathbf{91.9\pm 1.0}$	$\mathbf{91.7\pm 1.1}$	$66.7\pm 0.4$	$65.0\pm 0.9$	$2.09$ M	$11.1$
	GPT-5.6-Luna	$90.0\pm 0.4$	$89.6\pm 0.7$	$62.3\pm 1.3$	$61.5\pm 0.8$	$1.80$ M	$3.7$

Appendix

Table 6: Complete full-method backend sensitivity. Accuracy, Macro-F1, Evidence F1, and Sentence/Span F1 are reported as mean

\pm

standard deviation when repeated runs are available; single-run summaries use

\pm 0.0

. Total-token and latency columns report means across runs without dispersion terms. Bold identifies the best available result for each dataset and metric; ties are retained.

Retrieval-Augmented ICL

Retrieval-augmented in-context learning (ICL) selects relevant examples or knowledge to help models adapt without updating their weights.

See all 45 papers

Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval
Mar 19, 2026Hangeol Chang, Changsun Lee, Seungjoon Rho +2Retrieval-Augmented Generation Query Reformulation
Graduate School of AI, KAIST, Republic of Korea · School of Electrical Engineering, KAIST, Republic of Korea · Department of Industrial Engineering, Yonsei University, Republic of Korea
Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG
Sep 27, 2026Suting Chen, Peichun Hua, Yunming XiaoRetrieval-Augmented Generation Evidence-Grounded QA
Northwestern University · The Chinese University of Hong Kong, Shenzhen
Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG
Oct 15, 2026Jiaqi Wei, Wenxuan Huang, Chengdong Xu +2Retrieval-Augmented Generation Evidence Retrieval
Zhejiang University Hangzhou, China · Shanghai AI Laboratory Shanghai, China · Fudan University Shanghai, China +3

Mar 19, 2026J/K move · Enter open · S save

Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval

Hangeol Chang, Changsun Lee, Seungjoon Rho, Junho Yeo+1

Graduate School of AI, KAIST, Republic of Korea · School of Electrical Engineering, KAIST, Republic of Korea · Department of Industrial Engineering, Yonsei University, Republic of Korea

Retrieval-Augmented Generation Query Reformulation Multiple-Choice Question Answering Medical QA

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by grounding generation in external, non-parametric knowledge. However, when a task requires choosing among competing options, simply grounding generation in broadly relevant context is often insufficient to drive the final decision. Existing RAG methods typically rely on a single initial query, which often favors topical relevance over decision-relevant evidence, and therefore retrieves background information that can fail to discriminate among answer options. To address this issue, here we propose Hypothesis-Conditioned Query Rewriting (HCQR), a training-free pre-retrieval framework that reorients RAG from topic-oriented retrieval to evidence-oriented retrieval. HCQR first derives a lightweight working hypothesis from the input question and candidate options, and then rewrites retrieval into three targeted queries that seek evidence to: (1) support the hypothesis, (2) distinguish it from competing alternatives, and (3) verify salient clues in the question. This approach enables context retrieval that is more directly aligned with answer selection, allowing the generator to confirm or overturn the initial hypothesis based on the retrieved evidence. Experiments on MedQA and MMLU-Med show that HCQR consistently outperforms single-query RAG and re-rank/filter baselines, improving average accuracy over Simple RAG by 5.9 and 3.6 points, respectively. Code is available at https://anonymous.4open.science/r/HCQR-1C2E.

Abstract

Retrieval-Augmented ICL

Explore similar work

Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval

Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG

Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG