CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering

Tianyi Huang; Ying Kai Deng

cs.CLMar 17, 2026

CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering

Authors: Tianyi Huang, Ying Kai Deng

Organizations: Ryquo · App-In Club

Abstract

In factual question answering, many errors are not failures of access but failures of commitment: the system retrieves relevant evidence, yet still settles on the wrong answer. We present CounterRefine, a lightweight repair layer for short-form RAG that treats the first answer as a hypothesis to test. Given a draft, CounterRefine issues answer-conditioned expansion queries to retrieve candidate-specific evidence, then applies a constrained KEEP or REVISE refinement step whose proposed revisions are accepted only after deterministic validation. The design is intentionally narrow: it adds one evidence-gathering pass and one guarded refinement call rather than replacing the retriever or building a broad agentic system. On the full SimpleQA benchmark, CounterRefine improves a matched one-pass RAG baseline by up to 5.8 correct-rate points; in the full Claude trace, it changes only 5.6% of outputs, with 180 beneficial outcome changes and 8 harmful ones. These findings suggest a simple but important direction for knowledgeable foundation models: beyond accessing evidence, they should also be able to use that evidence to reconsider and, when necessary, repair their own answers.

Figures & tables

Figure 1: Schematic overview of CounterRefine . Starting from a draft answer produced by Baseline-RAG , CounterRefine collects additional evidence intended to support, check, or contradict that candidate, evaluates the draft answer against the resulting evidence set, and then passes the result to a constrained keep-vs.-revise gate. The final answer is either the original draft or a revised candidate accepted under evidence-based validation using the same evidence set. For readability, the figure abstracts away some implementation details from the text, including question-type-conditioned query selection and per-query snippet aggregation.

Step	Example trace
Question	Who was awarded the Oceanography Society’s Jerlov Award in 2018?
Draft	Collin Roesler
Second-pass queries	Original question; original question + “Collin Roesler”; “Collin Roesler”
Retrieved signal	Award-list evidence names Annick Bricaud as the 2018 recipient.
Decision	REVISE to Annick Bricaud

Table 1: Illustrative Stage-2/Stage-3 trace. The second pass does not solve the question from scratch; it tests a concrete draft answer against newly surfaced evidence.

System	Retrieval queries	Raw snippets	Model calls
Baseline-RAG	1	$\leq 5$	1 draft
CounterRefine	3–4	$\leq 15$ – $20$	1 draft + 1 refine

Table 2: Per-question computational profile for the SimpleQA web-retrieval setting before deduplication. The 3–4 retrieval-query range for CounterRefine includes the shared first-pass query and depends on whether the question type triggers the bare-answer query.

Benchmark	Metric	Claude 4.6			GPT-5
Benchmark	Metric	Base-RAG	CounterRefine	$\Delta$	Base-RAG	CounterRefine	$\Delta$
SimpleQA (100)	Correct $\uparrow$	65.0	75.0	+10.0	57.0	71.0	+14.0
SimpleQA (100)	F1 $\uparrow$	65.0	75.0	+10.0	59.7	73.2	+13.5
SimpleQA (4,326)	Correct $\uparrow$	63.7	67.7	+4.0	67.3	73.1	+5.8
SimpleQA (4,326)	F1 $\uparrow$	64.1	68.1	+4.0	58.6	72.1	+13.5
HotpotQA (100)	EM $\uparrow$	74.0	79.0	+5.0	71.0	76.0	+5.0

Table 3: Main results across backbones and benchmarks. Baseline-RAG is the matched one-pass retrieval baseline using the same retriever and backbone as CounterRefine . SimpleQA uses the official grader. HotpotQA uses the distractor development setting on 100-example and 300-example slices. The HotpotQA (300) CounterRefine values are the full-system row of Table 4 . All values are percentages.

Figure 2: Correct-rate comparison on the full SimpleQA benchmark. CounterRefine is compared only with the matched one-pass retrieval baseline using the same backbone and retrieval setup.

Variant	Claude 4.6		GPT-5
Variant	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$
Full CounterRefine	76.0	86.1	71.7	84.9
Second pass with original question only	76.0	86.1	71.7	84.9
Answer-conditioned, no validator	74.7	86.0	72.0	85.0
Answer-conditioned, no canonicalization	75.0	86.5	71.7	84.9
Simple reconsideration	74.7	86.0	72.0	85.0

Table 4: HotpotQA-300 component ablation. The table compares the full CounterRefine pipeline with variants that use only the original question in the second pass, remove deterministic validation, remove answer canonicalization, or replace the guarded refinement step with simple reconsideration. Values are percentages; the Full CounterRefine row corresponds to the HotpotQA-300 system row in Table 3 , and EM is rounded from exact-match counts over 300 examples.

Figure 3: Outcome transitions on the full SimpleQA benchmark for Claude 4.6. Of all examples, 63.5% stay correct, 32.1% stay not-correct, 4.2% are corrected (incorrect $\rightarrow$ correct), and 0.2% are harmed (correct $\rightarrow$ incorrect).

Question (abridged)	Baseline-RAG	Final	Gold	Diagnosis
Representative hurt cases
Who appointed the Chief Justice of India, Mirza Hameedullah Beg, in 1977?	Fakhruddin Ali Ahmed	Indira Gandhi	Fakhruddin Ali Ahmed	Nearby evidence named the government behind the elevation rather than the appointing president requested by the question.
What day, month, and year was the municipality of Arboletes, Antioquia, Colombia, founded?	1920	August 1958	July 20, 1920	The refinement evidence referred to municipal establishment rather than the original founding date.
On what day, month, and year did Manny Pacquiao marry Jinkee Jamora?	May 10, 1999	May 10, 2000	May 10, 1999	A noisy timeline-style snippet overrode an already correct baseline year.
Representative helped cases
Who was awarded the Oceanography Society’s Jerlov Award in 2018?	Collin Roesler	Annick Bricaud	Annick Bricaud	Answer-conditioned retrieval surfaced an award-list entry explicitly naming the 2018 winner.

Table 5: Representative failures and successes from the full Claude trace log.

Explore similar work

What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions
Aug 9, 2026Wenzhang DuRetrieval-Augmented Generation Counterfactual Reasoning in Language Models
Independent Researcher China
Detecting and Repairing Hallucinations in Retrieval-Augmented Generation
Aug 29, 2026Sai Krishna Reddy Mulakkayala, Niki van Stein, Aske PlaatLLM Hallucination Mitigation Retrieval-Augmented Generation
LIACS, Leiden University, Leiden, The Netherlands
MIRAGE: Defending Long-Form RAG Against Misinformation Pollution
Jul 6, 2026Saadeldine Eletter, Ruihong Zeng, Yuxia Wang +3RAG Poisoning Attacks Retrieval Robustness
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · INSAIT, Sofia University St. Kliment Ohridski

Aug 9, 2026J/K move · Enter open · S save

What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions

Wenzhang Du

Independent Researcher China

Retrieval-Augmented Generation Counterfactual Reasoning in Language Models RAG Evaluation Causal Interventions in Language Models

Open paper AskPDF arXiv

A failed retrieval-augmented generation (RAG) answer can be consistent with several unseen responses to evidence repair. We introduce Pair-ID, an offline audit that holds one query, retrieval state, and reader constant, then crosses two operations, adding missing support and deleting verified nonsupport, to measure a same-failure counterfactual response vector. A complete funnel over 19,981 benchmark queries identifies 11,105 eligible Qwen failures, from which a prospectively fixed SHA-256 ordering selects 1,200 before generating any sampled response. Among 1,190 regenerated-valid failures, support addition repairs 197/600 JOINT cases (0.328, 95% CI [0.292, 0.367]), and deletion repairs 162/1,190 cases (0.136, 95% CI [0.117, 0.155]); length- and position-matched shams retain semantic contrasts of 0.223 and 0.101. The original view carries partial predictive signal for individual response cells (macro AUROC 0.678; Brier 0.152 versus 0.160 for a marginal baseline), but exact-vector accuracy, 0.637, does not exceed the 0.646 majority-vector baseline, and vector macro-F1 is 0.170. Across four readers, both marginal sensitivities recur, while pooled exact-vector agreement is 0.675-0.765 and JOINT-only agreement falls to 0.538-0.691. These results show that evidence sensitivity occurs at meaningful rates in the hash-selected eligible-failure sample, is only partially predictable from the observed failure, and is conditional on the reader. The evidence supports a frame-scoped offline response audit, not an information-theoretic impossibility result, reader-independent taxonomy, or runtime repair policy.

Abstract

Explore similar work

What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions

Detecting and Repairing Hallucinations in Retrieval-Augmented Generation

MIRAGE: Defending Long-Form RAG Against Misinformation Pollution