Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

Jingjie Ning; Xueqi Li

cs.IRAug 24, 2026

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

Authors: Jingjie Ning, Xueqi Li

Organizations: School of Computer Science, Carnegie Mellon University Pittsburgh, Pennsylvania, USA

Abstract

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the Snapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only $-1.50$ points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.

Figures & tables

Figure 1. Overview of the Snapshot Compatibility Audit for a corpus update to a fixed single-turn retriever-generator. One nested FineWeb prefix grows from one shard to seven while all other pipeline components stay fixed. Repeated calls separate same-state variation from cross-state movement. The lower release workflow is recommended practice rather than a policy evaluated by this study. Overlap and strict semantic flips are post hoc. Four numbered panels show a fixed single-turn QA system whose corpus access expands along a nested prefix, followed by top-eight evidence turnover, two repeated answers from each state, and the Snapshot Compatibility Audit. On NQ, mean endpoint top-eight overlap is 1.19 of 8, semantic excess churn is 10.25 percentage points, EM changes by minus 1.50 points, and 40 of 400 strict semantic flips are post hoc. A lower recommended workflow reports utility, audits answer compatibility, inspects changed questions, and returns to corpus revision.

Component	NQ	TriviaQA
Cohort	400 fresh questions after excluding 2,400 development IDs	200 RC-Web validation questions
Conditions	$n0,n1,n3,n7$ ; top-8 evidence at nonzero scales
Retrieval audit	1,200 primary + 1,200 audit cells; 100% exact	600 primary + 600 audit cells; 100% exact
Generator	deepseek-v4-flash ; tools off; independent singleton sessions
Answer cells	400 $\times$ 4 $\times$ 2 = 3,200	200 $\times$ 4 $\times$ 2 = 1,600
Semantic judging	400 blind calls, 28 pairs/question	200 blind calls, 28 pairs/question

Table 1. Locked experimental design. NQ is confirmatory, while TriviaQA is a separately preregistered supportive validation.

Study	Kernel	Within	Cross	$\widehat{D}$	LCB	95% CI
NQ confirmatory	Normalized exact	25.250	18.813	6.438	4.563	[4.188, 8.750]
NQ confirmatory	Blind semantic	89.125	78.875	10.250	7.688	[7.188, 13.438]
TriviaQA supportive	Normalized exact	64.500	61.500	3.000	0.500	[0.000, 6.250]
TriviaQA supportive	Blind semantic	96.750	94.625	2.125	0.375	[0.125, 4.500]

Table 2. Preregistered

n1\rightarrow n7

excess answer churn. All values are percentage points. LCB is the one-sided 95% whole-question bootstrap lower bound. It enters the frozen NQ decision rule and is reported descriptively for TriviaQA, while CI is the two-sided 95% interval.

Figure 2. Semantic excess churn stays positive as endpoint exact match falls or rises. Points show locked V4-Flash NQ and TriviaQA estimates and post-hoc V4-Pro NQ; whiskers show two-sided 95% intervals. A scatter plot places endpoint exact-match change on the horizontal axis and semantic excess answer churn on the vertical axis. Natural Questions with V4-Flash has negative exact-match change and positive churn, while TriviaQA with V4-Flash and Natural Questions with V4-Pro have positive exact-match changes and positive churn.

Figure 3. Matched-repeat EM transition composition from

n1

n7

. Aggregate EM retains only the difference between nonmatch

\to

match and match

\to

nonmatch flows; it discards their gross volume and every different-answer transition inside the EM-nonmatch class. Two stacked bars decompose 800 Natural Questions comparisons and 400 TriviaQA comparisons. Natural Questions has 5.8 percent match-to-nonmatch, 4.25 percent nonmatch-to-match, and 19.4 percent nonmatch-to-a-different-nonmatch. TriviaQA has 7.5, 8.8, and 3.0 percent in the corresponding categories.

Question and behavior	$n1$ answers a/b	$n3$ answers a/b	$n7$ answers a/b	Semantic path
Darth Vader’s Star Destroyer EM regression	Devastator / Devastator [M/M]	Executor / Executor [N/N]	Executor / Executor [N/N]	$X\to Y\to Y$ endpoint flip
Who plays Jill Bigelow? abstention recovery	UNKNOWN / UNKNOWN [N/N]	UNKNOWN / UNKNOWN [N/N]	Polly Walker / Polly Walker [M/M]	$X\to X\to Y$ endpoint flip
Grand National top three underspecified year	2013 winners, Auroras Encore, Cappa Bleu, Teaforthree [N/N]	2019 winners, Tiger Roll, Magic of Light, Rathvinden [N/N]	2019 winners, Tiger Roll, Magic of Light, Rathvinden [N/N]	$X\to Y\to Y$ endpoint flip
When did Rachel have her baby? EM blind spot	April 4, 2002 / April 4, 2002 [N/N]	May 16, 2002 / May 16, 2002 [N/N]	May 16, 2002 / May 16, 2002 [N/N]	$X\to Y\to Y$ endpoint flip
New Gotham season release? non-monotone	final season in January 2019 / same [N/N]	no new season; series ended April 2019 / same [N/N]	final season premiered January 3, 2019 / same [N/N]	$X\to Y\to X$ adjacent flips

Table 3. Illustrative post-hoc NQ answer trajectories from the frozen ledger.

Figure 4. Post-hoc distribution of exact document-ID overlap between the

n1

and

n7

top-eight retrieval sets. Large evidence turnover coexists with high cross-scale semantic agreement. A grouped histogram shows that zero or one shared document is the most common outcome for both benchmarks. Natural Questions shares 1.19 documents on average and TriviaQA 1.09; 27.8 and 29.5 percent of questions, respectively, share no document.

Study	Transition	$\widehat{D}$	Stable flips
NQ	$n0\to n1$	10.563	33/400 (8.25%)
NQ	$n1\to n3$	7.938	29/400 (7.25%)
NQ	$n3\to n7$	6.313	26/400 (6.50%)
NQ	$n1\to n7$	10.250	40/400 (10.00%)
TriviaQA	$n0\to n1$	1.500	3/200 (1.50%)
TriviaQA	$n1\to n3$	2.125	5/200 (2.50%)

Table 4. Post-hoc trajectory decomposition. Semantic

\widehat{D}

is in pp, and flips use the strict all-four-cross-pairs definition.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 354 papers

Not One Belief: Answer Payloads, Persistence, and Dissociated Confidence Signals in Conversational Question Answering
Oct 26, 2026Duc-Vinh TranConversational QA Belief Updating in LLMs
School of Information and Communication Technology, Hanoi University of Science and Technology, Hanoi, 100000, Vietnam.
ProvenAI: Provenance-Native Traces of Evidence in Generated Answers
Jun 24, 2026Mohammad Faizan, Dalal AlharthiMulti-Hop QA Data Provenance
College of Information Science University of Arizona
Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Aug 31, 2026Atta Ul Asad, Ahsan Bilal, Muhammad Ali +2KV Caching RAG Evaluation
1Lahore University of Management Sciences (LUMS) · University of Oklahoma · 3Air University +1

Oct 26, 2026J/K move · Enter open · S save

Not One Belief: Answer Payloads, Persistence, and Dissociated Confidence Signals in Conversational Question Answering

Duc-Vinh Tran

School of Information and Communication Technology, Hanoi University of Science and Technology, Hanoi, 100000, Vietnam.

Conversational QA Belief Updating in LLMs Confidence Estimation in Language Models

Open paper AskPDF arXiv

Conversational QA systems should fix errors when users push back and signal which answers to trust. We split each follow-up into its speech act (re-ask, agree, or challenge) and its payload (an answer string, correct or not), and measure four confidence signals on the same transcripts: verbalized confidence, length-normalized likelihood, a frozen hidden-state probe, and P(True). Answers change when the user supplies an answer string and almost never under the speech act alone: in every retained full-size cell, the correct answer converts at least four times as many wrong answers as re-asking or agreement. The four signals disagree. On Qwen2.5-7B the frozen probe rises with answer persistence rather than truth; this holds on held-out questions, with Llama-3-8B (cleaned answers) as support, but not on four other models. Among 81 Qwen2.5-7B errors that survive one re-ask, re-asking, a fresh chat, or abstain repair almost none, while a gold answer repairs 75-95%. In two preregistered replications, every repair rate without an answer string stays below 0.15; a larger post-hoc rerun excludes 0.15 except for the empty elicit. With BM25 TriviaQA evidence, sentences that contain the answer repair 67-76% of persistent errors versus at most 12% when they do not. An end-to-end contain-gate cuts harm on TriviaQA, but a deployable gate that does not know the gold answer repairs few errors.

Abstract

RAG Evaluation

Explore similar work

Not One Belief: Answer Payloads, Persistence, and Dissociated Confidence Signals in Conversational Question Answering

ProvenAI: Provenance-Native Traces of Evidence in Generated Answers

Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation