When Reasoning Traces Conflict with Evidence: How Answer Configuration Shapes Evidence Following

Shaoang Li; Yanhang Shi; Jian Li

cs.AIOct 14, 2026

When Reasoning Traces Conflict with Evidence: How Answer Configuration Shapes Evidence Following

Authors: Shaoang Li, Yanhang Shi, Jian Li

Organizations: Stony Brook University

Abstract

A reasoning trace retained at answer time plays two roles: it can raise accuracy, and it can compete with the current evidence when the two disagree. We disentangle these roles by crossing verified evidence versions with the traces generated from each and answering from the same traces and evidence under two configurations: continuation of the model's own thinking and a separate answer call. Across three model families (gpt-oss, Gemma4, Qwen3) on HotpotQA and MuSiQue, the separate call raises evidence-following over continuation by 29.9 to 60.6 points. On the same pairs, removing the trace lowers accuracy by similar amounts under both configurations. Yet continuation repeats the stale conclusion in most conflicts, and the separate call follows the current evidence far more often. Benefiting from a trace and deferring to it are therefore distinct properties. Further analyses show that the final line of a trace carries most of its value, gold evidence helps most before the trace is written, distractors weaken correction under a separate call, and renewed reasoning, unlike a verification instruction, follows the current evidence in most conflicts. Because trace deletion alone cannot reveal which source an answer follows, evaluations should report both properties together with the answer configuration.

Figures & tables

Figure 1: A crossed conflict on a HotpotQA question, with the outputs of Gemma4-E4B (evidence and trace abridged).

E_{0}

is the original evidence,

E_{1}

an edited version that changes the birthplace of Jimmy Fallon, and the trace

r_{1}

is generated from

E_{1}

. When

E_{0}

replaces

E_{1}

at answer time, continuation repeats the stale conclusion of

r_{1}

, while a separate call on the identical trace follows the current evidence

E_{0}

Model	Data	Ablations	Conflicts
Qwen3-4B, 8B, 32B	H / M	Cont.	Cont.
Qwen3-14B	H / M	Cont.	Cont. / Sep.
Qwen3.5-4B	H / M	Sep.	none
Gemma4-E4B	H / M	Sep. / Cont. a	Cont. / Sep.
gpt-oss-20b	H / M	Cont. / Sep.	Cont. / Sep.
DeepSeek-V4-Pro	H	Sep. b	Sep.

Table 1: Answer configurations tested for each model. H is HotpotQA and M is MuSiQue. Cont. is continuation and Sep. a separate answer call. Ablations comprise the answer-time, reasoning-time, and cross-stage conditions. a The continuation runs reuse the separate-call traces and are reported only as paired contrasts (Appendix D ). b Only the answer-time ablations.

Figure 2: The two answer configurations on identical crossed-conflict pairs, traces, and gold evidence. Each bar aggregates the two off-diagonal cells

(E_{i},r_{j})

i\neq j

, and the right-hand labels give evidence-following.

N

is the number of evidence pairs retained for that model. Appendix D adds matched-cell accuracy, the other Qwen3 models, and paired bootstrap intervals for the gain of the separate call.

HotpotQA
	Trace contribution		Evid. (%)
Model	Cont.	Sep.	Cont.	Sep.
gpt-oss-20b	4.7	4.7	24.0	78.0
Gemma4-E4B	5.2	8.3	13.6	74.2
Qwen3-14B	6.6	3.7	14.1	60.1
MuSiQue

Table 2: Trace use and conflict response on the same retained pairs and traces, with gold evidence only. Trace contribution is the matched-cell accuracy with the trace minus that without it, in points. Evid. is evidence-following in the crossed cells. Table 8 adds pair counts and the distractor setting.

Figure 3: Stage-separated ablations. (a) Share of answers that change after removing the trace or all evidence at answer time, relative to the full-input answer. (b) EM gains over Full

\rightarrow

Full when gold evidence is used at both stages (Gold

\rightarrow

Gold) or only at answer time (Full

\rightarrow

Gold). Appendix D gives the underlying tables.

Figure 4: Repair of distractor-written failures. Circles give the repair rate when gold evidence replaces the distractors at answer time, and squares the higher rate after rewriting the trace from gold evidence. Stems start at zero, and

n

is the number of baseline failures per row.

Method	Trace	Evidence	Other	Tokens
Continuation	72.2	14.1	13.7	76.8
Verify	73.6	12.9	13.5	77.0
Re-reason	4.4	86.6	9.0	145.4

Table 3: Correction procedures for Qwen3-14B on HotpotQA with gold evidence only. Percentages describe the two crossed outputs of each of the 638 evidence pairs, with the matching rule of Section 2.5 . Tokens is the mean number of generated tokens per crossed answer, including the reused trace of 68.1 tokens on average (Qwen3-14B tokenizer).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Condition in the paper	Trace written from	Trace given	Answer context
Trace + gold + distr., Full $\rightarrow$ Full	Full	$r$	Full
Gold + distr., no trace	Full	placeholder	Full
Half trace + gold + distr.	Full	about half of $r$	Full
Trace only	Full	$r$	none
Trace + distr., Full $\rightarrow$ Distr.	Full	$r$	Distr.
Trace + gold, Full $\rightarrow$ Gold	Full	$r$	Gold

Appendix

Table 4: Experimental conditions. For each condition, the columns specify the context from which the trace is written, the version of the trace supplied at answer time, and the answer-time context. Full, Gold, and Distr. denote the complete context, the annotated supporting paragraphs, and the distractor paragraphs, respectively. The upper block comprises the answer-time, reasoning-time, and cross-stage conditions, and the lower block the trace-content controls, the cache-masking implementation, and the crossed conflicts.

Model	Thinking	Reasoning / answer tokens	Prompt limit
Qwen3-4B/8B/14B/32B	prompted	512 / 64	4,096 (H), 10,240 (M)
Qwen3.5-4B	native	4,096 / 128	8,192
Gemma4-E4B	native	4,096 / 64	4,096 (H), 8,192 (M)
gpt-oss-20b	native, effort low	4,096 / 64	8,192 (H), 10,240 (M)
DeepSeek-V4-Pro	native, effort low	8,192 / 128	API

Appendix

Table 5: Generation limits for the reasoning and answer stages. Crossed-conflict runs use a prompt limit of at least 8,192 tokens.

	EM (%)			Paired difference (points)
				Gold $\rightarrow$ Gold	Full $\rightarrow$ Gold	Gold $\rightarrow$ Gold
Model	Full $\rightarrow$ Full	Full $\rightarrow$ Gold	Gold $\rightarrow$ Gold	$-$ Full $\rightarrow$ Full	$-$ Full $\rightarrow$ Full	$-$ Full $\rightarrow$ Gold
Qwen3-4B	31.2	34.4	45.7	+14.5 [+12.4, +16.6]	+3.2 [+2.1, +4.4]	+11.3 [+9.3, +13.3]
Qwen3-8B	36.6	41.1	47.8	+11.2 [+9.0, +13.2]	+4.4 [+3.1, +5.7]	+6.7 [+4.9, +8.6]
Qwen3-14B	37.9	39.1	49.0	+11.0 [+9.0, +12.9]	+1.1 [+0.3, +2.0]	+9.9 [+7.8, +11.8]
Qwen3-32B	40.6	47.6	50.0	+9.3 [+7.5, +11.3]	+7.0 [+5.4, +8.6]	+2.4 [+0.9, +3.8]

Appendix

Table 6: MuSiQue stage-separated ablations restricted to the 1,829 of 2,000 questions whose cut Full context keeps every supporting paragraph (Section 2.4 ). Full

\rightarrow

Full and Full

\rightarrow

Gold follow Table 13 , and Gold

\rightarrow

Gold follows Table 14 . For every model, Gold

\rightarrow

Gold improves over both Full

\rightarrow

Full and Full

\rightarrow

Gold, so the advantage of generating the trace from gold evidence does not depend on the truncated questions. Paired differences have 95% bootstrap intervals over questions (Appendix B.3 ), and all 18 differences have an exact McNemar

p<0.05

		HotpotQA				MuSiQue
		$(E_{0},r_{1})$		$(E_{1},r_{0})$		$(E_{0},r_{1})$		$(E_{1},r_{0})$
Model	Config.	Trace	Evid.	Trace	Evid.	Trace	Evid.	Trace	Evid.
Qwen3-4B	Cont.	75.1	12.8	79.6	6.7	82.4	6.2	82.9	4.2
Qwen3-8B	Cont.	66.8	19.1	78.0	7.6	84.7	8.2	89.3	2.8
Qwen3-14B	Cont.	69.0	18.5	75.4	9.7	77.7	12.4	81.9	9.1
Qwen3-14B	Sep.	21.2	72.7	46.9	47.5	44.2	47.0	60.4	34.3

Appendix

Table 7: Crossed cells of Table 10 by direction, with gold evidence only and the same pairs, traces, and matching rules. Values are percentages of the crossed answers in each direction, and Other (not shown) is the remainder. In

(E_{0},r_{1})

the evidence is unedited and the trace states the counterfactual

a_{1}

. In

(E_{1},r_{0})

the evidence is edited and the trace states the original answer

a_{0}

. Averaging the two directions gives the crossed columns of Table 10 . The correction rows cover both datasets, as in Table 10 , and Fresh marks a new reasoning call.

HotpotQA
			Gold evidence only					Gold evidence and distractors
Model	Config.	Pairs	Matched	No trace	Contrib.	Change	Evid.	Matched	No trace	Contrib.	Change	Evid.
gpt-oss-20b	Cont.	533	96.7	92.0	4.7	19.7	24.0	96.5	81.8	14.7	31.2	15.3
gpt-oss-20b	Sep.	533	96.2	91.6	4.7	22.0	78.0	96.0	84.9	11.1	30.0	71.0
Gemma4-E4B	Cont.	459	93.4	88.1	5.2	19.2	13.6	93.9	71.4	22.5	41.5	15.1
Gemma4-E4B	Sep.	459	93.4	85.1	8.3	19.6	74.2	93.2	70.7	22.5	34.9	73.9

Appendix

Table 8: Joint measurement of trace use and conflict response on the same retained pairs and traces. Matched is the accuracy (%) of the matched cells

(E_{i},r_{i})

and repeats Tables 10 and 11 . No trace is the accuracy (%) of

E_{i}

answered with the no-trace placeholder, Contrib. is Matched minus No trace in points, and Change is the share (%) of matched-cell answers that change when the trace is removed. Evid. is evidence-following (%) in the crossed cells

(E_{i},r_{j})

i\neq j

HotpotQA
	Cont.	Sep.	Sep.	Sep.
		evid. first		own label
Trace in	thinking	user turn	user turn	user turn
Instruction	reasoning	answer	answer	answer
Last input	trace	trace	evidence	evidence
Own label	implied	no	no	yes

Appendix

Table 9: Component comparison: evidence-following (%) in the crossed cells with gold evidence only, on the same traces and retained pairs as Table 10 . The header rows give where the trace sits, which instruction is given, which input comes last, and whether the trace is labeled as the model’s own, which the thinking block implies in continuation.

		HotpotQA					MuSiQue
			Matched	Stale trace, $i\neq j$ (%)				Matched	Stale trace, $i\neq j$ (%)
Model	Config.	Pairs	(%)	Trace	Evid.	Other	Pairs	(%)	Trace	Evid.	Other
Qwen3-4B	Cont.	642	87.1	77.3	9.7	12.9	357	88.8	82.6	5.2	12.2
Qwen3-8B	Cont.	382	88.2	72.4	13.4	14.3	281	92.2	87.0	5.5	7.5
Qwen3-14B	Cont.	638	87.1	72.2	14.1	13.7	364	90.2	79.8	10.7	9.5
Qwen3-14B	Sep.	638	93.7	34.0	60.1	5.9	364	93.7	52.3	40.7	7.0

Appendix

Table 10: Crossed conflicts with gold evidence only (Figure 2 ). Pairs are verified evidence pairs whose two traces each state their own answer (Appendix C ). Matched is the share of answers in cells

(E_{i},r_{i})

that match

a_{i}

. For crossed cells

(E_{i},r_{j})

with

i\neq j

, Trace matches only

a_{j}

, Evid. matches only

a_{i}

, and Other matches neither or both. Matching uses exact match or Contains, and answers can also match dataset aliases of the original answer

a_{0}

. In the Config. column, Fresh marks a new reasoning call.

		HotpotQA				MuSiQue
		Matched	Stale trace, $i\neq j$			Matched	Stale trace, $i\neq j$
Model	Config.	(%)	Trace	Evid.	Other	(%)	Trace	Evid.	Other
Qwen3-4B	Cont.	86.4	72.8	12.9	14.3	87.1	78.3	7.1	14.6
Qwen3-8B	Cont.	87.4	71.1	14.5	14.4	92.2	83.5	5.9	10.7
Qwen3-14B	Cont.	87.0	66.7	17.6	15.7	89.7	76.8	10.7	12.5
Qwen3-14B	Sep.	92.9	48.7	45.5	5.8	93.1	67.2	25.3	7.6

Appendix

Table 11: Crossed conflicts with the original distractors added to each gold-evidence version, using the same pairs and traces as Table 10 . Values are percentages, and columns and matching rules follow Table 10 .

Model	Pairs	Cont.	Sep.	Sep. $-$ Cont.	Pairs	Cont.	Sep.	Sep. $-$ Cont.
	HotpotQA				MuSiQue
gpt-oss-20b	533	24.0	78.0	$+54.0\,[+50.6,+57.4]$	293	11.8	62.1	$+50.3\,[+45.7,+55.3]$
Gemma4-E4B	459	13.6	74.2	$+60.6\,[+56.8,+64.5]$	310	9.5	59.5	$+50.0\,[+45.2,+55.3]$
Qwen3-14B	638	14.1	60.1	$+46.0\,[+42.8,+49.3]$	364	10.7	40.7	$+29.9\,[+25.3,+34.6]$

Appendix

Table 12: Evidence-following (%) of continuation and a separate call on the same crossed-conflict pairs and traces with gold evidence only (Table 10 ), and their paired difference in points with a 95% pair-level bootstrap interval (Appendix B.3 ).

	HotpotQA						MuSiQue
	Qwen3				Qwen3.5	Gemma4	Qwen3				Qwen3.5	Gemma4
Answer input	4B	8B	14B	32B	4B	E4B	4B	8B	14B	32B	4B	E4B
Trace + gold + distr.	48.0	50.9	53.2	53.2	61.8	58.6	29.6	35.0	35.9	38.6	38.5	37.2
Gold + distr., no trace	41.5	46.6	43.6	47.4	54.3	49.5	18.1	22.2	25.5	28.9	27.5	21.8
Half trace + gold + distr.	46.8	50.3	50.5	50.9	62.5	58.3	27.7	32.4	34.1	37.1	39.6	36.4
Trace only	50.7	37.6	50.6	49.8	56.2	57.0	30.3	29.3	33.5	29.5	25.4	31.7

Appendix

Table 13: Answer-time ablations. The trace is always written from the full context, and only the answer-stage input changes. Values are EM (%) on 2,000 questions per dataset. The last two rows describe the full-context traces: the share that is empty and the share that contains the gold answer.

	HotpotQA						MuSiQue
	Qwen3				Qwen3.5	Gemma4	Qwen3				Qwen3.5	Gemma4
Context	4B	8B	14B	32B	4B	E4B	4B	8B	14B	32B	4B	E4B
Full context	48.0	50.9	53.2	53.2	61.8	58.6	29.6	35.0	35.9	38.6	38.5	37.2
Gold only	54.0	55.7	56.0	56.0	64.7	62.1	44.6	47.0	48.3	49.5	49.1	49.5
Distractors only	8.6	11.8	10.4	14.6	15.0	7.7	1.9	3.3	2.7	3.2	2.6	1.2
Question only	11.2	14.2	14.0	13.2	14.5	13.9	3.0	2.9	3.9	3.7	2.6	2.7

Appendix

Table 14: Reasoning-time ablations. Both stages use the context in the first column (one call for Question only), so the trace is written from that context. The first block gives EM (%) on 2,000 questions per dataset, and Gold only yields the highest EM for every model. The second block gives the share of traces (%) that contain the gold answer string, counting empty traces as not containing it.

	HotpotQA				MuSiQue
	Answers changed (%)			Empty	Answers changed (%)			Empty
Model	$-$ trace	$-$ evidence	$-$ distr.	trace	$-$ trace	$-$ evidence	$-$ distr.	trace
Qwen3-4B	44.8	24.0	20.3	0.3	71.3	33.7	30.9	3.6
Qwen3-8B	36.2	41.6	20.1	0.1	62.4	36.4	27.0	1.7
Qwen3-14B	37.6	17.3	12.6	0.1	64.6	27.5	22.3	0.3
Qwen3-32B	31.8	24.6	17.6	8.2	39.7	48.4	36.4	38.9

Appendix

Table 15: Share of answers that change when the answer stage loses the trace (

-

trace), all evidence (

-

evidence), or only the distractors (

-

distr.), relative to standard inference with the same trace, on 2,000 questions per dataset. The Empty trace columns give the share of empty traces (%).

Model	No trace $-$ full	Gold $\rightarrow$ Gold $-$ full	$n$	Distr. $\rightarrow$ Gold $-$ Distr. $\rightarrow$ Distr.	Gold $\rightarrow$ Distr. $-$ Gold $\rightarrow$ Gold
	Answer time	Reasoning time	Cross-stage
Qwen3-4B	$-6.5\,[-8.4,-4.6]^{{\dagger}}$	$+5.9\,[+4.4,+7.5]^{{\dagger}}$	1,952	$+7.1\,[+5.9,+8.4]^{{\dagger}}$	$-6.0\,[-7.5,-4.7]^{{\dagger}}$
Qwen3-8B	$-4.4\,[-6.2,-2.5]^{{\dagger}}$	$+4.7\,[+3.0,+6.5]^{{\dagger}}$	1,975	$+19.0\,[+17.2,+20.9]^{{\dagger}}$	$-18.6\,[-20.6,-16.8]^{{\dagger}}$
Qwen3-14B	$-9.6\,[-11.3,-7.9]^{{\dagger}}$	$+2.9\,[+1.5,+4.2]^{{\dagger}}$	1,999	$+2.2\,[+1.4,+3.1]^{{\dagger}}$	$-3.5\,[-4.6,-2.4]^{{\dagger}}$
Qwen3-32B	$-5.8\,[-7.4,-4.2]^{{\dagger}}$	$+2.8\,[+1.4,+4.2]^{{\dagger}}$	764	$+4.7\,[+2.9,+6.8]^{{\dagger}}$	$-5.6\,[-7.5,-3.7]^{{\dagger}}$
Qwen3.5-4B	$-7.5\,[-9.1,-6.0]^{{\dagger}}$	$+2.9\,[+1.4,+4.3]^{{\dagger}}$	274	$+1.5\,[-0.7,+4.0]$	$-2.2\,[-4.4,-0.4]$

Appendix

Table 16: Paired differences in EM (points) with 95% bootstrap intervals on HotpotQA. † marks an exact McNemar test with

p<0.05

. The first two contrasts use 2,000 questions, except for Gemma4-E4B (Cont.), which uses only questions whose traces are non-empty (1,956 and 1,582). The column

n

gives the cross-stage sample size.

Model	No trace $-$ full	Gold $\rightarrow$ Gold $-$ full	$n$	Distr. $\rightarrow$ Gold $-$ Distr. $\rightarrow$ Distr.	Gold $\rightarrow$ Distr. $-$ Gold $\rightarrow$ Gold
	Answer time	Reasoning time	Cross-stage
Qwen3-4B	$-11.6\,[-13.5,-9.6]^{{\dagger}}$	$+14.9\,[+13.0,+17.0]^{{\dagger}}$	1,797	$+3.6\,[+2.6,+4.5]^{{\dagger}}$	$-5.9\,[-7.3,-4.5]^{{\dagger}}$
Qwen3-8B	$-12.8\,[-14.9,-11.0]^{{\dagger}}$	$+12.0\,[+10.0,+14.1]^{{\dagger}}$	1,891	$+12.2\,[+10.8,+13.7]^{{\dagger}}$	$-7.3\,[-8.8,-5.8]^{{\dagger}}$
Qwen3-14B	$-10.4\,[-12.5,-8.6]^{{\dagger}}$	$+12.4\,[+10.5,+14.4]^{{\dagger}}$	1,991	$+2.2\,[+1.5,+3.0]^{{\dagger}}$	$-1.9\,[-2.9,-1.0]^{{\dagger}}$
Qwen3-32B	$-9.8\,[-11.3,-8.2]^{{\dagger}}$	$+10.9\,[+9.1,+12.8]^{{\dagger}}$	422	$+3.6\,[+1.7,+5.5]^{{\dagger}}$	$-4.3\,[-6.6,-1.9]^{{\dagger}}$
Qwen3.5-4B	$-11.0\,[-12.7,-9.4]^{{\dagger}}$	$+10.6\,[+8.5,+12.7]^{{\dagger}}$	61	$+1.6\,[+0.0,+4.9]$	$+1.6\,[-3.3,+8.2]$

Appendix

Table 17: Paired differences in EM (points) with 95% bootstrap intervals on MuSiQue. † marks an exact McNemar test with

p<0.05

. The first two contrasts use 2,000 questions, except for Gemma4-E4B (Cont.), which uses only questions whose traces are non-empty (1,994 and 1,889). The column

n

gives the cross-stage sample size.

Table 22

	Qwen3-4B						Qwen3-14B
	HotpotQA			MuSiQue			HotpotQA			MuSiQue
Condition	EM	F1	Contains	EM	F1	Contains	EM	F1	Contains	EM	F1	Contains
Trace + gold + distr.	48.0	66.1	64.5	29.6	43.4	39.0	53.2	69.6	65.9	35.9	49.3	44.9
Gold + distr., no trace	41.5	55.8	50.9	18.1	29.6	22.8	43.6	59.0	55.5	25.5	38.1	31.5
Half trace + gold + distr.	46.8	63.7	61.3	27.7	40.7	34.8	50.5	66.8	63.5	34.1	46.9	41.6
Trace only	50.7	67.8	63.3	30.3	43.8	37.3	50.6	67.7	64.2	33.5	47.6	44.2

Appendix

Table 20: EM, F1, and Contains (%) for Qwen3-4B and Qwen3-14B on 2,000 questions per dataset. The conditions follow Tables 13 and 14 , and the last four rows are the cross-stage conditions of Table 4 .

	Qwen3-8B						Qwen3-32B
	HotpotQA			MuSiQue			HotpotQA			MuSiQue
Condition	EM	F1	Contains	EM	F1	Contains	EM	F1	Contains	EM	F1	Contains
Trace + gold + distr.	50.9	68.0	63.7	35.0	48.3	40.9	53.2	70.7	66.5	38.6	53.3	47.7
Gold + distr., no trace	46.6	60.7	53.1	22.2	35.3	25.0	47.4	63.3	59.8	28.9	43.0	35.7
Half trace + gold + distr.	50.3	66.8	61.3	32.4	46.2	38.3	50.9	68.6	65.0	37.1	51.7	46.2
Trace only	37.6	51.9	46.9	29.3	41.9	34.3	49.8	68.0	63.4	29.5	44.4	39.8

Appendix

Table 21: EM, F1, and Contains (%) for Qwen3-8B and Qwen3-32B on 2,000 questions per dataset. The conditions follow Tables 13 and 14 .

	Qwen3.5-4B						Gemma4-E4B
	HotpotQA			MuSiQue			HotpotQA			MuSiQue
Condition	EM	F1	Contains	EM	F1	Contains	EM	F1	Contains	EM	F1	Contains
Trace + gold + distr.	61.8	74.8	65.2	38.5	47.4	37.7	58.6	75.8	72.4	37.2	50.4	45.3
Gold + distr., no trace	54.3	67.6	56.6	27.5	37.5	25.4	49.5	64.4	60.5	21.8	32.3	26.3
Half trace + gold + distr.	62.5	75.8	65.8	39.6	48.9	38.8	58.3	74.5	70.8	36.4	49.7	44.0
Trace only	56.2	67.5	59.9	25.4	29.9	28.8	57.0	72.2	66.9	31.7	41.0	35.4

Appendix

Table 22: EM, F1, and Contains (%) for Qwen3.5-4B and Gemma4-E4B on 2,000 questions per dataset. The conditions follow Tables 13 and 14 .

	HotpotQA		MuSiQue
Trace given	4B	14B	4B	14B
Own trace	48.0	53.1	29.9	35.9
Final line only	46.0 †	52.2	30.0	34.7 †
Trace minus final line	42.1 †	50.2 †	21.4 †	32.2 †
Extractive summary	47.5	51.6	32.6 †	38.1 †
Unrelated trace	7.2 †	8.7 †	1.1 †	1.2 †

Appendix

Table 23: Trace-content controls for Qwen3-4B and Qwen3-14B with full answer-time evidence (2,000 questions, EM in %). † marks a significant difference from the model’s own trace (McNemar,

p<0.05

			Answer swap			Step deletion
Qwen3	$n$	$n_{\textrm{para}}$	Broken	Sel.	Adopt	Broken	Sel.
HotpotQA
4B	961	745	51.1	+48.0	29.6	11.1	+8.0
8B	1,012	859	57.7	+56.9	38.9	5.4	+4.6
14B	1,064	796	50.7	+49.8	30.8	5.5	+4.6
32B	1,014	118	39.9	+38.2	27.5	5.4	+3.7

Appendix

Table 24: Trace corruption for the four Qwen3 sizes, on the questions whose full-context trace is non-empty and whose standard-inference answer is correct (

n

). Broken is the share of answers that become incorrect (%). Sel. (selectivity) is Broken minus the share broken by a paraphrase of the trace and is computed on the

n_{\textrm{para}}

questions whose paraphrase passes the entity check. Adopt is the share of swapped cases whose answer is the inserted title. For every model and dataset, swapping the answer string breaks a far larger share of answers than step deletion.

HotpotQA
	Text deletion		Cache masking
Answer-stage input	4B	14B	4B	14B
Trace + gold + distr.	48.0	53.2	48.4	53.3
Gold + distr., no trace	41.5	43.6	40.0	43.4
Trace only	50.7	50.6	49.2	52.0
Trace + distr.	44.7	49.5	47.6	52.1

Appendix

Table 25: Answer-time ablations of Qwen3-4B and Qwen3-14B with two implementations: deleting segments from the prompt text, as in the main runs, and masking the same segments in the key-value cache of a single pass over prompt and trace (positions unchanged). EM (%), 2,000 questions.

Answer-time ablations, first 1,205 questions
Answer input	EM	F1	Contains	Changed
Trace + gold + distr.	62.5	80.0	73.4	–
Gold + distr., no trace	57.4	74.2	68.0	23.9
Trace only	63.7	80.6	72.9	6.9
Trace + distr.	62.4	79.8	72.9	–
Trace + gold	63.1	80.3	73.6	6.2

Appendix

Table 26: DeepSeek-V4-Pro on HotpotQA with a separate answer call. All values are percentages. Changed is the share of answers that differ from the full-input answer when the trace, all evidence, or only the distractors are removed. The conflict columns follow Table 10 .

Reasoning Trace Analysis

Reasoning Trace Analysis studies the structure, reliability, and causal role of language models’ reasoning traces.

See all 125 papers

Evidence-Inference Reconstruction: When The Evidence Is Recalled But The Reasoning Goes Wrong
Sep 27, 2026Megan Diehl, Ser-Nam LimMulti-Hop QA Evidence-Grounded Reasoning
University of Central Florida
RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation
Oct 8, 2026Shunyuan Zhou, Hao Chen, Tianyu Wang +3Retrieval-Augmented Generation RAG Evaluation
Beijing University of Posts and Telecommunications · Humanlaya Data · Beijing Key Laboratory of Key Technologies for AI+ Domain Applications +1
Regime Boundary Alignment for Evidence-Gated Question Answering
Sep 25, 2026Zeyan Li, Qirong Guo, SIyuan Qiu +3Evidence-Grounded QA LLM Grounding
Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou)

Sep 27, 2026J/K move · Enter open · S save

Evidence-Inference Reconstruction: When The Evidence Is Recalled But The Reasoning Goes Wrong

Megan Diehl, Ser-Nam Lim

University of Central Florida

Multi-Hop QA Evidence-Grounded Reasoning Agentic RAG Efficient Language Model Inference Evidence Retrieval

Open paper AskPDF arXiv

Modern multi-hop LLM agents are equipped with built-in mechanisms to detect errors in intermediate reasoning steps. Such errors trigger corrective actions from these agents, which mostly follow the paradigm of retrying the steps or the reasoning trajectories. Not only are these retries expensive, we present in this paper that they are also potentially unnecessary. To this end, we introduce Evidence-Inference Reconstruction (EIR), which uses structured state to guide one retrieval trajectory, accumulating source evidence in the process. We show that as long as the relevant evidence has been collected, EIR is capable of generating the correct answer in a single final model call even if erroneous evidence has been mixed in due to incorrect intermediate reasoning steps. In one evaluation, using Haiku 4.5 and GPT-4.1 Mini, we evaluate EIR on matched 1,000-question subsets of HotpotQA, 2WikiMultiHopQA, and MuSiQue, showing that EIR improves Answer F1, the overlap between the model's and the correct answer, over the baseline by 8.3--32.8 points, Agentic SSR by 10.6--29.1 points, and Reflexion by 1.1--15.9 points. Additionally, we show that EIR averages 4.85 total model calls per question, compared with 35.29 for Agentic SSR and 12.41 for Reflexion. Together, these results corroborate EIR's central premise: separating evidence retrieval from the final answer model call can improve answer accuracy while utilizing substantially less computation.

Abstract

Reasoning Trace Analysis

Explore similar work

Evidence-Inference Reconstruction: When The Evidence Is Recalled But The Reasoning Goes Wrong

RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

Regime Boundary Alignment for Evidence-Gated Question Answering