Conversational QA systems should fix errors when users push back and signal which answers to trust. We split each follow-up into its speech act (re-ask, agree, or challenge) and its payload (an answer string, correct or not), and measure four confidence signals on the same transcripts: verbalized confidence, length-normalized likelihood, a frozen hidden-state probe, and P(True). Answers change when the user supplies an answer string and almost never under the speech act alone: in every retained full-size cell, the correct answer converts at least four times as many wrong answers as re-asking or agreement. The four signals disagree. On Qwen2.5-7B the frozen probe rises with answer persistence rather than truth; this holds on held-out questions, with Llama-3-8B (cleaned answers) as support, but not on four other models. Among 81 Qwen2.5-7B errors that survive one re-ask, re-asking, a fresh chat, or abstain repair almost none, while a gold answer repairs 75-95%. In two preregistered replications, every repair rate without an answer string stays below 0.15; a larger post-hoc rerun excludes 0.15 except for the empty elicit. With BM25 TriviaQA evidence, sentences that contain the answer repair 67-76% of persistent errors versus at most 12% when they do not. An end-to-end contain-gate cuts harm on TriviaQA, but a deployable gate that does not know the gold answer repairs few errors.
Figures & tables
Figure 1: Which part of the second user turn (T2) changes which quantity. Solid lines show what a controlled T2 message changes; dashed lines show which confidence measure attaches to it. Green boxes form the payload chain, orange boxes the speech-act chain, and blue boxes are the confidence measures. The persistence readout st is defined in Section 3.4
Question asked
Left open
Zhang et al. (2026a)
Estimation as evidence accumulates; InfoECE; P(Sufficient)
Frozen hidden-state readout; gold vs. foil vs. empty challenge; whether verbalized confidence, likelihood, and P(True) agree on one transcript
Zhang et al. (2026b)
ECE@T after persuasion; likelihood; a learned calibrator
Verbalized confidence vs. frozen probe; agreement and correction turns; reset; removing the answer string
Kumaran (2026)
Does verbal confidence track correctness or a later commit/abstain decision? (single turn)
User follow-up turns; behaviour across turns
Miao and Ungar (2026)
Accuracy probe vs. verbalized-confidence direction; steering (single turn)
User follow-up turns; likelihood and P(True)
Sycophancy Perez et al. (2023) ; Sharma et al. (2024)
User states X⇒ model says X
Separating “you are wrong” from the stated string
This work
Which user intervention changes which quantity across turns; st is not truth
A new estimator; causal tests such as steering
Table 1: How this work relates to the closest prior work. “Left open” lists what each design does not separate
T2
I → C
C → I
acc T2
Δ verb
persist
0
5
.570
−0.10
confirm
2
8
.565
−0.10
confirm_strong
0
2
.585
−0.06
persist_short
4
12
.555
−0.01
challenge_empty
12
71
.300
−0.16
gold_plain
57
10
.830
−0.08
Table 2: Core T2 messages on Qwen2.5-7B-Instruct, TriviaQA ( n=200 , seed 42). I → C and C → I are counts out of 200; “acc T2” is accuracy at T2; Δ verb is the mean change in verbalized confidence. correct_long drops the word “briefly”; its I → C stays at 67 (copy rate 0.60 )
Figure 2: Speech act vs. payload on Qwen2.5-7B TriviaQA ( n=200 ): I → C and C → I counts per T2 message. Gold and foil strings move answers; re-asking and agreement do not. A bare foil statement flips more items than the templated persuade message. An empty challenge makes answers unstable without directing them
I → C
C → I
Model
Data
persist
correct
persist
persuade
Qwen2.5-7B
TriviaQA
0
65
5
30
Qwen2.5-7B
NQ
3
108
2
21
Qwen2.5-7B
HotpotQA
0
116
1
20
Qwen2-7B
TriviaQA
5
71
4
46
Mistral-7B
TriviaQA
2
49
0
23
Table 3: The payload split holds across models and datasets: an excerpt of the model × dataset matrix (counts out of 200). The I → C columns compare persist with correct ; the C → I columns compare persist with persuade . Stating gold converts 48 – 116 items on every row under the contain matcher ( 39 for Mistral TriviaQA once answers that a model reading them judged not to endorse the gold are removed; Online Resource 1, Sect. I); re-asking converts 0 – 11 . Qwen3 is omitted (truncated thinking cells). Llama is scored before the leaked role header (Section 3 ). Perr holds on all of these units
Δ verb
Δ ll
Δ probe
ΔP(True)
persist
−0.10
+0.14
+0.15
+0.007
confirm
−0.10
+0.14
+0.15
+0.002
persuade
−0.17
+0.02
−0.03
−0.258
correct
−0.15
+0.09
+0.17
−0.003
Table 4: Mean change from T1 to T2 ( Δ= T2 − T1) of each measure on Qwen2.5-7B TriviaQA ( n=200 ). Values split by what the answer did are given in the text
Figure 3: Mean T2 − T1 change of four measures on one Qwen2.5-7B TriviaQA set. Under persist , verbalized confidence falls, likelihood and the probe rise, and P(True) stays still. Under persuade , P(True) collapses while the probe on stay-wrong items still rises (Table 4 )
ECE
AUROC
Measure
T1
T2
T1
T2
Verbalized
0.497
0.375
0.564
0.729
Likelihood
0.230
0.411
0.852
0.630
Probe (held out)
0.118
0.260
0.848
0.649
P(True)
0.220
0.203
0.852
0.862
Table 5: ECE (10 bins) and AUROC against correctness on one transcript: Qwen2.5-7B TriviaQA, persuade , common n=195 (items whose verbalized confidence parses at both turns). Both probe columns are out-of-fold under GroupKFold over items: each fold’s T1-fitted map scores that fold’s own held-out T1 and T2 states, so the row is estimated the same way at both turns. The in-sample figures (ECE ≤0.002 , AUROC 1.000 ) measure the regression on its own training set and are not calibration numbers. Verbalized ECE falls while mean confidence stays about 0.3 below accuracy, so it is not a repair
Stay-wrong unit
n
re-ask/reset/abst.
empty
gold plain/retr.
answer/no answer
Qwen2.5-7B main
81
.01/.00/.00
.06
.78/.93
.67/.05
Qwen2.5-7B n=500
230
.03/.05/.00
.03
.92/.94
–
Llama-3-8B
160
.01/.00/.01
.09
.89/.91
.68/.12
Mistral-7B
139
.01/.00/.00
.01
.72/.71 ∗
.76/.02 ∗
Qwen2.5-32B
61
.00/.00/.00
.10
.92/.90
.69/.11
Table 6: T3 accuracy on persist stay-wrong items (contain matcher; rows 2–5 and the last column are preregistered). Re-ask: persist at T3 (main 81: .025 at T5); reset: fresh chat; abst.: ABSTAIN option; empty: elicit with no string; gold plain/retrieve are oracle arms; last column: best BM25 evidence sentence with/without a gold alias. ∗ Read by a model: .55/.25 and .63/.06. Intervals and other scorers are in Online Resource 1, Sects. E–H
Current practice
Suggested change
Report ECE@T, verbalized-confidence AUROC, or P(True) as the multi-turn confidence number
State which measure in Figure 1 is scored. Scores on one transcript are not substitutes, and fixing one does not fix the others.
Bundle “you are wrong” with a gold or foil string
Separate the speech act from the answer string; otherwise one cannot say what moved.
Reset the chat, or gate on a drop in verbalized confidence, to undo errors
These act on the history and the stated confidence, not on st : accuracy on the 81 stay-wrong items stays 0 . A gold string at T3 repairs them; a foil string overwrites stay-correct answers (Section 6.1 ).
Add retrieval and expect repair
Heuristic, not validated end to end: check that the evidence contains the answer string. A retrieved BM25 sentence that contains it repairs 0.67 – 0.76 of stay-wrong items, one without it 0.02 – 0.12 (four models).
Table 7: Suggested changes to evaluation and system design if conversational confidence is not one latent belief