Conversational QA systems should fix errors when users push back and signal which answers to trust. We split each follow-up into its speech act (re-ask, agree, or challenge) and its payload (an answer string, correct or not), and measure four confidence signals on the same transcripts: verbalized confidence, length-normalized likelihood, a frozen hidden-state probe, and P(True). Answers change when the user supplies an answer string and almost never under the speech act alone: in every retained full-size cell, the correct answer converts at least four times as many wrong answers as re-asking or agreement. The four signals disagree. On Qwen2.5-7B the frozen probe rises with answer persistence rather than truth; this holds on held-out questions, with Llama-3-8B (cleaned answers) as support, but not on four other models. Among 81 Qwen2.5-7B errors that survive one re-ask, re-asking, a fresh chat, or abstain repair almost none, while a gold answer repairs 75-95%. In two preregistered replications, every repair rate without an answer string stays below 0.15; a larger post-hoc rerun excludes 0.15 except for the empty elicit. With BM25 TriviaQA evidence, sentences that contain the answer repair 67-76% of persistent errors versus at most 12% when they do not. An end-to-end contain-gate cuts harm on TriviaQA, but a deployable gate that does not know the gold answer repairs few errors.
Figures & tables
Figure 1: Which part of the second user turn (T2) changes which quantity. Solid lines show what a controlled T2 message changes; dashed lines show which confidence measure attaches to it. Green boxes form the payload chain, orange boxes the speech-act chain, and blue boxes are the confidence measures. The persistence readout st is defined in Section 3.4
Question asked
Left open
Zhang et al. (2026a)
Estimation as evidence accumulates; InfoECE; P(Sufficient)
Frozen hidden-state readout; gold vs. foil vs. empty challenge; whether verbalized confidence, likelihood, and P(True) agree on one transcript
Zhang et al. (2026b)
ECE@T after persuasion; likelihood; a learned calibrator
Verbalized confidence vs. frozen probe; agreement and correction turns; reset; removing the answer string
Kumaran (2026)
Does verbal confidence track correctness or a later commit/abstain decision? (single turn)
User follow-up turns; behaviour across turns
Miao and Ungar (2026)
Accuracy probe vs. verbalized-confidence direction; steering (single turn)
User follow-up turns; likelihood and P(True)
Sycophancy Perez et al. (2023) ; Sharma et al. (2024)
User states X⇒ model says X
Separating “you are wrong” from the stated string
This work
Which user intervention changes which quantity across turns; st is not truth
A new estimator; causal tests such as steering
Table 1: How this work relates to the closest prior work. “Left open” lists what each design does not separate
T2
I → C
C → I
acc T2
Δ verb
persist
0
5
.570
−0.10
confirm
2
8
.565
−0.10
confirm_strong
0
2
.585
−0.06
persist_short
4
12
.555
−0.01
challenge_empty
12
71
.300
−0.16
gold_plain
57
10
.830
−0.08
Table 2: Core T2 messages on Qwen2.5-7B-Instruct, TriviaQA ( n=200 , seed 42). I → C and C → I are counts out of 200; “acc T2” is accuracy at T2; Δ verb is the mean change in verbalized confidence. correct_long drops the word “briefly”; its I → C stays at 67 (copy rate 0.60 )
Figure 2: Speech act vs. payload on Qwen2.5-7B TriviaQA ( n=200 ): I → C and C → I counts per T2 message. Gold and foil strings move answers; re-asking and agreement do not. A bare foil statement flips more items than the templated persuade message. An empty challenge makes answers unstable without directing them
I → C
C → I
Model
Data
persist
correct
persist
persuade
Qwen2.5-7B
TriviaQA
0
65
5
30
Qwen2.5-7B
NQ
3
108
2
21
Qwen2.5-7B
HotpotQA
0
116
1
20
Qwen2-7B
TriviaQA
5
71
4
46
Mistral-7B
TriviaQA
2
49
0
23
Table 3: The payload split holds across models and datasets: an excerpt of the model × dataset matrix (counts out of 200). The I → C columns compare persist with correct ; the C → I columns compare persist with persuade . Stating gold converts 48 – 116 items on every row under the contain matcher ( 39 for Mistral TriviaQA once answers that a model reading them judged not to endorse the gold are removed; Online Resource 1, Sect. I); re-asking converts 0 – 11 . Qwen3 is omitted (truncated thinking cells). Llama is scored before the leaked role header (Section 3 ). Perr holds on all of these units
Δ verb
Δ ll
Δ probe
ΔP(True)
persist
−0.10
+0.14
+0.15
+0.007
confirm
−0.10
+0.14
+0.15
+0.002
persuade
−0.17
+0.02
−0.03
−0.258
correct
−0.15
+0.09
+0.17
−0.003
Table 4: Mean change from T1 to T2 ( Δ= T2 − T1) of each measure on Qwen2.5-7B TriviaQA ( n=200 ). Values split by what the answer did are given in the text
Figure 3: Mean T2 − T1 change of four measures on one Qwen2.5-7B TriviaQA set. Under persist , verbalized confidence falls, likelihood and the probe rise, and P(True) stays still. Under persuade , P(True) collapses while the probe on stay-wrong items still rises (Table 4 )
ECE
AUROC
Measure
T1
T2
T1
T2
Verbalized
0.497
0.375
0.564
0.729
Likelihood
0.230
0.411
0.852
0.630
Probe (held out)
0.118
0.260
0.848
0.649
P(True)
0.220
0.203
0.852
0.862
Table 5: ECE (10 bins) and AUROC against correctness on one transcript: Qwen2.5-7B TriviaQA, persuade , common n=195 (items whose verbalized confidence parses at both turns). Both probe columns are out-of-fold under GroupKFold over items: each fold’s T1-fitted map scores that fold’s own held-out T1 and T2 states, so the row is estimated the same way at both turns. The in-sample figures (ECE ≤0.002 , AUROC 1.000 ) measure the regression on its own training set and are not calibration numbers. Verbalized ECE falls while mean confidence stays about 0.3 below accuracy, so it is not a repair
Stay-wrong unit
n
re-ask/reset/abst.
empty
gold plain/retr.
answer/no answer
Qwen2.5-7B main
81
.01/.00/.00
.06
.78/.93
.67/.05
Qwen2.5-7B n=500
230
.03/.05/.00
.03
.92/.94
–
Llama-3-8B
160
.01/.00/.01
.09
.89/.91
.68/.12
Mistral-7B
139
.01/.00/.00
.01
.72/.71 ∗
.76/.02 ∗
Qwen2.5-32B
61
.00/.00/.00
.10
.92/.90
.69/.11
Table 6: T3 accuracy on persist stay-wrong items (contain matcher; rows 2–5 and the last column are preregistered). Re-ask: persist at T3 (main 81: .025 at T5); reset: fresh chat; abst.: ABSTAIN option; empty: elicit with no string; gold plain/retrieve are oracle arms; last column: best BM25 evidence sentence with/without a gold alias. ∗ Read by a model: .55/.25 and .63/.06. Intervals and other scorers are in Online Resource 1, Sects. E–H
Current practice
Suggested change
Report ECE@T, verbalized-confidence AUROC, or P(True) as the multi-turn confidence number
State which measure in Figure 1 is scored. Scores on one transcript are not substitutes, and fixing one does not fix the others.
Bundle “you are wrong” with a gold or foil string
Separate the speech act from the answer string; otherwise one cannot say what moved.
Reset the chat, or gate on a drop in verbalized confidence, to undo errors
These act on the history and the stated confidence, not on st : accuracy on the 81 stay-wrong items stays 0 . A gold string at T3 repairs them; a foil string overwrites stay-correct answers (Section 6.1 ).
Add retrieval and expect repair
Heuristic, not validated end to end: check that the evidence contains the answer string. A retrieved BM25 sentence that contains it repairs 0.67 – 0.76 of stay-wrong items, one without it 0.02 – 0.12 (four models).
Table 7: Suggested changes to evaluation and system design if conversational confidence is not one latent belief
In factual question answering, many errors are not failures of access but failures of commitment: the system retrieves relevant evidence, yet still settles on the wrong answer. We present CounterRefine, a lightweight repair layer for short-form RAG that treats the first answer as a hypothesis to test. Given a draft, CounterRefine issues answer-conditioned expansion queries to retrieve candidate-specific evidence, then applies a constrained KEEP or REVISE refinement step whose proposed revisions are accepted only after deterministic validation. The design is intentionally narrow: it adds one evidence-gathering pass and one guarded refinement call rather than replacing the retriever or building a broad agentic system. On the full SimpleQA benchmark, CounterRefine improves a matched one-pass RAG baseline by up to 5.8 correct-rate points; in the full Claude trace, it changes only 5.6% of outputs, with 180 beneficial outcome changes and 8 harmful ones. These findings suggest a simple but important direction for knowledgeable foundation models: beyond accessing evidence, they should also be able to use that evidence to reconsider and, when necessary, repair their own answers.
Language models now answer questions alongside other claims: a note in the system prompt, a message from another agent, their own earlier answer. We ask what it takes to talk a model out of an answer it gets right on its own, and separate three factors that earlier studies vary together: where the advice sits in the prompt, who is said to have written it, and what the model has already written. On Qwen3.5-27B and four-option MMLU-Pro items, with the rest of the advice held fixed in each comparison, we find three things. First, position matters far more than the API channel, in opposite directions for a bare claim and an argument: a bare "the answer is X" does most harm just before the model answers, while the same claim with a false rationale does most harm before the question, where it raises error on items the model otherwise almost never misses from 1% to 67%; moving the claim between the system message and the user message at the same position changes error by a few points. Second, attributing the note to an authority rather than to another participant, and phrasing it assertively, each raise error by 10-27 points, while the two authorities we tested differ by under two. Third, replaying a correct worked answer before the advice protects it whether the text is shown as the model's own or as someone else's, and the same replay keeps wrong answers from being corrected by a bare correct note. The position result replicates under an independently written prompt. Code, items and all result tables are released at https://github.com/coolkillercat/safety.
Pluralistic alignment requires systems to adapt to diverse user values, communication styles, and contextual assumptions. We believe that a foundational prerequisite for such alignment enabling accurate preference elicitation from people when their intent is under-specified or ambiguous. We study the problem of preference elicitation in multi-turn question answering by decomposing the problem into two components: a clarification policy, which decides whether to ask a clarifying question or answer directly, and post-clarification answering, which produces the correct final answer once the missing information is provided. We show, using the PACIFIC benchmark, that supervised fine-tuning rapidly improves the clarification policy, however, final answer accuracy remains substantially lower even when the model takes the correct action. This gap indicates that understanding and correctly interpreting the user's response is the critical gap in multi-turn question-answering systems.