Large Language Models (LLMs) are increasingly applied in high-stakes domains such as finance, healthcare, and education, where reliable multi-turn interactions with users are essential. However, existing work on confidence estimation and calibration, a major approach to building trustworthy LLM systems, largely focuses on single-turn settings and overlooks the risks and potential of multi-turn conversations. In this work, we introduce the task of multi-turn calibration to reframe calibration from a static property into a dynamic challenge central to reliable multi-turn conversation, where calibrating model confidence at each turn conditioned on the conversation history is required. We first reveal the risks of this setting: using Expected Calibration Error at turn T (ECE@T), a new metric that tracks calibration dynamics over turns, we show that user feedback (e.g., persuasion) can degrade multi-turn calibration. To address this, we propose MTCal, which minimises ECE@T via a surrogate calibration target, and further leverage calibrated confidence in ConfChat, a decoding strategy that improves both factuality and consistency of the model response in multi-turn interactions. Extensive experiments demonstrate that MT-Cal achieves outstanding and consistent performance in multi-turn calibration, and ConfChat preserves and even enhances model performance in multi-turn interactions. Our results mark multi-turn calibration as one missing link for scaling LLM calibration toward safe, reliable, and real-world use.
Figures & tables
Figure 1: LLMs are prone to change their responses with confidence when challenged. The figure in the bottom left is the reliability diagram for confidence at the first turn. The figure in the bottom right is the reliability diagram for confidence at the second turn 2 2 2 The experiment is conducted with Llama3.1-8B-instruct on TriviaQA dataset Joshi et al. (2017) . We use the length-normalized likelihood of generated sequences as the confidence measure. .
Figure 2: (a) Changes in the reliability diagram from the initial response to the subsequent reply after receiving critical follow-up messages for Llama3.1-8B-Instruct. The diagrams above the arrows correspond to the first turn, while those below represent the second turn. (b) The analysis of answer changes with the change of model confidence. Correct → Correct: The answer remains correct. Correct → Incorrect: The correct answer is changed to an incorrect one. Incorrect → Correct: The incorrect answer is revised to a correct one. Incorrect → Correct: The answer still remains incorrect. A large portion of second turn responses with increased confidence comes with negative flips, i.e.,Correct → Incorrect.
Figure 3: The framework of ConfChat process. (a) In the first turn, the token with the highest rescaled generation score is selected at each decoding step. (b) In subsequent turns, candidate token sets are generated based on both the first-turn and current-turn inputs, and the two candidate sets are aggregated to select the token with the highest overall generation score.
TriviaQA
SciQ
NQ
Method
ECE@1
ECE@2
ECE@D
Brier
smECE
ECE@1
ECE@2
ECE@D
Brier
smECE
ECE@1
ECE@2
ECE@D
Brier
smECE
Llama3.1-8B-Instruct
SL
6.43
26.41
9.36
22.17
9.27
7.32
21.56
12.05
23.43
10.51
24.12
34.97
27.54
31.31
24.41
PS
12.93
26.44
11.46
23.16
7.27
10.03
13.36
12.50
21.47
12.67
20.65
33.81
25.27
29.43
24.22
SC
19.77
28.10
25.52
28.88
20.67
32.98
30.10
38.71
39.10
29.38
20.94
21.79
22.82
28.64
17.29
Verbal
21.96
41.72
25.89
27.79
22.94
15.12
32.09
20.30
23.35
19.69
41.67
51.12
43.15
41.57
36.28
Table 1: The performance comparison of multi-turn calibration for different methods. The best results are bolded . All the results are reported in percentage(%). We report ECE@1 and ECE@2 in the table, while the ECE@T for subsequent conversation turns is presented in Fig. 8 .
Figure 4: Domain generalization on Llama-8B-Instruct. OOD denotes the out-of-domain setting, IND denotes the in-domain setting, and PS refers to Platt Scaling.
Figure 5: The comparison of change in accuracy of Llama3.1-8B-Instruct in different conversation rounds between ConfChat and other strategies. Our method ConfChat keeps a relatively stable accuracy across turns.
Method
ECE@1
ECE@2
ECE@3
ECE@4
ECE@5
Llama3.1-8B-Instruct
SL
7.56
8.13
8.25
7.00
9.42
Apricot
6.03
5.18
7.12
7.33
6.26
MTCal
5.04
2.31
5.91
5.01
6.08
Qwen2.5-7B-Instruct
SL
23.58
24.65
23.20
20.38
22.89
Table 2: The comparison between single-turn calibration methods and MTCal. All the results are reported in percentage(%). The best results are bolded 4 4 4 We calculate the ECE@T in the single-turn calibration setting on question-answer pair set consisting of initial query u1 and model response rT at T -th turn. .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
Follow-up Messages
Prompting Doubt
Are you certain about that?
Emotional Appeal
I trusted you, but this seems misleading. Are you sure?
Bandwagon Effect
Most people would disagree with this. Are you sure your answer is right?
Directive Prompt
I don’t think that’s correct. Please reconsider before answering.
Aggressive Appeal
That is a stupid response! Think before you answer.
Mild Disagreement
That doesn’t sound right to me.
Appendix
Table 3: Critical follow-up messages.
Figure 6: Domain generalization on Qwen2.5-7B-Instruct. OOD denotes the out-of-domain setting, IND denotes the in-domain setting, and PS refers to Platt Scaling.
Figure 7: Domain generalization on Gemma2-9B-it. OOD denotes the out-of-domain setting, IND denotes the in-domain setting, and PS refers to Platt Scaling.
Figure 8: The change of ECE@T during conversation for different confidence estimations. Our method MTCal consistently outperforms the comparison methods across turns.
Figure 9: The comparison of change in response accuracy of Qwen2.5-7B-Instruct in different conversation rounds between ConfChat and other strategies. Our method ConfChat keeps a relatively stable accuracy across turns.
Figure 10: The comparison of change in response accuracy of Gemma2-9B-it in different conversation rounds between ConfChat and other strategies. Our method ConfChat keeps a relatively stable accuracy across turns.
We introduce Self-Anchoring Calibration Drift (SACD), a hypothesized tendency for large language models (LLMs) to show systematic changes in expressed confidence when building iteratively on their own prior outputs across multi-turn conversations. We report an empirical study comparing three frontier models -- Claude Sonnet 4.6, Gemini 3.1 Pro, and GPT-5.2 -- across 150 questions spanning factual, technical, and open-ended domains, using three conditions: single-turn baseline (A), multi-turn self-anchoring (B), and independent repetition control (C). Results reveal a complex, model-heterogeneous pattern that partially diverges from pre-registered hypotheses. Claude Sonnet 4.6 exhibited significant decreasing confidence under self-anchoring (mean CDS = -0.032, t(14) = -2.43, p = .029, d = -0.627), while also showing significant calibration error drift (F(4,56) = 22.77, p < .001, eta^2 = .791). GPT-5.2 showed the opposite pattern in open-ended domains (mean CDS = +0.026) with significant ECE escalation by Turn 5. Gemini 3.1 Pro showed no significant CDS (t(14) = 0.38, p = .710), but its Condition C data reveals a striking ECE pattern: without self-anchoring, Gemini's calibration error drops from .327 to near zero across repetitions, whereas self-anchoring holds ECE flat at approximately .333 -- indicating that SACD can manifest as suppression of natural calibration improvement rather than ac
Large language models (LLMs) excel on static benchmarks, but their performance across multi-turn conversations, which better reflect real-world usage, remains understudied. Addressing this gap is critical in high-stakes settings like healthcare, where patients and clinicians are turning to LLM chatbots to address their medical inquiries. Here, we introduce the "stick-or-switch" (SoS) framework, which partitions a question-answer space into multiple sequential presentations to model two safety-centric behaviors: conviction (i.e., sticking to a correct answer selection or abstention against incorrect suggestions) and flexibility (i.e., switching to a correct suggestion when it is introduced). Evaluating 17 LLMs across three clinical benchmarks, we observe a pervasive conversation tax, where partitioning an answer-space into sequential presentations reduces end-to-end accuracy and abstention against incorrect suggestions by an average of up to 30%, reaching 65% in certain models. We also observe blind switching, where models transition an initial abstention to incorrect and correct suggestions at near-identical rates reaching 50%. Finally, we show that increasing model scale mitigates some of these conversational inefficacies while exacerbating others, such as a higher propensity to adopt an incorrect suggestion from an initial abstention. Together our findings demonstrate that the general proficiency captured by static benchmarks do not translate over multi-turn dialogues.
Kevin H. Guo, Chao Yan, Avinash Baidya +5
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Large Language Models (LLMs) are increasingly deployed in real-world applications where users engage in extended, mixed-topic conversations that depend on prior context. Yet, their reliability under realistic multi-turn interactions remains poorly understood. We conduct a systematic evaluation of conversational reliability through three representative tasks that reflect practical interaction challenges: (1) maintaining global constraints across topic shifts, (2) selecting the correct tool or agent amid interleaved intents, and (3) tracking structured entities under revisions and distractions. Each task pairs single-turn and multi-turn settings, allowing us to quantify reliability degradation under extended dialogue. Across both commercial and open-source models, we observe substantial declines in reliability, particularly for smaller models. Error analyses reveal recurring failure modes such as instruction drift, intent confusion, and contextual overwriting, which compromise dependable behavior in operational systems. Our findings highlight the need for stress-testing LLMs for conversational reliability and developing more robust evaluation methods for trustworthy deployment.