Confidence Should Be Calibrated More Than One Turn Deep

Zhaohan Zhang; Chengzhengxu Li; Xiaoming Liu; Chao Shen; Ziquan Liu; Ioannis Patras

cs.CLApr 7, 2026

Confidence Should Be Calibrated More Than One Turn Deep

Authors: Zhaohan Zhang, Chengzhengxu Li, Xiaoming Liu, Chao Shen, Ziquan Liu, Ioannis Patras

Organizations: Queen Mary University of London · Xi’an Jiaotong University

Abstract

Large Language Models (LLMs) are increasingly applied in high-stakes domains such as finance, healthcare, and education, where reliable multi-turn interactions with users are essential. However, existing work on confidence estimation and calibration, a major approach to building trustworthy LLM systems, largely focuses on single-turn settings and overlooks the risks and potential of multi-turn conversations. In this work, we introduce the task of multi-turn calibration to reframe calibration from a static property into a dynamic challenge central to reliable multi-turn conversation, where calibrating model confidence at each turn conditioned on the conversation history is required. We first reveal the risks of this setting: using Expected Calibration Error at turn T (ECE@T), a new metric that tracks calibration dynamics over turns, we show that user feedback (e.g., persuasion) can degrade multi-turn calibration. To address this, we propose MTCal, which minimises ECE@T via a surrogate calibration target, and further leverage calibrated confidence in ConfChat, a decoding strategy that improves both factuality and consistency of the model response in multi-turn interactions. Extensive experiments demonstrate that MT-Cal achieves outstanding and consistent performance in multi-turn calibration, and ConfChat preserves and even enhances model performance in multi-turn interactions. Our results mark multi-turn calibration as one missing link for scaling LLM calibration toward safe, reliable, and real-world use.

Figures & tables

Figure 1: LLMs are prone to change their responses with confidence when challenged. The figure in the bottom left is the reliability diagram for confidence at the first turn. The figure in the bottom right is the reliability diagram for confidence at the second turn 2 2 2 The experiment is conducted with Llama3.1-8B-instruct on TriviaQA dataset Joshi et al. (2017) . We use the length-normalized likelihood of generated sequences as the confidence measure. .

Figure 2: (a) Changes in the reliability diagram from the initial response to the subsequent reply after receiving critical follow-up messages for Llama3.1-8B-Instruct. The diagrams above the arrows correspond to the first turn, while those below represent the second turn. (b) The analysis of answer changes with the change of model confidence. Correct $\to$ Correct: The answer remains correct. Correct $\to$ Incorrect: The correct answer is changed to an incorrect one. Incorrect $\to$ Correct: The incorrect answer is revised to a correct one. Incorrect $\to$ Correct: The answer still remains incorrect. A large portion of second turn responses with increased confidence comes with negative flips, i.e.,Correct $\to$ Incorrect.

Figure 3: The framework of ConfChat process. (a) In the first turn, the token with the highest rescaled generation score is selected at each decoding step. (b) In subsequent turns, candidate token sets are generated based on both the first-turn and current-turn inputs, and the two candidate sets are aggregated to select the token with the highest overall generation score.

	TriviaQA					SciQ					NQ
Method	ECE@1	ECE@2	ECE@D	Brier	smECE	ECE@1	ECE@2	ECE@D	Brier	smECE	ECE@1	ECE@2	ECE@D	Brier	smECE
Llama3.1-8B-Instruct
SL	6.43	26.41	9.36	22.17	9.27	7.32	21.56	12.05	23.43	10.51	24.12	34.97	27.54	31.31	24.41
PS	12.93	26.44	11.46	23.16	7.27	10.03	13.36	12.50	21.47	12.67	20.65	33.81	25.27	29.43	24.22
SC	19.77	28.10	25.52	28.88	20.67	32.98	30.10	38.71	39.10	29.38	20.94	21.79	22.82	28.64	17.29
Verbal	21.96	41.72	25.89	27.79	22.94	15.12	32.09	20.30	23.35	19.69	41.67	51.12	43.15	41.57	36.28

Table 1: The performance comparison of multi-turn calibration for different methods. The best results are bolded . All the results are reported in percentage(%). We report ECE@1 and ECE@2 in the table, while the ECE@T for subsequent conversation turns is presented in Fig. 8 .

Figure 4: Domain generalization on Llama-8B-Instruct. OOD denotes the out-of-domain setting, IND denotes the in-domain setting, and PS refers to Platt Scaling.

Figure 5: The comparison of change in accuracy of Llama3.1-8B-Instruct in different conversation rounds between ConfChat and other strategies. Our method ConfChat keeps a relatively stable accuracy across turns.

Method	ECE@1	ECE@2	ECE@3	ECE@4	ECE@5
Llama3.1-8B-Instruct
SL	7.56	8.13	8.25	7.00	9.42
Apricot	6.03	5.18	7.12	7.33	6.26
MTCal	5.04	2.31	5.91	5.01	6.08
Qwen2.5-7B-Instruct
SL	23.58	24.65	23.20	20.38	22.89

Table 2: The comparison between single-turn calibration methods and MTCal. All the results are reported in percentage(%). The best results are bolded 4 4 4 We calculate the ECE@T in the single-turn calibration setting on question-answer pair set consisting of initial query

u_{1}

and model response

r_{T}

T

-th turn. .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Follow-up Messages
Prompting Doubt	Are you certain about that?
Emotional Appeal	I trusted you, but this seems misleading. Are you sure?
Bandwagon Effect	Most people would disagree with this. Are you sure your answer is right?
Directive Prompt	I don’t think that’s correct. Please reconsider before answering.
Aggressive Appeal	That is a stupid response! Think before you answer.
Mild Disagreement	That doesn’t sound right to me.

Appendix

Table 3: Critical follow-up messages.

Figure 6: Domain generalization on Qwen2.5-7B-Instruct. OOD denotes the out-of-domain setting, IND denotes the in-domain setting, and PS refers to Platt Scaling.

Figure 7: Domain generalization on Gemma2-9B-it. OOD denotes the out-of-domain setting, IND denotes the in-domain setting, and PS refers to Platt Scaling.

Figure 8: The change of ECE@T during conversation for different confidence estimations. Our method MTCal consistently outperforms the comparison methods across turns.

Figure 9: The comparison of change in response accuracy of Qwen2.5-7B-Instruct in different conversation rounds between ConfChat and other strategies. Our method ConfChat keeps a relatively stable accuracy across turns.

Figure 10: The comparison of change in response accuracy of Gemma2-9B-it in different conversation rounds between ConfChat and other strategies. Our method ConfChat keeps a relatively stable accuracy across turns.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 229 papers

Self-Anchoring Calibration Drift in Large Language Models: How Multi-Turn Conversations Reshape Model Confidence
Mar 1, 2026HarshavardhanConfidence Estimation in Language Models Language Model Calibration
Independent Researcher
Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability
Mar 12, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5LLM Reliability Multi-Turn Dialogue Evaluation
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Quantifying Conversational Reliability of Large Language Models under Multi-Turn Interaction
Mar 2, 2026Jiyoon MyungTool Selection for Language Model Agents LLM Reliability
Samsung SDS

Mar 1, 2026J/K move · Enter open · S save

Self-Anchoring Calibration Drift in Large Language Models: How Multi-Turn Conversations Reshape Model Confidence

Harshavardhan

Independent Researcher

Confidence Estimation in Language Models Language Model Calibration

Open paper AskPDF arXiv

We introduce Self-Anchoring Calibration Drift (SACD), a hypothesized tendency for large language models (LLMs) to show systematic changes in expressed confidence when building iteratively on their own prior outputs across multi-turn conversations. We report an empirical study comparing three frontier models -- Claude Sonnet 4.6, Gemini 3.1 Pro, and GPT-5.2 -- across 150 questions spanning factual, technical, and open-ended domains, using three conditions: single-turn baseline (A), multi-turn self-anchoring (B), and independent repetition control (C). Results reveal a complex, model-heterogeneous pattern that partially diverges from pre-registered hypotheses. Claude Sonnet 4.6 exhibited significant decreasing confidence under self-anchoring (mean CDS = -0.032, t(14) = -2.43, p = .029, d = -0.627), while also showing significant calibration error drift (F(4,56) = 22.77, p < .001, eta^2 = .791). GPT-5.2 showed the opposite pattern in open-ended domains (mean CDS = +0.026) with significant ECE escalation by Turn 5. Gemini 3.1 Pro showed no significant CDS (t(14) = 0.38, p = .710), but its Condition C data reveals a striking ECE pattern: without self-anchoring, Gemini's calibration error drops from .327 to near zero across repetitions, whereas self-anchoring holds ECE flat at approximately .333 -- indicating that SACD can manifest as suppression of natural calibration improvement rather than ac

Abstract

Confidence Estimation in Language Models

Explore similar work

Self-Anchoring Calibration Drift in Large Language Models: How Multi-Turn Conversations Reshape Model Confidence

Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability

Quantifying Conversational Reliability of Large Language Models under Multi-Turn Interaction