Large language models (LLMs) excel on static benchmarks, but their performance across multi-turn conversations, which better reflect real-world usage, remains understudied. Addressing this gap is critical in high-stakes settings like healthcare, where patients and clinicians are turning to LLM chatbots to address their medical inquiries. Here, we introduce the "stick-or-switch" (SoS) framework, which partitions a question-answer space into multiple sequential presentations to model two safety-centric behaviors: conviction (i.e., sticking to a correct answer selection or abstention against incorrect suggestions) and flexibility (i.e., switching to a correct suggestion when it is introduced). Evaluating 17 LLMs across three clinical benchmarks, we observe a pervasive conversation tax, where partitioning an answer-space into sequential presentations reduces end-to-end accuracy and abstention against incorrect suggestions by an average of up to 30%, reaching 65% in certain models. We also observe blind switching, where models transition an initial abstention to incorrect and correct suggestions at near-identical rates reaching 50%. Finally, we show that increasing model scale mitigates some of these conversational inefficacies while exacerbating others, such as a higher propensity to adopt an incorrect suggestion from an initial abstention. Together our findings demonstrate that the general proficiency captured by static benchmarks do not translate over multi-turn dialogues.
Figures & tables
Figure 1: Stick-or-Switch (SoS) Evaluation Framework. The SoS framework partitions static decision spaces into sequential multi-turn dialogue to measure three distinct behaviors. i) Positive Conviction measures a model’s ability to defend an initially correct answer against the sequential introduction of incorrect distractors. ii) Negative Conviction similarly measures a model’s ability to defend a safe abstention. iii) Flexibility assesses a model’s ability to recognize and adopt the correct answer when it is eventually introduced after an initial abstention.
Figure 2: Simplifying the Decision Space. Accuracy improves across all models and datasets when narrowing the original multi-choice answer space to a binary decision between the correct answer and a single distractor.
Figure 3: Positive conviction across multi-turn conversation. The cumulative survival rate (Ct) of an initially correct answer over t successive turns of incorrect suggestions is lower than baseline accuracy under a single-shot presentation, which we define as the conversation tax . See Table A.1 in the Appendix for all models and datasets.
Figure 4: The Abstention Penalty. Narrowing the decision space improves initial abstention rates, but performance remains highly variable across model scales—with several architectures performing worse than random chance, contrasting with the consistent scaling benefits observed in accuracy (Figure 2 ).
Figure 5: Negative conviction across multi-turn conversation. The cumulative survival rate ( Ct ) of an initially correct abstention over t successive turns of incorrect suggestions is lower than baseline single-shot abstention rates. See Table A.2 in the Appendix for all models and datasets.
Figure 6: Flexibility and Blind Switching (MedQA). Correct versus incorrect switch rates after an initial abstention. GPT-5.2 approaches ideal flexibility toward the bottom right quadrant, while the other models trend toward the blind switching diagonal ( y=x ) where models adopt correct and incorrect suggestions at near-identical rates. See Table A.3 for all models and datasets
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
MedQA
JAMA CC
MedMCQA
SS
MT
SS
MT
SS
MT
GPT-5.2
90.5
89.8 ( -0.8 )
78.5
76.2 ( -2.3 )
84.0
81.5 ( -2.5 )
GPT-4o
85.5
80.0 ( -5.5 )
69.2
52.5 ( -16.8 )
81.0
70.8 ( -10.3 )
Qwen-2.5 (72B)
72.4
70.8 ( -1.7 )
61.3
60.2 ( -1.0 )
68.8
63.6 ( -5.2 )
Llama-3.1 (70B)
73.6
56.3 ( -17.2 )
67.2
37.8 ( -29.4 )
73.8
56.7 ( -17.2 )
Qwen-3 (32B)
70.2
5.9 ( -64.2 )
60.7
12.8 ( -47.8 )
70.3
13.4 ( -56.9 )
Appendix
Table A.1: Positive Conviction Rates. Single-shot accuracy (SS) versus multi-turn positive conviction (MT). Values are in percentages (%). Parentheses show the percentage point change in conviction compared to the single-shot baseline.
Model
MedQA
JAMA CC
MedMCQA
SS
MT
SS
MT
SS
MT
GPT-5.2
90.5
69.5 ( -21.0 )
78.5
49.0 ( -29.5 )
84.0
48.8 ( -35.2 )
GPT-4o
85.5
41.0 ( -44.5 )
69.2
19.2 ( -50.0 )
81.0
30.0 ( -51.0 )
Qwen-2.5 (72B)
72.4
33.8 ( -38.6 )
61.3
19.2 ( -42.0 )
68.8
45.7 ( -23.2 )
Llama-3.1 (70B)
73.6
32.7 ( -40.9 )
67.2
25.8 ( -41.5 )
73.8
29.3 ( -44.5 )
Qwen-3 (32B)
70.2
6.1 ( -64.1 )
60.7
12.8 ( -47.8 )
70.3
12.1 ( -58.2 )
Appendix
Table A.2: Negative Conviction Rates. Single-shot accuracy (SS) versus multi-turn negative conviction (MT). Values are in percentages (%). Parentheses show the percentage point change in conviction compared to the single-shot baseline.
Model
MedQA
JAMA CC
MedMCQA
−
+
−
+
−
+
GPT-5.2
4.2
93.1 ( +88.9 )
20.1
92.9 ( +72.8 )
21.4
88.8 ( +67.4 )
GPT-4o
15.4
92.0 ( +76.6 )
53.0
94.4 ( +41.4 )
37.0
90.8 ( +53.8 )
Qwen-2.5 (72B)
28.7
85.0 ( +56.2 )
46.9
88.2 ( +41.3 )
22.4
72.3 ( +49.9 )
Llama-3.1 (70B)
38.5
91.8 ( +53.2 )
59.7
93.4 ( +33.7 )
38.9
86.2 ( +47.3 )
Qwen-3 (32B)
49.2
47.7 ( -1.5 )
47.2
47.2 ( +0.0 )
47.7
52.8 ( +5.1 )
Appendix
Table A.3: Flexibility Analysis. The invalid switch rate ( − ) represents the precentage of incorrect suggestions which the model adopted. ( + ) represents the percentage of correct suggestions which a model adopted. Values are in percentages (%). Parentheses denote the margin (Valid − Invalid). Almost all models across all datasets demonstrate an invalid switch rate of at least 10%, which averages 30–40% depending on the dataset.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
Large Language Models (LLMs) are increasingly deployed in real-world applications where users engage in extended, mixed-topic conversations that depend on prior context. Yet, their reliability under realistic multi-turn interactions remains poorly understood. We conduct a systematic evaluation of conversational reliability through three representative tasks that reflect practical interaction challenges: (1) maintaining global constraints across topic shifts, (2) selecting the correct tool or agent amid interleaved intents, and (3) tracking structured entities under revisions and distractions. Each task pairs single-turn and multi-turn settings, allowing us to quantify reliability degradation under extended dialogue. Across both commercial and open-source models, we observe substantial declines in reliability, particularly for smaller models. Error analyses reveal recurring failure modes such as instruction drift, intent confusion, and contextual overwriting, which compromise dependable behavior in operational systems. Our findings highlight the need for stress-testing LLMs for conversational reliability and developing more robust evaluation methods for trustworthy deployment.
Users interacting with Large Language Models (LLMs) in a multi-turn conversation routinely refine their requests or pivot to new topics. LLMs, however, often miss these topic shifts and carry over irrelevant context from previous turns, leading to inaccurate responses. In this paper, we stress-test the multi-turn understanding of LLMs and study the following two sub-tasks: (1) detecting whether the user pivots or refines in the current turn, and (2) shortlisting relevant context from previous turns. To this end, we construct synthetic benchmarks based on real-world datasets from varied domains, as to simulate context shifts of different levels of difficulty. We then evaluate the zero-shot performance of ten LLMs (open-weight, closed-source and reasoning), and demonstrate that only some reasoning and strongly instructed LLMs are accurate in detecting pivots; open-weight LLMs struggle with the task and frequently carry stale context even with explicit cues; and all models suffer from a position bias. Based on the results, we discuss key takeaways for improving long-term robustness in multi-turn capabilities for LLMs.
Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.