cs.CLMar 3, 2026

Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems

Authors: Raad Khraishi, Iman Zafar, Katie Myles, Greig A Cowan

Organizations: NatWest AI Research · University College London

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deployed multi-turn LLM systems routinely switch models mid-interaction due to upgrades, cross-provider routing, and fallbacks. Such handoffs create a context mismatch: the model generating later turns must condition on a dialogue prefix authored by a different model, potentially inducing silent performance drift. We introduce a switch-matrix benchmark that measures this effect by running a prefix model for early turns and a suffix model for the final turn, and comparing against the no-switch baseline using paired episode-level bootstrap confidence intervals. Across CoQA conversational QA and Multi-IF benchmarks, even a single-turn handoff yields prevalent and statistically significant, directional effects and may swing outcomes by -8 to +13 percentage points in Multi-IF strict success rate and +/- 4 absolute F1 on CoQA, comparable to the no-switch gap between common model tiers (e.g., GPT-5-nano vs GPT-5-mini). We further find systematic compatibility patterns: some suffix models degrade under nearly any non-self dialogue history, while others improve under nearly any foreign prefix. To enable compressed handoff risk monitoring, we decompose switch-induced drift into per-model prefix influence and suffix susceptibility terms, accounting for ~70% of variance across benchmarks. These results position handoff robustness as an operational reliability dimension that single-model benchmarks miss, motivating explicit monitoring and handoff-aware mitigation in multi-turn systems.

Figures & tables

 gpt-5-nano-2025-08-07gpt-5-mini-2025-08-07gpt-5.2-2025-12-11gemini-3-flash-previewgemini-2.5-flashdeepseek-v3.2qwen-2.5-72b-instructclaude-haiku-4.5claude-sonnet-4.5
gpt-5-nano-2025-08-070.000 ***−- 0.005 ***0.000 ***−- 0.021 ***−- 0.020 ** *−- 0.045 ***0.002 ***0.013 ***−- 0.008 ***
gpt-5-mini-2025-08-070.013 ***0.000 ***0.007 ***0.002 ***0.016 ***−- 0.007 ***0.013 ***0.015 ***−- 0.002 ***
gpt-5.2-2025-12-110.011 ***0.004 ***0.000 ***−- 0.006 ***0.008 ***−- 0.016 ** *0.027 ** *0.017 ***−- 0.003 ***
gemini-3-flash-preview0.042 ** *−- 0.004 ***0.005 ***0.000 ***0.013 ** *−- 0.005 ***0.030 ***0.028 ** *0.007 ***
gemini-2.5-flash−- 0.010 ***0.007 ***0.008 ***−- 0.015 * **0.000 ***−- 0.016 ***0.029 ***0.030 ***−- 0.002 ***
deepseek-v3.20.006 ***−- 0.000 ***0.008 ***−- 0.006 ***0.005 ***0.000 ***0.025 * **0.026 ** *0.004 ***
Table 1: CoQA switch effect relative to the no-switch baseline. Each cell reports ΔA→B\Delta_{A\rightarrow B} , where rows represent prefix models AA and columns are suffix models BB . Stars indicate bootstrapped CI excludes 0 at 90% ( ∗ ), 95% ( ∗∗ ), or 99% ( ∗∗∗ ) confidence.
 gpt-5-nano-2025-08-07gpt-5-mini-2025-08-07gpt-5.2-2025-12-11gemini-3-flash-previewgemini-2.5-flashdeepseek-v3.2qwen-2.5-72b-instructclaude-haiku-4.5claude-sonnet-4.5
gpt-5-nano-2025-08-070.000 ***−- 0.015 ***−- 0.025 ***−- 0.010 ***0.045 ***0.000 ***−- 0.010 ***−- 0.076 ** *−- 0.035 ***
gpt-5-mini-2025-08-070.096 ***0.000 ***−- 0.045 * **−- 0.061 ***0.051 * **0.005 ***0.000 ***−- 0.045 ***−- 0.045 ***
gpt-5.2-2025-12-110.106 ***0.051 ** *0.000 ***0.005 ***0.081 ***0.020 ***0.071 ***−- 0.015 ***−- 0.010 ***
gemini-3-flash-preview0.086 ** *−- 0.005 ***−- 0.005 ***0.000 ***0.086 ***0.040 * **0.061 * **0.015 ***−- 0.005 ***
gemini-2.5-flash0.061 * **0.061 ** *0.020 ***0.010 ***0.000 ***0.015 ***0.051 ***−- 0.015 ***−- 0.035 ***
deepseek-v3.20.106 ***0.015 ***−- 0.010 ***−- 0.005 ***0.071 ***0.000 ***0.056 * **0.000 ***−- 0.035 ***
Table 2: Multi-IF switch effect relative to the no-switch baseline. Each cell reports ΔA→B\Delta_{A\rightarrow B} , where rows represent prefix models AA and columns are suffix models BB . Stars indicate bootstrapped CI excludes 0 at 90% ( ∗ ), 95% ( ∗∗ ), or 99% ( ∗∗∗ ) confidence.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

 gpt-5-nano-2025-08-07gpt-5-mini-2025-08-07gpt-5.2-2025-12-11gemini-3-flash-previewgemini-2.5-flashdeepseek-v3.2qwen-2.5-72b-instructclaude-haiku-4.5claude-sonnet-4.5
gpt-5-nano-2025-08-070.689 [0.636,0.738]0.751 [0.694,0.796]0.751 [0.700,0.801]0.765 [0.713,0.812]0.758 [0.709,0.802]0.730 [0.672,0.775]0.726 [0.674,0.773]0.737 [0.685,0.787]0.776 [0.725,0.821]
gpt-5-mini-2025-08-070.703 [0.645,0.754]0.757 [0.711,0.802]0.757 [0.704,0.803]0.788 [0.726,0.831]0.793 [0.743,0.834]0.768 [0.717,0.810]0.737 [0.680,0.786]0.739 [0.686,0.782]0.783 [0.734,0.833]
gpt-5.2-2025-12-110.700 [0.645,0.757]0.760 [0.704,0.808]0.750 [0.697,0.799]0.780 [0.731,0.827]0.786 [0.735,0.832]0.758 [0.703,0.805]0.751 [0.697,0.798]0.740 [0.692,0.787]0.781 [0.731,0.822]
gemini-3-flash-preview0.731 [0.675,0.779]0.753 [0.705,0.802]0.755 [0.709,0.804]0.786 [0.732,0.829]0.790 [0.740,0.834]0.769 [0.717,0.815]0.754 [0.702,0.800]0.752 [0.703,0.798]0.791 [0.744,0.839]
gemini-2.5-flash0.679 [0.620,0.735]0.763 [0.713,0.808]0.759 [0.709,0.803]0.771 [0.723,0.825]0.778 [0.732,0.822]0.758 [0.705,0.802]0.753 [0.696,0.798]0.754 [0.704,0.799]0.782 [0.733,0.826]
deepseek-v3.20.695 [0.636,0.747]0.756 [0.704,0.803]0.758 [0.710,0.803]0.781 [0.728,0.825]0.783 [0.738,0.825]0.774 [0.728,0.820]0.749 [0.696,0.797]0.750 [0.703,0.796]0.789 [0.742,0.833]
Appendix
Table 3: CoQA absolute performance by switch cell (mean with 95% CI). Each entry reports mean last-turn F1 with a 95% BCa bootstrap CI for the switch cell (A ⁣→ ⁣B)(A\!\rightarrow\!B) on CoQA under a final-turn handoff. Rows are prefix models AA and columns are suffix models BB ; the diagonal corresponds to the no-switch condition (B ⁣→ ⁣B)(B\!\rightarrow\!B) .
 gpt-5-nano-2025-08-07gpt-5-mini-2025-08-07gpt-5.2-2025-12-11gemini-3-flash-previewgemini-2.5-flashdeepseek-v3.2qwen-2.5-72b-instructclaude-haiku-4.5claude-sonnet-4.5
gpt-5-nano-2025-08-070.333 [0.273,0.399]0.419 [0.354,0.490]0.447 [0.382,0.513]0.490 [0.424,0.562]0.404 [0.338,0.480]0.399 [0.338,0.470]0.333 [0.268,0.394]0.354 [0.283,0.419]0.429 [0.356,0.500]
gpt-5-mini-2025-08-070.429 [0.364,0.500]0.434 [0.364,0.505]0.429 [0.364,0.505]0.439 [0.374,0.510]0.409 [0.338,0.475]0.404 [0.333,0.470]0.343 [0.275,0.409]0.384 [0.318,0.444]0.419 [0.348,0.495]
gpt-5.2-2025-12-110.439 [0.374,0.515]0.485 [0.419,0.556]0.475 [0.408,0.550]0.505 [0.429,0.566]0.439 [0.374,0.510]0.419 [0.354,0.490]0.414 [0.354,0.488]0.414 [0.343,0.485]0.455 [0.384,0.520]
gemini-3-flash-preview0.419 [0.348,0.485]0.429 [0.359,0.500]0.470 [0.404,0.540]0.500 [0.429,0.571]0.444 [0.374,0.510]0.439 [0.374,0.505]0.404 [0.335,0.480]0.444 [0.369,0.510]0.460 [0.391,0.535]
gemini-2.5-flash0.394 [0.328,0.465]0.495 [0.429,0.561]0.495 [0.424,0.566]0.510 [0.444,0.581]0.359 [0.288,0.424]0.414 [0.348,0.485]0.394 [0.326,0.470]0.414 [0.354,0.480]0.429 [0.354,0.490]
deepseek-v3.20.439 [0.369,0.505]0.449 [0.384,0.520]0.465 [0.398,0.530]0.495 [0.424,0.561]0.429 [0.369,0.505]0.399 [0.338,0.480]0.399 [0.335,0.470]0.429 [0.362,0.495]0.429 [0.359,0.495]
Appendix
Table 4: Multi-IF absolute performance by switch cell (mean with 95% CI). Each entry reports mean turn-3 conversation-level strict success rate with a 95% BCa bootstrap CI for the switch cell (A ⁣→ ⁣B)(A\!\rightarrow\!B) on Multi-IF under a final-turn handoff. Rows are prefix models AA and columns are suffix models BB ; the diagonal corresponds to the no-switch condition (B ⁣→ ⁣B)(B\!\rightarrow\!B) .
Figure 1: Observed vs. predicted mean drift under a two-way additive model Δ^A→B=c+αA+βB\hat{\Delta}_{A\to B}=c+\alpha_{A}+\beta_{B} on off-diagonal switch cells. CoQA (left): drift in last-turn F1. Multi-IF (right): drift in turn-3 strict success. The model explains most variance across switch cells ( R2=0.83R^{2}=0.83 CoQA, R2=0.85R^{2}=0.85 Multi-IF), supporting a factorized view of handoff effects into prefix influence and suffix susceptibility.
Figure 2: Estimated prefix influence αA\alpha_{A} (sum-to-zero) and suffix susceptibility βB\beta_{B} (sum-to-zero) from the additive drift model on CoQA. For αA\alpha_{A} , positive values indicate prefixes that tend to improve downstream suffix performance relative to suffix no-switch baselines; negative values indicate harmful prefix regimes. For βB\beta_{B} , positive values indicate suffix models that tend to benefit from foreign prefixes; negative values indicate degradation under context mismatch.
Figure 3: Estimated prefix influence αA\alpha_{A} (sum-to-zero) and suffix susceptibility βB\beta_{B} (sum-to-zero) from the additive drift model on Multi IF. For αA\alpha_{A} , positive values indicate prefixes that tend to improve downstream suffix performance relative to suffix no-switch baselines; negative values indicate harmful prefix regimes. For βB\beta_{B} , positive values indicate suffix models that tend to benefit from foreign prefixes; negative values indicate degradation under context mismatch.
Prefix AASuffix BBObserved ΔA→B\Delta_{A\to B}Pred. Δ^\hat{\Delta}Resid. ϵ\epsilon∣ϵ∣|\epsilon|
gemini-3-flash-previewgpt-5-nano-2025-08-07++ 0.042++ 0.019++ 0.0220.022
gemini-2.5-flashgpt-5-nano-2025-08-07−- 0.010++ 0.010−- 0.0200.020
gemini-3-flash-previewgpt-5-mini-2025-08-07−- 0.004++ 0.010−- 0.0130.013
gpt-5-mini-2025-08-07qwen-2.5-72b-instruct++ 0.013++ 0.024−- 0.0110.011
gpt-5-nano-2025-08-07deepseek-v3.2−- 0.045−- 0.033−- 0.0110.011
gpt-5-nano-2025-08-07gemini-2.5-flash−- 0.020−- 0.008−- 0.0110.011
Appendix
Table 5: Largest pair-specific interaction residuals on CoQA after subtracting the additive prefix/suffix factorization (top 10 by ∣ϵA,B∣|\epsilon_{A,B}| ). Observed drift is the mean switch effect ΔA→B\Delta_{A\to B} in last-turn F1; predicted drift is Δ^A→B=μ+αA+βB\hat{\Delta}_{A\to B}=\mu+\alpha_{A}+\beta_{B} fit on off-diagonal cells.
Prefix AASuffix BBObserved ΔA→B\Delta_{A\to B}Pred. Δ^\hat{\Delta}Resid. ϵ\epsilon∣ϵ∣|\epsilon|
gemini-2.5-flashgpt-5-nano-2025-08-07++ 0.061+0.104−- 0.0440.044
gemini-3-flash-previewgpt-5-mini-2025-08-07−- 0.005++ 0.030−- 0.0350.035
gemini-2.5-flashgpt-5-mini-2025-08-07++ 0.061++ 0.026++ 0.0350.035
claude-haiku-4.5qwen-2.5-72b-instruct++ 0.000++ 0.033−- 0.0330.033
claude-haiku-4.5gpt-5.2-2025-12-11++ 0.035++ 0.005++ 0.0310.031
gpt-5-nano-2025-08-07claude-haiku-4.5−- 0.076−- 0.046−- 0.0300.030
Appendix
Table 6: Largest pair-specific interaction residuals on Multi-IF after subtracting the additive prefix/suffix factorization (top 10 by ∣ϵA,B∣|\epsilon_{A,B}| ). Observed drift is the mean switch effect ΔA→B\Delta_{A\to B} in turn-3 strict success; predicted drift is Δ^A→B=μ+αA+βB\hat{\Delta}_{A\to B}=\mu+\alpha_{A}+\beta_{B} fit on off-diagonal cells.
More in this topic

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

Explore similar work

CardsList

  1. Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability

    Mar 12, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5LLM ReliabilityMulti-Turn Dialogue Evaluation

    Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research

  2. Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs

    May 10, 2026Aditya Sinha, Harald Steck, Vito Ostuni +1LLM EvaluationMulti-Turn Dialogue Evaluation

    Netflix Inc. · Los Gatos, CA, USA

  3. What if LLMs Ate Their Words: Causal History Effects in Multi-Turn Interaction

    Sep 5, 2026Jinnan Li, Zheren Fu, Yue Wang +3LLM Context ManagementCausal Interventions in Language Models

    School of Artificial Intelligence Jilin University Changchun, China · School of Cyber Science and Technology, University of Science and Technology of China Hefei, China · School of Information and Library Science, University of North Carolina at Chapel Hill Chapel Hill, North Carolina, USA