TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

Victoria Graf; Valentina Pyatkin; Nouha Dziri; Nathan Lambert; Hannaneh Hajishirzi

cs.CLMar 17, 2026

TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

Authors: Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi

Organizations: University of Washington · Allen Institute for AI

Abstract

Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.

Figures & tables

Figure 1: TurnWiseEval -Absolute example. Model responses to a multi-turn conversation are compared to a strong model’s response on an equivalent single-turn prompt. Our evaluation metric is the win-rate of the evaluated model as determined by an LM judge.

Model	TW-Absolute	AlpacaEval2	$\Delta$	TW-Self
Olmo 3 7B	36.8	42.2	$-$ 5.4	38.9
Olmo 3.1 32B	52.4	60.1	$-$ 7.7	34.0
Qwen 3 8B	48.9	49.7	$-$ 0.8	53.2
Qwen 3 32B	67.7	69.6	$-$ 1.9	48.5
Llama 3.1 8B	18.7	16.3	+2.4	40.2
Llama 3.1 70B	26.3	27.7	$-$ 1.4	36.5

Table 1: Performance of chat models on TurnWiseEval . Olmo 3 refers to Olmo 3 Instruct models; Qwen 3 results are without thinking enabled.

\Delta

is the difference between TW-Absolute and AlpacaEval scores.

Figure 2: An example of using TurnWiseData to create a synthetic conversation. User turns are independently generated from the seed prompt (green) and conversations are appended with the seed prompt and response (blue).

Training Data	TW-Absolute	AlpacaEval2	$\Delta$	TW-Self	IFEval	MMLU
SFT (original)	10.0	23.9	$-$ 13.9	30.1	81.3	66.7
+ 10k TurnWiseData conversations	24.1	25.1	$-$ 1.0	42.9	76.7	61.4
+ 10k TurnWiseData and self-talk	20.7	25.6	$-$ 4.9	41.9	72.8	63.2
+ 20k TurnWiseData and self-talk	22.2	24.8	$-$ 2.6	42.7	75.6	62.7
DPO (no multi-turn)	37.4	42.8	$-$ 5.4	35.0	82.6	68.0
+ 10k TurnWiseData conversations	39.1	43.8	$-$ 4.7	41.2	81.5	68.3

Table 2: Performance of models trained with TurnWiseData conversations. All training is done on Olmo 3 7B Instruct SFT models. The SFT (original) line represents the base SFT model and each additional line represents fine-tuning with different additional multi-turn data. The DPO (no multi-turn) line represents the SFT model preference tuned with only single-turn Olmo 3 preference data; each additional line represents adding different additional data to the preference mixture.

\Delta

= TurnWiseEval -Absolute

-

AlpacaEval.

Figure 3: Win rate of models on TurnWiseEval -Absolute decays across conversation turns. One-turn conversations are represented by AlpacaEval.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The judge template for pairwise comparison.

Panel of Figure 5 — Figure 5: An example conversation from TurnWiseEval .

Model	TW-A	AE2	$\Delta$	TW-S	IFEval	MMLU
SFT (original)	10.0	23.9	$-$ 13.9	30.1	81.3	66.7
+ max 1 turn	16.9	25.5	$-$ 8.6	40.2	73.0	59.7
+ max 2 turn	20.7	22.7	$-$ 2.0	43.5	76.3	62.2
+ max 4 turn	21.3	25.3	$-$ 4.0	41.8	75.6	63.1
+ max 8 turn	20.7	25.6	$-$ 4.9	41.9	72.8	63.2
DPO (no MT)	37.4	42.8	$-$ 5.4	35.0	82.6	68.0

Appendix

Table 3: Ablations of long conversations in multi-turn training data. + max

n

turns indicates truncating the additional multi-turn data to

n

user turns.

Model	TW-A	TW-A (Weak)	TW-S	TW-S (Weak)
SFT (original)	10.0	9.4	30.1	28.4
+ MT (10k)	20.7	20.4	41.9	40.9
+ MT (20k)	22.2	23.3	42.7	43.0
DPO (no MT)	37.4	35.7	35.0	33.0
+ MT (10k)	38.7	40.3	40.9	39.4
+ MT (20k)	43.4	44.3	44.2	42.7

Appendix

Table 4: Performance of models trained with synthetic multi-turn data on TurnWiseEval with weak context. TW-A (Weak) and TW-S (Weak) indicate alternative versions of TurnWiseEval that use GPT-3.5 Turbo generated assistant turns in context. + MT indicates training with additional multi-turn data ( TurnWiseData and self-talk).

Multi-Turn Dialogue Evaluation

Multi-turn dialogue evaluation measures whether AI systems remain accurate, coherent, safe, and useful as conversations evolve.

See all 194 papers

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding
Jul 31, 2026Eileen Ye, Jiawen Tao, Yaoming Li +7Conversational Memory LLM Evaluation
Hunyuan Team, Tencent · Peking University
Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
Aug 18, 2026Syeda Faiza Ahmed Sara, Zien Sheikh Ali, Hunzalah Hassan Bhatti +2Conversational Agents Multimodal Agents
Qatar Computing Research Institute, HBKU, Qatar

May 23, 2026J/K move · Enter open · S save

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Tianlang Chen, Shirley Wu, Jure Leskovec

Stanford

Language Model Distillation Teacher-Student Learning Language Model Self-Improvement LLM Training Multi-Turn Dialogue Management

Open paper AskPDF arXiv

Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.

Abstract

Multi-Turn Dialogue Evaluation

Explore similar work

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding

Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges