MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation

Jyotika Singh; Fang Tu; Miguel Ballesteros; Weiyi Sun; Sandip Ghoshal; Michelle Yuan; Yassine Benajiba; Sujith Ravi

cs.CLApr 9, 2026

MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation

Authors: Jyotika Singh, Fang Tu, Miguel Ballesteros, Weiyi Sun, Sandip Ghoshal, Michelle Yuan, Yassine Benajiba, Sujith Ravi,

Organizations: Oracle AI

Abstract

Large language models (LLMs) suffer significant performance degradation when user instructions and context are distributed over multiple conversational turns, yet multi-turn (MT) interactions dominate chat interfaces. The routine approach of appending full chat history to prompts rapidly exhausts context windows, leading to increased latency, higher computational costs, and diminishing returns as conversations extend. We introduce MT-OSC, a One-off Sequential Condensation framework that efficiently and automatically condenses chat history in the background without disrupting the user experience. MT-OSC employs a Condenser Agent that uses a few-shot inference-based Condenser and a lightweight Decider to selectively retain essential information, reducing token counts by up to 72% in 10-turn dialogues. Evaluated across 13 state-of-the-art LLMs and diverse multi-turn benchmarks, MT-OSC consistently narrows the multi-turn performance gap - yielding improved or preserved accuracy across datasets while remaining robust to distractors and irrelevant turns. Our results establish MT-OSC as a scalable solution for multi-turn chats, enabling richer context within constrained input spaces, reducing latency and operational cost, while balancing performance.

Figures & tables

Figure 1: MT-OSC framework with one-off sequential condensation of chat history (w turns/exchanges at a time). This figure illustrates the chat flow when the Condenser Agent initiates condensation of prior history.

Figure 2: Examples of Multi-Turn Chat Condensing with Reasoning (original chat in yellow; condensed chat below.)

Figure 3: Comparison of performance aggregated across various datasets on simple summarization (Summ) using our one-off sequential application approach and our MT-OSC approach with window sizes 2, 3, 4.

	Performance		Chat history tokens
	MT-baseline	MT-OSC*	MT-baseline	MT-OSC*
BFCL shrd	81.13%	86.79 %	165	124
GSM shrd	83.45%	84.80 %	1260	354
HEval shrd	74.67%	77.33 %	539	272
Spider shrd	76.95%	79.44 %	105	104
ToTTo shrd	0.19	0.19	2585	2585

Table 1: Average performance and total chat history tokens (per turn) comparing MT-baseline and MT-OSC (w=4) across datasets. Metrics: accuracy (BFCL, GSM, HEval, Spider, recoll), 10-point rating (refine, follow, expan), BLEU (ToTTo), and joint score (SoH).

Figure 4: Chat history tokens growth in MT-baseline compared to using Condenser ( $w=4$ ) showing reduction in token consumption in the user-facing chat.

Figure 5: Model performance scores for various Chat Models for conversations with 6 or more turns across baseline multi-turn and MT-OSC with window size $w=4$ . See Table 5 for broken down details in tabular form.

Figure 6: Total tokens consumed by chat history in the main chat + background condenser, showing overall cost benefit and overhead balancing as the chat grows.

	Performance		Chat Hist Tokens
Dataset	w/ D	w/o D	w/ D	w/o D
ToTTo shrd	0.18	0.09	2585	1888
SoH shrd	0.13	0.08	7495	4086
Refin mtev	5.35	5.18	1342	593

Table 2: Performance and chat history tokens with and without Decider (D) across the datasets where Decider induces a performance or token count difference.

Figure 7: Comparison of performance between MT and MT-OSC, and reduction in chat history tokens, on the three different variants created for robustness testing (1. Repetition Infusion (RI); 2. Filler Injection (FI); 3. Contextual Diversion (CD)).

Figure 8: Performance of MT-baseline and MT-OSC under Contextual Diversion (CD) variants varied by number of distraction turns added.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Histogram showing the number of samples for each maximum number of turns per sample in sharded + MTEval datasets.

Dataset	Source	Success Metric	Size	Split (Dev, Val, Test)
Sharded BFCL	Laban et al. (2025)	Accuracy [Exact match (Action matching)]	105	(30.5%, 19%, 50.5%)
Sharded GSM8K	Laban et al. (2025)	Accuracy [Exact match (Final Answer)]	103	(-, 36%, 64%)
Sharded HumanEval	Laban et al. (2025)	Accuracy [Exact match (Code test passing)]	45	(-, -, 100%)
Sharded Spider	Laban et al. (2025)	Accuracy [Exact match (SQL execution output match)]	107	(-, -, 100%)
Sharded ToTTo	Laban et al. (2025)	BLEU Score	120	(15%, 12.5%, 72.5%)
Sharded Summary of Haystack	Laban et al. (2025)	LLMaaJ ("Joint Score") to measure info coverage & attribution F1	94	(-, -, 100%)

Appendix

Table 3: Dataset name, source, success metric for evaluation, size, and split. The sharded datasets were synthetically generated and manual inspection and validation by the authors of their respective sources cited above. The MT-EVAL datasets also went through human-in-the-loop creation, validation, and filtering process where the initial data generation utilized GPT-4.

Figure 10: These examples shows the original turns from a conversation (in darker color in each example) and the perturbed version with new random turns (shown in the lighter color in each example) added to the conversation.

Figure 11: Comparison of performance across datasets of simple summarization (Summ) using the one-off sequential application approach (window sizes 2, 3, and 4) and the MT-OSC (window sizes 2, 3, and 4). Chat model is Llama-3.3-70B-Instruct.

	Llama-3.3-70B	GPT 4.1
MT-baseline	81.13%	61.54%
MT-OSC*	86.79%	69.23%
FIFO	80.77% $\downarrow$	51.92% $\downarrow$

Appendix

Table 4: FIFO compared to MT-baseline and MT-OSC for managing chat history shows that FIFO performs worse than the baseline (MT-baseline) approach on the Sharded BFCL dataset. *=ours

	GSM	BFCL	HEval	Recoll	Spider	Expan	Refin	Avg
	shrd	shrd	shrd	mte+	shrd *	mte+	mte	acc%
Metric	accuracy %					rating out of 10
GROK3
MT	0.82	0.67	0.51	0.96	0.5	9.01	6.26	0.75
MTOSC	0.86	0.71	0.62	0.95	0.5	8.38	6.22	0.79
GROK4

Appendix

Table 5: Results across datasets and chat models for samples with 6 turns or more using MT-baseline and MT-OSC. (*Spider has a very small sample size of 6 for >=6 turns. Average accuracy per LLM doesn’t include Spider due to this reason.)

BFCL shrd
	Llama4-mav	GPT4.1	GPT4o
Condenser
N/A (MT-baseline)	83.61	67.44	55.74
Llama-3.3-70B	90.16 $\uparrow$	80.33 $\uparrow$	68.85 $\uparrow$
Llama-4-Mav	83.61	72.13 $\uparrow$	68.85 $\uparrow$
GPT 4.1	86.89 $\uparrow$	75.41 $\uparrow$	65.57 $\uparrow$

Appendix

Table 6: Across Condenser Models, performance in (%) of MT-OSC compared to MT-baseline for data samples with at least 6 turns or more in sharded BFCL and HumanEval sets.

	1. Repetition Infusion		2. Filler Injection		3. Contextual Diversion
	MT	MT-OSC	MT	MT-OSC	MT	MT-OSC
GSM shrd	85.80	86.63	84.77	87.86	82.92	84.77
BFCL shrd	63.45	70.64	64.02	69.89	59.77	67.24
HumanEval shrd	70.81	75.89	74.40	79.17	66.67	79.63
Spider shrd	67.68	70.20	63.13	66.67	57.07	61.62
refinement mte	5.67	5.66	5.7	5.64	5.01	5.00

Appendix

Table 7: Comparison of performance between MT and MT-OSC on the three different variants created for robustness testing (1. Repetition Infusion; 2. Filler Injection; 3. Contextual Diversion).

Figure 12: Percentage reduction in chat history tokens in MT-OSC over MT-baseline across datasets with and without Decider component.

Figure 13: Sensitivity of the decider to $\gamma$ and $\tau$ across datasets. Each panel corresponds to one dataset. Points show, for each ( $\gamma$ , $\tau$ ) setting, the end-to-end task success score (y-axis; reported in the dataset’s native units) versus the number of samples that were routed to compression (x-axis). Color indicates $\gamma$ and marker shape indicates $\tau$ . Flat panels indicate insensitivity to ( $\gamma$ , $\tau$ ), while sloped panels highlight datasets where increased compression changes downstream success.

Figure 14: Example showing MT-baseline vs MT-OSC, where MT-OSC handles incorrect assumptions made by the model earlier in a chat.

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 250 papers

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability
Mar 12, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5LLM Reliability Multi-Turn Dialogue Evaluation
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Improving Multi-turn Dialogue Consistency with Self-Recall Thinking
May 14, 2026Renning Pang, Tian Lan, Leyuan Liu +3Memory-Augmented Language Models Long-Term Conversational Memory
University of Electronic Science and Technology of China

May 23, 2026J/K move · Enter open · S save

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Tianlang Chen, Shirley Wu, Jure Leskovec

Stanford

Language Model Distillation Teacher-Student Learning Language Model Self-Improvement LLM Training Multi-Turn Dialogue Management

Open paper AskPDF arXiv

Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.

Abstract

LLM Context Management

Explore similar work

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability

Improving Multi-turn Dialogue Consistency with Self-Recall Thinking