Do LLMs Benefit From Their Own Words?

Jenny Y. Huang; Leshem Choshen; Wei Sun; Omar Khattab; Ramón Fernandez Astudillo; Mehul Damani; Tamara Broderick; Jacob Andreas

cs.CLFeb 27, 2026

Do LLMs Benefit From Their Own Words?

Authors: Jenny Y. Huang, Leshem Choshen, Wei Sun, Omar Khattab, Ramón Fernandez Astudillo, Mehul Damani, Tamara Broderick, Jacob Andreas

Organizations: Department of Electrical Engineering and Computer Science, MIT · MIT–IBM Watson AI Lab · IBM Research

Abstract

In multi-turn conversations, large language models typically condition on the full conversation history: both past user prompts and assistant responses. We revisit this design choice by comparing full-context prompting to four alternative, substantially-reduced context configurations. Analyzing in-the-wild multi-turn conversations across three open reasoning and one state-of-the-art model, we find that response quality is largely preserved under aggressive context filtering: replacing all prior assistant turns with one-sentence summaries or keeping only the most recent user--assistant exchange often matches storing full context in performance while using roughly 8x less context. To understand this result, we observe that a substantial fraction of user turns (36.4%) in multi-turn conversations are self-contained and that many follow-up turns can be addressed by seeing only the immediately preceding user--assistant exchange. Furthermore, we find that when models condition on their own past responses, this can lead to context pollution, a phenomenon in which reasoning errors, hallucinations, or stylistic artifacts propagate across turns. Motivated by these findings, we design a context-filtering approach that selectively omits the assistant-side history. Taken together, these findings suggest moving away from storing full dialogue transcripts and instead retaining only what is relevant.

Figures & tables

Figure 1: Real example of overrelying on irrelevant context. In Turn 2, the user requests UMAP clustering code. In Turn 5, the user says, “use t-SNE instead.” Left: When the previous assistant response remains in context, the model incorrectly carries over the Jaccard metric from UMAP into the t-SNE implementation, introducing a bug. Right: Without the previous response in context, the model generates correct t-SNE code.

Figure 2: Pairwise win rates between Full Context (FC) prompting and three reduced-context configurations across four models (Qwen3-4B, DeepSeek-R1-Distill-Llama-8B, GPT-OSS-20B, and GPT-5.2), evaluated on the Quality and On-Topic dimensions. Top row: FC vs. summarized assistant history (SUM), where each prior assistant response is replaced with a one-sentence summary. Second row: FC vs. Last Turn Only (LT), which retains only the most recent user–assistant round. Third row: FC vs. Assistant Omitted (AO), which retains all user turns but replaces assistant responses with a placeholder. Bottom row: FC vs. Delete Everything (DE), which provides only the current user prompt. Error bars indicate binomial proportion 95% confidence intervals.

Panel of Figure 3 — Figure 3: Three example conversations on WildChat illustrating different types of follow-ups. Left: the user provides feedback that is concrete enough that the model can respond from scratch using the previous user turn and the updated specifications alone. Middle: the follow-ups reference specific parts in past assistant turns, making it necessary to see the referenced assistant turn. Right: the follow-up references a prior user turn without concrete feedback; no assistant history is necessary.

Figure 4: Left: Distribution of conversation rounds referenced by user prompts from round 2 onward, across 150 WildChat conversations. The X-axis represents the number of rounds back from the current round that a given prompt references. The red stem at 0 indicates prompts that are self-contained (no prior rounds referenced). Middle: Mean cumulative context length across conversations on WildChat as a function of conversation round, comparing Full Context, Assistant Omitted, Last Turn Only , and Summarized context. Shaded band shows $\pm$ 2 standard errors. Right: The number of conversations that can be answered using a 5K-token budget as a function of conversation round.

Figure 5: Left: Accuracy (%) on multi-turn GSM8K under Full Context (FC) and Assistant Omitted (AO) context. Right: Confusion matrices showing the number of problems answered correctly across models and context configurations.

	Assistant-response History		Filler History
Category	Count	Conf.	Count	Conf.
Intent Override	39	0.74	0	0.96
Incorrect Code Carryover	30	0.77	0	0.95
Factual Hallucination Carryover	27	0.75	0	0.95
Reasoning Trace Carryover	6	–	–	–
Formula Carryover	1	–	–	–

Table 1: Prevalence of context pollution instances across 729 responses generated by GPT-5.2 on WildChat conversations, under the original history and a history in which assistant-side responses are replaced with length-matched filler text. Confidence values indicate the LLM annotator’s average confidence across instances in that category.

Figure 6: Ratio of adaptive wins to full-context-only wins vs. mean token count (in thousands) for operating points obtained by sweeping the decision rule’s threshold $\tau$ and cost weight $\lambda$ . The green triangle marks a heuristic that omits assistant responses on all turns categorized as “New Asks.”

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Qwen3-4B	DeepSeek-R1	GPT-OSS-20B	GPT-5.2
Sum	$\geq$	$\geq$	$\geq$	$\geq$
LT	$\geq$	$\geq$	$\sim$	$\sim$
AO		$\sim$		$\sim$
DE

Appendix

Table 2: On-Topic: comparison of reduced-context configurations against Full Context (FC), per model.

\geq

indicates that reduced context performs significantly better than or is statistically indistinguishable from FC (95% CIs overlap);

\sim

indicates that the win-rate gap is within 10pp.

Config	Qwen3-4B	DeepSeek-R1	GPT-OSS-20B	GPT-5.2
Sum	$\geq$	$\geq$		$\geq$
LT	$\geq$	$\geq$	$\sim$
AO				$\sim$
DE

Appendix

Table 3: Quality: comparison of reduced-context configurations against Full Context (FC), per model.

\geq

indicates that the reduced context performs significantly better than or is statistically indistinguishable from FC (95% CIs overlap);

\sim

indicates that the win-rate gap is within 10pp.

Figure 8: Pairwise win rates by prompt category (new ask, follow up with feedback, follow up without feedback) for Qwen3-4B (top) and GPT-5.2 (bottom), comparing Full-Context (FC) and Assistant-Omitted (AO) responses. Stars indicate statistically significant differences. Error bars indicate binomial proportion 95% confidence intervals.

Figure 9: Pairwise win rates between Full Context (FC) and three reduced-context configurations across all four models (Qwen3-4B, DeepSeek-R1-Distill-Llama-8B, GPT-OSS-20B, and GPT-5.2), evaluated on the Quality and On-Topic dimensions. Top row: FC vs. Last Turn Only (LT), which retains only the most recent user–assistant round. Middle row: FC vs. Assistant Omitted (AO), which retains all user turns but replaces assistant responses with a placeholder. Bottom row: FC vs. Delete Everything (DE), which provides only the current user prompt. Error bars indicate binomial proportion 95% confidence intervals.

Figure 10: Pairwise win rates between Full Context (FC) and three reduced-context configurations across all four models (Qwen3-4B, DeepSeek-R1-Distill-Llama-8B, GPT-OSS-20B, and GPT-5.2), evaluated on the Quality and On-Topic dimensions. Top row: FC vs. Last Turn Only (LT), which retains only the most recent user–assistant round. Middle row: FC vs. Assistant Omitted (AO), which retains all user turns but replaces assistant responses with a placeholder. Bottom row: FC vs. Delete Everything (DE), which provides only the current user prompt. Error bars indicate binomial proportion 95% confidence intervals.

Figure 11: Pairwise win rates between Full-Context (FC) and Assistant-Omitted (AO) responses when the judge evaluates only final answers. Compare with Fig. 9 , in which the judge receives both thinking traces and final answers.

Model	Fraction of Turns $\geq$ FC	Tokens Saved
GPT-5.2 (self-generated context)	0.916	37.7%
Qwen3-4B (self-generated context)	0.915	49.2%
DeepSeek-R1 (historical context)	0.875	49.4%
GPT-OSS-20B (historical context)	0.927	43.5%

Appendix

Table 4: Fraction of turns meeting or exceeding the full-context (FC) baseline, and the corresponding token savings, across four models. Parentheses labels indicate whether the conversation history was self-generated or drawn from historical logs.

Feature	Coef.	SE	95% CI
PromptPC3	$-0.2003$	$0.1200$	$[-0.4355,\;0.0350]$
HistoryPC17	$-0.1606$	$0.1056$	$[-0.3677,\;0.0464]$
HistoryPC2	$\phantom{-}0.1412$	$0.1320$	$[-0.1176,\;0.4000]$
category_new_ask	$-0.1308$	$0.1098$	$[-0.3460,\;0.0843]$
PromptPC24	$\phantom{-}0.0963$	$0.0941$	$[-0.0881,\;0.2807]$
HistoryPC26	$\phantom{-}0.0920$	$0.0880$	$[-0.0805,\;0.2644]$

Appendix

Table 5: Logistic regression coefficients with 95% confidence intervals. This table displays the 20 features with the largest absolute coefficient magnitudes.

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 248 papers

Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
May 10, 2026Aditya Sinha, Harald Steck, Vito Ostuni +1LLM Evaluation Multi-Turn Dialogue Evaluation
Netflix Inc. · Los Gatos, CA, USA
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Do Personalized LLMs Need the Whole Conversation? Matching the Full History with Only 5% of Its Text via Relevance-Filtered Context
Oct 10, 2026Erika Hayashi, Andrew ShinLLM Context Management Conversational Memory
Keio University

May 10, 2026J/K move · Enter open · S save

Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs

Aditya Sinha, Harald Steck, Vito Ostuni, Matteo Rinaldi

Netflix Inc. · Los Gatos, CA, USA

LLM Evaluation Multi-Turn Dialogue Evaluation LLM Context Management Multi-Turn Dialogue Management

Open paper AskPDF arXiv

Users interacting with Large Language Models (LLMs) in a multi-turn conversation routinely refine their requests or pivot to new topics. LLMs, however, often miss these topic shifts and carry over irrelevant context from previous turns, leading to inaccurate responses. In this paper, we stress-test the multi-turn understanding of LLMs and study the following two sub-tasks: (1) detecting whether the user pivots or refines in the current turn, and (2) shortlisting relevant context from previous turns. To this end, we construct synthetic benchmarks based on real-world datasets from varied domains, as to simulate context shifts of different levels of difficulty. We then evaluate the zero-shot performance of ten LLMs (open-weight, closed-source and reasoning), and demonstrate that only some reasoning and strongly instructed LLMs are accurate in detecting pivots; open-weight LLMs struggle with the task and frequently carry stale context even with explicit cues; and all models suffer from a position bias. Based on the results, we discuss key takeaways for improving long-term robustness in multi-turn capabilities for LLMs.

Abstract

LLM Context Management

Explore similar work

Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Do Personalized LLMs Need the Whole Conversation? Matching the Full History with Only 5% of Its Text via Relevance-Filtered Context