Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction

Xingwu Chen; Zhanqiu Zhang; Yiwen Guo; Difan Zou

cs.AIMar 5, 2026

Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction

Authors: Xingwu Chen, Zhanqiu Zhang, Yiwen Guo, Difan Zou

Organizations: School of Computing and Data Science, The University of Hong Kong · LIGHTSPEED · Independent Researcher

Abstract

While LLMs demonstrate strong reasoning capabilities when provided with full information in a single turn, they exhibit substantial vulnerability in multi-turn interactions. Specifically, when information is revealed incrementally or requires updates, models frequently fail to integrate new constraints, leading to a collapse in performance compared to their single-turn baselines. We term the root cause as Contextual Inertia: a phenomenon where models rigidly adhere to previous reasoning traces. Even when users explicitly provide corrections or new data in later turns, the model ignores them, preferring to maintain consistency with its previous (incorrect) reasoning path. To address this, we introduce Reinforcement Learning with Single-Turn Anchors (RLSTA), a generalizable training approach designed to stabilize multi-turn interaction across diverse scenarios and domains. RLSTA leverages the model's superior single-turn capabilities as stable internal anchors to provide reward signals. By aligning multi-turn responses with these anchors, RLSTA empowers models to break contextual inertia and self-calibrate their reasoning based on the latest information. Experiments show that RLSTA significantly outperforms standard fine-tuning and abstention-based methods. Notably, our method exhibits strong cross-domain generalization (e.g., math to code) and proves effective even without external verifiers, highlighting its potential for general-domain applications. Code is available at https://github.com/Tencent/RLSTA.

Figures & tables

Figure 1: Contextual Inertia in multi-turn interaction: The persistence of the initial response leads to an unsatisfactory final answer.

Figure 2: Overview of our data preparation and multi-turn simulation pipeline. We partition single-turn prompts into segments to simulate two multi-turn scenarios: MT-Add (incremental information addition) and MT-Refine (correction of initially incorrect conditions).

Figure 3: Distributions of Contextual Inertia Intensity

\texttt{I}_{\mathrm{CI}}(m_{n},m_{n-1})

. We use GPT-4o to categorize the inertia intensity as Weak (1), Moderate (2), and Strong (3). In most cases, the model’s final answer

m_{n}

exhibits strong inertia intensity to the preceding response

m_{n-1}

. Notably, the distribution of this intensity remains indistinguishable regardless of the conversation history quality (high vs. low), providing empirical evidence for the indiscriminate nature of contextual inertia.

Figure 4: Root cause of failures in multi-turn conversations: failures predominantly originate from previous responses ( Misleading Context and Propagated Error ), which are driven by Contextual Inertia.

Figure 5: Single-Turn Anchor Reward ( $R_{s}$ ). We leverage the model’s superior single-turn ability on full instruction ( $i^{\mathrm{full}}$ ) as anchor for the multi-turn final answer $m_{n}$ . Our filtering strategy (Equation 2 ) ensures the anchor is reliable by retaining only histories where single-turn performance exceeds multi-turn.

Method	Qwen2.5-3B-Instruct				Qwen2.5-7B-Instruct				Qwen3-4B-Instruct-2507				Llama-3.2-3B-Instruct
	MT-Add		MT-Refine	Avg	MT-Add		MT-Refine	Avg	MT-Add		MT-Refine	Avg	MT-Add		MT-Refine	Avg
	Math	Code	Math	Avg	Math	Code	Math	Avg	Math	Code	Math	Avg	Math	Code	Math	Avg
Base	0.493	0.220	0.603	0.439	0.638	0.312	0.669	0.540	0.772	0.370	0.814	0.652	0.470	0.140	0.585	0.398
SFT	0.546	0.203	0.533	0.427	0.740	0.310	0.694	0.581	0.826	0.444	0.809	0.693	0.568	0.177	0.519	0.421
DPO	0.546	0.203	0.568	0.439	0.633	0.312	0.522	0.489	0.784	0.416	0.801	0.667	0.559	0.198	0.368	0.375
GRPO	0.636	0.190	0.734	0.520	0.803	0.336	0.836	0.659	0.839	0.472	0.882	0.731	0.608	0.190	0.620	0.473

Table 1: Experimental results on MT-Add and MT-Refine Tasks. We compare RLSTA with SFT, DPO, and vanilla GRPO. Bold indicates best performance, underline indicates improvement over Base, and gray indicates performance lower than Base.

Figure 6: The distributions of contextual inertia intensity after RLSTA. Compared with base model in Figure 3 , RLSTA successfully breaks the contextual inertia, while preserving the ability to utilize beneficial history.

Method	Math	Actions	Database	Code	Avg
Base	0.728	0.365	0.564	0.525	0.546
RLAAR	0.950	0.530	0.873	0.702	0.764
RLSTA	1.001	0.688	0.691	0.596	0.744
RLSTA (+ Abstain)	0.988	0.702	0.762	0.642	0.773

Table 2: Comparison of LiC Scores (

\uparrow

) between RLSTA and RLAAR on MT-Add scenario. Underline indicates improvement over the Base Model.

Method	Code		Math
Method	Sing-turn	Multi-turn	Sing-turn	Multi-turn
Base	0.320 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}142.3}}$	0.223 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}360.2}}$	0.782 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}224.8}}$	0.443 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}319.9}}$
CollabLLM	0.302 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}125.2}}$	0.214 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}{321.5}}}$	0.758 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}250.0}}$	0.433 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}{294.6}}}$
RLSTA	0.343 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}149.4}}$	0.309 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}339.3}}$	0.754 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}230.4}}$	0.654 ${}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}374.0}}$

Table 3: Comparison between RLSTA and CollabLLM on MT-Add scenario, we report model performance (

\uparrow

) and token consumption (

\downarrow

) (Performance

{}_{/\text{\color[rgb]{0.3984,0.8008,0.3984}Tokens}}

)

Model	Method	Multi-turn	Single-turn	LiC Score
Qwen2.5-3B Instruct	Base	0.359	0.524	0.685
Qwen2.5-3B Instruct	RLSTA	0.433	0.520	0.832
Qwen2.5-7B Instruct	Base	0.446	0.594	0.751
Qwen2.5-7B Instruct	RLSTA	0.441	0.590	0.747
Qwen3-4B Instruct-2507	Base	0.555	0.750	0.740
Qwen3-4B Instruct-2507	RLSTA	0.632	0.758	0.835

Table 4: Model performance (Coverage Score

\uparrow

) on the Summary Task across multi and single turn scenarios. RLSTA maintains, and in most cases improves, the model’s long-context capabilities while simultaneously stabilizing multi-turn interactions.

Figure 7: Training dynamics of Single-turn (ST) and Multi-turn (MT) performance across different methods. The gray dashed line ( ) represents the corresponding SFT MT performance baseline. Notably, even in the absence of external verifiable rewards ( RLSTA w/o verifier ), our method effectively narrows the performance gap between single-turn and multi-turn settings.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Vulnerability of LLMs in multi-turn settings: While Pass@1 drops significantly compared to single-turn, Pass@8 remains relatively stable. This indicates that the model retains the latent capability to solve the problem but is biased by the decoding path.

Figure 9: Additional experiments: the distribution of the contextual inertia intensity for more models.

Figure 10: Additional experiments: Tracing the root cause of LLM failures.

Figure 11: Training dynamics of Single-turn (ST) and Multi-turn (MT) performance on datasets without the filtering procedure.

Method	Qwen3-4B (No Think)				Qwen3-4B (Think)
	MT-Add		MT-Refine	Avg	MT-Add		MT-Refine	Avg
	Math	Code	Math	Avg	Math	Code	Math	Avg
Base	0.670	0.285	0.812	0.589	0.788	0.628	0.868	0.761
RLSTA	0.801	0.333	0.873	0.669	0.920	0.790	0.941	0.884

Appendix

Table 5: Experimental results comparing Base and RLSTA on Qwen3-4B. The model was trained in No Think mode but evaluated in both Think and No Think modes. RLSTA yields consistent improvements across both inference settings.

Model	Base	RLSTA	$\Delta(\%)$
Llama-3.2-3B Instruct	0.663	0.684	+3.3
Qwen2.5-3b Instruct	0.771	0.763	-0.9
Qwen2.5-7b Instruct	0.853	0.856	+0.3
Qwen3-4B Instruct-2507	0.894	0.898	+0.4
Qwen3-4B (no think)	0.877	0.893	+1.8
Qwen3-4B (think)	0.898	0.890	-0.9

Appendix

Table 6: Comparison of Base and RLSTA Performance Across Different Models. The

\Delta

column indicates the relative difference in single-turn performance between Base and RLSTA.

Figure 12: Distributions of Contextual Inertia Intensity

\texttt{I}_{\mathrm{CI}}(m_{n},m_{n-1})

. The inertia intensity are categorized using Gemini-2.5-pro.

Source of $\mathcal{D}_{\mathrm{raw}}$	Qwen2.5-3B-Instruct				Llama-3.2-3B-Instruct
	MT-Add		MT-Refine	Avg	MT-Add		MT-Refine	Avg
	Math	Code	Math	Avg	Math	Code	Math	Avg
Qwen2.5-3B	0.715	0.256	0.745	0.572	0.649	0.205	0.640	0.498
Llama-3.2-3B	0.690	0.239	0.729	0.553	0.609	0.245	0.646	0.500

Appendix

Table 7: Cross-model evaluation of Latent Capability Filtering on different data sources. We compare the performance when using raw data generated by Qwen2.5-3B-Instruct versus Llama-3.2-3B-Instruct to train different target models.

Model	Setting	Base (Acc)	RLSTA (Acc)
Qwen2.5-3B-Instruct	AtR	0.476	0.665
Qwen2.5-3B-Instruct	AiwR	0.471	0.572
Qwen2.5-7B-Instruct	AtR	0.568	0.794
Qwen2.5-7B-Instruct	AiwR	0.542	0.600
Qwen3-4B-2507	AtR	0.574	0.858
Qwen3-4B-2507	AiwR	0.615	0.863

Appendix

Table 8: Evaluation of Base and RLSTA models on hybrid multi-turn scenarios (Add-then-Refine and Add-Interleaved-with-Refine). Models trained with RLSTA show consistent improvements on unseen hybrid interaction patterns.

Explore similar work

Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL
Jun 11, 2026Shu Tong Luo, Wenqin Liu, Rui Liu +2Memory-Augmented Language Models RL for Language Model Reasoning
The University of Melbourne · The University of Tokyo
Adaptive Stopping for Multi-Turn LLM Reasoning
Apr 1, 2026Xiaofan Zhou, Huy Nguyen, Bo Yu +2Efficient Language Model Reasoning Conformal Prediction
Department of Computer Science, University of Illinois Chicago, Chicago, IL 60607, USA · Data science, Augustana College, Rock Island, IL 61201, USA · Department of Civil and Environmental Engineering, University of Utah, Salt Lake City, UT 84112, USA
Geometry of Divergence: Tracking Hidden-State Trajectories for Adaptive Multi-Turn Reasoning
Aug 31, 2026Jie Liang, Zhengxin Yu, Hamid Nasiri +1Long-Horizon LLM Agents Representation Geometry in Language Models
School of Computing and Communications, Lancaster University, UK · School of Computing and Engineering, University of Huddersfield, UK

Jun 11, 2026J/K move · Enter open · S save

Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL

Shu Tong Luo, Wenqin Liu, Rui Liu, Mingming Gong+1

The University of Melbourne · The University of Tokyo

Memory-Augmented Language Models RL for Language Model Reasoning Persistent Memory for Language Models LLM Context Management

Open paper AskPDF arXiv

When a user reveals task-critical information across several conversation turns, LLM accuracy drops by up to 65% despite full context availability. We show that this Lost in Conversation degradation can be substantially mitigated by training models to maintain a compact rolling memory instead of attending to a growing history. To make such training scalable, we introduce a low-cost sharding pipeline that converts single-turn QA datasets into multi-turn fragmented-information episodes, eliminating the need for hours of manual annotation. Training only on sharded GSM8K, our memory-augmented policy significantly improves multi-turn accuracy and generalises zero-shot to harder math and out-of-domain long-context QA. Moreover, memory-trained models outperform full-history baselines even when given the full history at test time, suggesting that learning to compress induces more robust incremental reasoning than full-context exposure alone.

Abstract

Explore similar work

Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL

Adaptive Stopping for Multi-Turn LLM Reasoning

Geometry of Divergence: Tracking Hidden-State Trajectories for Adaptive Multi-Turn Reasoning