Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation

Chenxing Wei; Hong Wang; Ying He; Zhongxiang Dai; Bo Jiang; F. Richard Yu; Yao Shu

cs.AIMar 2, 2026

Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation

Authors: Chenxing Wei, Hong Wang, Ying He, Zhongxiang Dai, Bo Jiang, F. Richard Yu, Yao Shu

Organizations: Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Bytedance · University of Science and Technology of China ·

Abstract

Test-time policy adaptation for multi-turn interactions (T2PAM) is essential for aligning Large Language Models (LLMs) with dynamic user needs during inference time. However, existing paradigms commonly treat test-time adaptation as a single-axis problem, either purely refining instructions (Prompt Engineering) or only adjusting weights (Test-Time Training), ignoring that interaction failures stem from a coupled mix of ambiguity and incapacity. We argue that these two optimization paths are not merely additive but synergistic: semantic clarity acts as a pre-conditioner for effective parameter updates. To this end, we propose ROSA2, a framework that reformulates interaction as a joint optimization problem over the heterogeneous space of Words and Weights. By mathematically decomposing the error signal, ROSA2 utilizes textual gradients to rectify intent ambiguity and parameter updates to bridge capability gaps. Theoretically, we prove that this co-adaptation strictly reduces the required parameter shift for convergence. Empirically, ROSA2 outperforms state-of-the-art baselines by 30% on MATH while reducing interaction turns by 40%, demonstrating that refining the context unlocks the true potential of parameter updates.

Figures & tables

Figure 1: Overview of the ROSA2 Framework. We formulate T 2 PAM as a joint optimization over the coupled variables $\phi_{t}=\{x_{t+1},\theta_{t}\}$ . During the Forward Phase (solid lines), the model generates a response $y_{t}$ conditioned on history $H_{t-1}$ . The Backward Phase (dashed lines) approximates the full gradient $\nabla_{\text{joint}}$ of the interaction loss $\mathcal{L}$ through two synergistic modules. Textual Optimization (top, green) uses textual gradients ( $\nabla_{\mathrm{x}}$ ) to refine user feedback into a clarified instruction ( $x_{t+1}\rightarrow x_{t+1}^{*}$ ), resolving context ambiguity. Simultaneously, Parameter Optimization (bottom, blue) applies gradient updates ( $\nabla_{\theta}$ ) to adjust adapter weights ( $\theta_{t}\rightarrow\theta_{t+1}$ ), enhancing the intrinsic capability of the model. This co-adaptation ensures the system is both “Clearer” and “Stronger” for the next turn.

Figure 2: Empirical Observations and Theoretical Landscape. Figure (a) In the experimental results on MATH (Qwen3-8B) reveal that single-axis methods ( Green / Blue solid lines) suffer from premature stagnation. However, the immediate recovery observed in the Switch experiments ( Green / Blue dashed lines) suggests this bottleneck is structural. Figure (b) We map these dynamics to the optimization landscape using consistent color and line styling : the Prompt-Only path ( Green ) stalls in the Deficit Trap (Hitting capability ceilings), while the Param-Only path ( Blue ) gravitates towards the Overfitting Trap (Memorizing noise). The dashed arrows in Figure (b) visualize how the Switch Method escapes these local minima by activating the missing axis. Crucially, ROSA2 ( Red ) approximates the joint gradient

\nabla_{\text{joint}}

, forming an Optimal Trajectory that bypasses these traps and proceeds directly to the Success Zone , corresponding to the superior convergence shown in Figure (a).

Figure 3: Dynamics of approximation error terms on MATH using Qwen3-8B. The semantic update magnitude

\|\Delta\mathbf{x}_{t}\|_{2}^{2}

is computed as an

L_{2}

distance in the continuous token-embedding space. The parametric curve of ROSA2 and the ROSA baseline start from the same adapter initialization at Turn 1. The total error includes both semantic and parametric components, so it can start higher due to the non-zero initial semantic discrepancy.

Model	Method	MATH	MATH-500	MMLU-R	SuperGPQA	MT-AIME24	MT-MATH100	HumanEval
		Mathematical Reasoning		General Reasoning		Multilingual Reasoning		Code Gen.
Qwen2.5-0.5B -Instruct	Baseline	23.0	24.0	9.4	3.8	2.6	15.4	31.1
	TextGrad	31.2 (+8.2)	29.6 (+5.6)	12.4 (+3.0)	3.8 (+0.0)	2.2 (-0.4)	18.4 (+3.0)	36.6 (+5.5)
	ROSA	29.2 (+6.2)	30.4 (+6.4)	11.4 (+2.0)	4.0 (+0.2)	3.8 (+1.2)	19.6 (+4.2)	38.4 (+7.3)
	ROSA2	40.8 (+17.8)	39.6 (+15.6)	18.4 (+9.0)	6.4 (+2.6)	4.4 (+1.8)	25.2 (+9.8)	44.5 (+13.4)
Qwen3-0.6B -Instruct	Baseline	19.6	22.4	24.0	3.8	3.2	26.2	41.5

Table 1: Main Results on Standard Reasoning Benchmarks. We report the accuracy (%) across mathematical (MATH, MATH-500), general (MMLU-R, SuperGPQA), multilingual (MT-AIME24, MT-MATH100), and code generation (HumanEval) tasks. The gains are calculated relative to the Baseline. Best scores are bolded , and second-best scores are underlined .

Figure 4: Performance trajectory on challenging benchmarks. We plot the accuracy on AIME25, GPQA-Diamond, M_IMO, and BigCodeBench-Hard as a function of interaction turns. ROSA2 (red line) demonstrates sustained accuracy improvements, successfully solving complex problems where baselines plateau.

Dataset	Method	Correction Uplift ( $\uparrow$ )	Avg Turn ( $\downarrow$ )
MATH	Baseline	70.0%	7.2
	TextGrad	75.1% (+5.1%)	6.0 (-1.2)
	ROSA	77.3% (+7.3%)	6.3 (-0.9)
	ROSA2	81.4% (+11.4%)	4.4 (-2.8)
MMLU	Baseline	50.9%	6.6
	TextGrad	59.5% (+8.6%)	5.2 (-1.4)

Table 2: Analysis of Interaction Dynamics on Qwen3-8B. Correction Uplift indicates the percentage of eventually solved problems that were corrected after the initial failure. Avg Turn denotes the average interaction turns required to solve a problem.

Model	OSWorld	AndroidWorld
UI-TARS-7B-SFT ( Qin et al., 2025 )	13.2	27.6
UI-TARS-7B-SFT + TextGrad	13.7 (+0.5)	28.3 (+0.7)
UI-TARS-7B-SFT + ROSA	17.8 (+4.6)	30.9 (+3.3)
UI-TARS-7B-SFT + ROSA2	23.6 (+10.4)	35.3 (+7.7)
UI-TARS-7B-DPO	14.8	28.9
UI-TARS-7B-DPO + TextGrad	14.9 (+0.1)	28.7 (-0.2)

Table 3: Adaptability in sparse-reward environments (UI Agents).

Dataset	Method	Avg Time (s) ( $\downarrow$ )	Peak Memory (GB) ( $\downarrow$ )
MATH	Baseline	334.5	90.6
MATH	ROSA2	297.6 (-36.9)	93.7 (+3.1)
AIME25	Baseline	557.4	94.9
AIME25	ROSA2	467.2 (-90.2)	95.4 (+0.5)
HumanEval	Baseline	543.7	94.8
HumanEval	ROSA2	521.3 (-22.4)	95.2 (+0.4)

Table 4: Computational Cost Analysis on Qwen3-8B.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Interaction Turn	Parameter Update	Semantic Update	Policy Shift
Turn 1	0.0352	0.0018	0.142
Turn 5	0.0114	0.0004	0.051
Turn 10	0.0021	$<0.0001$	0.008

Appendix

Table 5: Empirical validation of local smoothness on MATH using Qwen3-8B.

Domain	Dataset	Train Size	Test Size
Mathematical Reasoning	MATH	7,500	5,000
	AIME25	N/A	30
	MATH-500	N/A	500
General Reasoning	GPQA-diamond	N/A	198
	MMLU-Redux	N/A	3,000
	SuperGPQA	26,500	N/A

Appendix

Table 6: Summary of benchmarks used for evaluation. ”N/A” denotes datasets primarily used for testing that lack a standard pre-defined training split.

Category	Model	Params	Type
Small-Scale	Qwen2.5-0.5B-Instruct	0.5B	Instruct
Small-Scale	Qwen3-0.6B	0.6B	Base
Large-Scale	Qwen2.5-7B-Instruct	7B	Instruct
Large-Scale	Qwen3-8B	8B	Base
Reasoning-Focused	DeepSeek-R1-Distill-Llama-8B	8B	Reasoning
Reasoning-Focused	DeepSeek-R1-Distill-Qwen-7B	7B	Reasoning

Appendix

Table 7: Categorization of language models used in experiments.

Hyperparameter	Value / Configuration
Interaction Limit ( $T_{\max}$ )	10 turns
Responses per Turn	$N=1$ for all multi-turn methods
LoRA Rank ( $r$ )	1
LoRA Alpha ( $\alpha$ )	8
Learning Rate (Param Optimization)	$1\times 10^{-4}$
KL Penalty Coefficient ( $\beta$ )	1.0

Appendix

Table 8: Comprehensive hyperparameter setup for ROSA2 and core baselines.

Method	Final Acc.	Correction Uplift	Avg Turn
Baseline	$50.0\pm 5.3$	$70.0\pm 3.9$	$7.2\pm 1.2$
TextGrad	$63.4\pm 1.7$	$75.1\pm 0.6$	$6.0\pm 0.2$
ROSA	$62.2\pm 0.9$	$77.3\pm 0.8$	$6.3\pm 0.3$
ROSA2	$\mathbf{80.8\pm 0.3}$	$\mathbf{81.4\pm 0.5}$	$\mathbf{4.4\pm 0.1}$

Appendix

Table 9: Robustness across five random seeds on Qwen3-8B/MATH. Final Acc. is solved-within-

T_{\max}

accuracy, while Correction Uplift is computed only over problems that fail at Turn 0.

Method	Paradigm	Avg. Time to Solve	Accuracy
Zero-Shot (Base)	Static	1.0x	50.0%
SFT Before Test	Offline Training	$\sim$ 1.2x	64.4%
Best-of-16 (Voting)	Test-Time Sampling	$\sim$ 16.0x	67.8%
Self-Critique / Reflexion	LLM Self-Correction	$\sim$ 2.1x	70.8%
TextGrad	LLM Self-Correction	$\sim$ 2.2x	71.4%
ROSA2 (ROSA + Self-Critique)	Joint Optimization	$\sim$ 4.1x	81.6%

Appendix

Table 10: Comparison with stronger optimization baselines on Qwen3-8B/MATH.

Method	Prior Training	Turn-0 Acc.	Final Acc.	Avg Turns
Baseline (Multi-turn Prompting)	None	15.8%	50.0%	7.2
SDPO (Test-time only)	None	15.8%	53.6%	6.7
ROSA2	None	15.8%	80.8%	4.4
SDPO (Train + Test)	Train-time RL	34.0%	69.0%	5.9

Appendix

Table 11: Test-time adaptation comparison with SDPO on MATH using Qwen3-8B. Turn-0 Acc. is the correctness of the first response only; Final Multi-turn Acc. is the cumulative solved-within-10-turn accuracy.

Metric per Turn	Qwen3-8B	Qwen3-32B
Baseline Peak Memory (Generation)	90.6 GB	93.1 GB
Peak Semantic Update Memory (Words)	91.2 GB	93.2 GB
Peak Parametric Update Memory (Weights)	93.6 GB	94.6 GB
ROSA2 Parallel Peak Memory	93.6 GB (+3.0 GB)	94.6 GB (+1.5 GB)
Baseline Avg. Generation Time	46.4s	67.2s
ROSA2 Avg. Generation Time	37.6s	49.9s

Appendix

Table 12: Per-turn computational decomposition on one H20 GPU with vLLM.

Method	Avg Turn to Acceptance	Win Rate vs. Base
One-Turn	N/A	50.0%
Standard Prompting	4.8	62.3%
TextGrad Only	3.5	71.5%
ROSA	3.8	75.5%
ROSA2	2.1	85.4%

Appendix

Table 13: Open-ended dialogue alignment on MT-Bench with GPT-5.2-as-a-Judge.

Inference-Time Adaptation of Language Models

See all 25 papers

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
Mar 21, 2026Haoyu Wang, Yuxin Chen, Liang Luo +3RL for Language Models Process Reward Models
Georgia Institute of Technology · Meta AI
Query-Conditioned Test-Time Self-Training for Large Language Models
May 13, 2026Chaehee Song, Minseok Seo, Yeeun Seong +2Test-Time Adaptation Test-Time Optimization
School of Electrical Engineering, KAIST · 2Graduate School of Green Growth and Sustainability, KAIST
Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction
Mar 5, 2026Xingwu Chen, Zhanqiu Zhang, Yiwen Guo +1Multi-Turn Language Model Reasoning RL for Language Model Reasoning
School of Computing and Data Science, The University of Hong Kong · LIGHTSPEED · Independent Researcher

Mar 21, 2026J/K move · Enter open · S save

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang+2

Georgia Institute of Technology · Meta AI

RL for Language Models Process Reward Models Reward Modeling

Open paper AskPDF arXiv

Multi-turn human-AI collaboration is fundamental to deploying interactive services such as adaptive tutoring, conversational recommendation, and professional consultation. However, optimizing these interactions via reinforcement learning is hindered by the sparsity of verifiable intermediate rewards and the high stochasticity of user responses. To address these challenges, we introduce Implicit Turn-wise Policy Optimization (ITPO). ITPO leverages an implicit process reward model to derive fine-grained, turn-wise process rewards from sparse outcome signals. Unlike volatile token-level rewards, these turn-level signals exhibit superior robustness and may utilize a normalization mechanism to further enhance training stability. We evaluate ITPO across three representative multi-turn collaborative tasks: math tutoring, document writing, and medical recommendation. Empirical results demonstrate that ITPO, when combined with PPO, GRPO, or RLOO, consistently achieves improved convergence than existing baselines. Elaborate trajectory analysis confirms that ITPO infers turn-wise preferences that are semantically aligned with human judgment. Code is publicly available at https://github.com/Graph-COM/ITPO.

Abstract

Inference-Time Adaptation of Language Models

Explore similar work

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Query-Conditioned Test-Time Self-Training for Large Language Models

Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction