Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs

Chenxing Wei; Hong Wang; Ying He; Fei Yu; Yao Shu

cs.CLSep 27, 2025

Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs

Authors: Chenxing Wei, Hong Wang, Ying He, Fei Yu, Yao Shu

Organizations: School of Computing and Data Science, The University of Hong Kong · Shenzhen Loop Area Institute · University of Science and Technology of China · School of Information Technology, Carleton University ·

Abstract

Large Language Models (LLMs) employ multi-turn interaction as a fundamental paradigm for completing complex tasks. However, their performance often degrades in extended interactions, as they are typically trained on static, single-turn data, which hinders their ability to adapt to real-time user feedback. To address this limitation, we first propose a new paradigm: Test-Time Policy Adaptation for Multi-Turn Interactions (T2PAM), which utilizes user feedback from the ongoing interaction as a reward signal to estimate a latent optimal policy aligned with user preferences, then updates a small subset of parameters to steer the model toward this policy, ultimately enabling efficient in-conversation self-correction. We then introduce Optimum-Referenced One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes T2PAM. ROSA guides the model parameters toward a theoretical optimal policy in a single, efficient update step, avoiding costly iterative gradient-based optimization and minimizing computational overhead. We provide a rigorous theoretical analysis guaranteeing that the policy of ROSA converges to the preference of user as the number of interactions increases. Extensive experiments on challenging benchmark demonstrate that ROSA achieves significant improvements in both task effectiveness and efficiency.

Figures & tables

Figure 1: An illustration of the Test-Time Policy Adaptation for Multi-Turn Interactions (T 2 PAM) paradigm. Different from static inference where the policy of model remains fixed ( $\theta_{0}$ , Turn 0), this paradigm treats conversational feedback as an active signal that guides real-time parameter updates (e.g., from $\theta_{0}$ to $\theta_{1}$ ). This iterative process of in-conversation self-correction allows the policy to progressively evolve and align with the preference of user ( $\theta_{n}$ ) throughout the interaction.

Figure 2: LLM accuracy after 10 rounds of interaction with humans. Although LLM accuracy shows a continuous and gradual improvement, this prompt-based correction process is inefficient.

Figure 3: Number of newly solved problems per turn on the MATH dataset.

Feature	Prompt-based Methods	Multi-turn Data Training	T²PAM (Ours)
Intervention Timing	During inference	During training	During inference
Operating Mode	Reactive	Proactive (at training)	Proactive (at inference)
Additional Offline Training Cost	Zero (uses single-turn model)	High	Zero (uses single-turn model)
Inference Cost	Low (long context)	Near-zero	Low (preference alignment)
Parameter Modification	No	Yes (offline)	Yes (online)
Real-time Adaptability	Low (context-dependent)	Low (static policy)	High (policy-level)

Table 1: Conceptual comparison of paradigms for improving multi-turn LLM performance.

Model	Method	MATH	MATH-500	MMLU-R	SuperGPQA	MT-AIME24	MT-MATH100	HumanEval
		Mathematical Reasoning		General Reasoning		Multilingual Reasoning		Code Gen.
Qwen2.5-0.5B -Instruct	Baseline	13.40	12.20	7.27	1.90	3.48	15.40	31.09
	ROSA (+LM + R)	30.40 (+17.00)	28.00 (+15.80)	9.07 (+1.80)	5.63 (+3.73)	3.67 (+0.19)	22.80 (+7.40)	37.19 (+6.10)
	ROSA (+HS + R)	25.40 (+12.00)	25.00 (+12.80)	11.00 (+3.73)	5.00 (+3.10)	4.90 (+1.42)	20.90 (+5.50)	37.27 (+6.18)
	ROSA (+LM + M)	27.00 (+13.60)	28.40 (+16.20)	13.72 (+6.45)	6.57 (+4.67)	6.13 (+2.65)	25.20 (+9.80)	39.37 (+8.28)
Qwen3-0.6B	Baseline	25.00	26.00	18.60	4.20	4.80	31.30	41.46

Table 2: Accuracy (%) of ROSA across diverse domains. Baseline is standard multi-turn interaction. (+A+B) denotes update location (LM: LM Head, HS: Hidden States) and reward type (R: rule-based, M: model-based). Red values show absolute improvements. Further details on parameter updates and reward models are provided in Appendix C.5 and C.4 , respectively.

Figure 4: Comparison of baselines , RL (Appendix F.1 ), and ROSA across datasets. Unlike the slow improvement in Figure 2 , ROSA yields steeper learning curves and higher absolute accuracy, demonstrating its superior efficiency in online error correction during multi-turn interactions.

Model	Method	MATH	AIME25	MATH-500	HumanEval
Qwen2.5-0.5B-Instruct	Baseline	6.88	0.00	6.79	14.39
Qwen2.5-0.5B-Instruct	ROSA	25.48 (+18.60)	6.67 (+6.67)	24.05 (+17.26)	26.09 (+11.70)
Qwen3-0.6B	Baseline	17.40	3.57	17.78	19.33
Qwen3-0.6B	ROSA	48.87 (+31.47)	16.67 (+13.10)	51.31 (+33.53)	31.01 (+11.68)
Qwen2.5-7B-Instruct	Baseline	12.54	3.57	13.65	28.57
Qwen2.5-7B-Instruct	ROSA	41.53 (+28.99)	20.69 (+17.12)	36.91 (+23.26)	40.00 (+11.43)

Table 3: Comparison of Correction Uplift (%).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Dataset Name	Training Size	Test Size
Mathematical Reasoning	MATH	7,500	5,000
	AIME25	N/A	30
	MATH-500	N/A	500
General Reasoning	GPQA-diamond	N/A	198
	MMLU-Redux	N/A	3,000
	SuperGPQA	26,500	N/A

Appendix

Table 4: Overview of the datasets used for evaluation. "N/A" indicates that the dataset is primarily for evaluation and does not have a standard, predefined training set.

Category	Model Name	Parameters	Variant
Small-Scale Models	Qwen2.5-0.5B-Instruct	0.5B	Instruct
Small-Scale Models	Qwen3-0.6B	0.6B	Base
Large-Scale Models	Qwen2.5-7B-Instruct	7B	Instruct
Large-Scale Models	Qwen3-8B	8B	Base
Reasoning-Focused	DeepSeek-R1-Distill-Llama-8B	8B	Reasoning-Tuned
Reasoning-Focused	DeepSeek-R1-Distill-Qwen-7B	7B	Reasoning-Tuned

Appendix

Table 5: Overview of the language models used in our experiments, categorized by scale and specialization.

Score	Criterion
$+1.0$	Correct reasoning and correct final answer.
$+0.5$	Essentially correct method with a local arithmetic or presentation error.
$+0.1$	Partially correct core idea, but a critical step is missing or flawed.
$-0.5$	Limited valid progress; the main reasoning is incorrect.
$-1.0$	Incorrect or irrelevant reasoning and answer.

Appendix

Table 6: Five-level reward rubric.

Hyperparameter	Value
target_modules	lm_head
Rank	1
lora_alpha	8
lora_dropout	0.1

Appendix

Table 7: LoRA Hyperparameter Configuration.

Method	Template A (Default)	Template B (Direct)	Template C (Guiding)
ROSA (+LM+R)	50.20%	48.40%	52.60%
ROSA (+HS+R)	50.80%	50.20%	51.60%
ROSA (+LM+M)	52.20%	53.60%	54.40%

Appendix

Table 8: Robustness to different feedback templates on the MATH dataset using Qwen3-0.6B. The baseline accuracy without ROSA is 25.00%.

Figure 5: Time-to-Accuracy comparison across different models and datasets. The x-axis represents cumulative wall-clock time. Our method ROSA consistently exhibits steeper learning curves, highlighting its significant advantage in time efficiency.

Model	Method	Averaged Latency (s)		Peak GPU Memory (GB)
Model	Method	Inference	Update	Inference	Update
Qwen2.5-0.5B-Instruct	Baseline	8.42	0	11.85	0
Qwen2.5-0.5B-Instruct	ROSA	7.97	7.58	11.68	18.60
Qwen3-0.6B	Baseline	18.71	0	15.61	0
Qwen3-0.6B	ROSA	18.85	13.16	16.24	23.65
Qwen2.5-7B-Instruct	Baseline	19.40	0	42.01	0

Appendix

Table 9: Efficiency analysis of ROSA on models up to 8B parameters. We report the averaged inference latency and peak GPU memory per turn. The “Update” columns show the additional overhead introduced by ROSA.

Model / Method	Peak GPU Memory (GB)	Avg. Time to Solve
Qwen3-32B (Baseline)	94.2	1.0x (Reference)
+ ROSA (+LM+R)	94.9 (+0.7)	1.9x

Appendix

Table 10: Computational overhead on the Qwen3-32B model, evaluated on the MATH dataset.

Model	Setup	OSWorld (Online)	AndroidWorld (Online)
UI-TARS-7B-SFT	Baseline	13.23	27.61
	+ ROSA (+LM+R)	16.71 (+3.48)	30.26 (+2.65)
	+ ROSA (+LM+M)	17.83 (+4.60)	30.93 (+3.32)
UI-TARS-7B-DPO	Baseline	14.80	28.90
	+ ROSA (+LM+R)	17.32 (+2.52)	30.37 (+1.47)
	+ ROSA (+LM+M)	18.04 (+3.24)	31.68 (+2.78)

Appendix

Table 11: Success Rate (%) on delayed-reward UI Agent tasks (OSWorld and AndroidWorld). The models complete long trajectories (

\sim

15 turns) before receiving a single delayed reward.

Task Domain	Benchmark	Metric	Baseline	ROSA
Subjective	MT-Bench	GPT-5.2 Score (Scaled)	60.36	75.57 (+15.21)
Instruction	IFEval	Strict Acc. (%)	58.31	70.26 (+11.95)

Appendix

Table 12: Generalization to open-ended and subjective tasks using Qwen3-0.6B.

Paradigm	Specific Method	Final Acc. (%)	Correction Uplift (%)	Avg. Time to Solve
Standard	Zero-Shot Baseline	26.00	17.40	1.0x (Reference)
Automatic Prompt Eng.	APE (DSPy Khattab et al. (2024) )	29.20	16.83	$\sim$ 1.0x
	APE (TextGrad Yuksekgonul et al. (2025) )	29.40	16.76	$\sim$ 1.0x
	APE (TSGD-M Ding et al. (2025) )	30.20	17.16	$\sim$ 1.0x
Search & Sampling	Best-of- $N$ ( $N=10$ )	36.40	20.36	$\sim$ 10.0x
Search & Sampling	Reranking ( $N=10$ + RM)	37.80	21.57	$\sim$ 10.0x + RM

Appendix

Table 13: Comprehensive comparison against test-time compute baselines on the MATH dataset using Qwen3-0.6B. ROSA achieves the highest accuracy and correction uplift with a highly competitive time overhead.

Model / Method	MATH	MMLU-R	MT-AIME24	HumanEval
Qwen3-32B (Baseline)	76.40	73.63	67.93	85.37
+ ROSA (+LM+R)	87.80 (+11.40)	85.12 (+11.49)	76.19 (+8.26)	87.80 (+2.43)
+ ROSA (+HS+R)	87.40 (+11.00)	86.05 (+12.42)	77.27 (+9.34)	87.80 (+2.43)

Appendix

Table 14: Performance scalability of ROSA on the Qwen3-32B model across diverse benchmarks.

Method	Turn 5	Turn 10	Turn 15	Turn 20
Baseline	19.40	25.00	27.20	27.20
ROSA	25.80 (+6.40)	50.20 (+25.20)	57.40 (+30.20)	64.80 (+37.60)

Appendix

Table 15: Extended horizon evaluation on the MATH dataset over 20 conversational turns (Accuracy %).

Method	Reward Granularity	Final Acc. (%)	Avg. Turns to Solve
Baseline	None	26.00	8.9
ROSA (Rule ORM)	Final Outcome Reward	50.20	6.9
ROSA (LLM ORM)	Continuous Outcome Reward	52.20	6.7
ROSA (Rubric)	Five-Level Outcome Reward	51.80	6.7
ROSA (PRM)	Step-level Process Reward	60.40 (+10.20)	5.8

Appendix

Table 16: Performance comparison of ROSA using Outcome Reward Models (ORM) versus Process Reward Models (PRM) on the MATH dataset using Qwen3-0.6B.

Model State	MMLU (Zero-shot Acc. %)	Wikitext Perplexity	Average KL Divergence
Base Model	51.03	18.2037	0.0
ROSA Adapted Model (10 updates)	51.36	18.2045	0.001

Appendix

Table 17: General knowledge retention stress test on Qwen3-0.6B. The model was evaluated after undergoing 10 consecutive ROSA updates on unresolved math problems.

Method	Feedback Samples / Turn	Final Acc. (%)	Avg. Turns to Solve	Avg. Time to Solve
Baseline	–	26.00	8.9	1.0x
ROSA-Single (Ours)	$M=1$	50.20	6.7	$\sim$ 1.8x
ROSA-Multi	$M=4$	52.60 (+2.40)	6.5	$\sim$ 6.9x

Appendix

Table 18: Comparison of single-sample (

M=1

) versus multi-sample (

M=4

) feedback on the MATH dataset using Qwen3-0.6B.

	MATH		AIME25		MATH-500		HumanEval
Method	Final Acc. $\uparrow$	Correction Uplift $\uparrow$	Final Acc. $\uparrow$	Correction Uplift $\uparrow$	Final Acc. $\uparrow$	Correction Uplift $\uparrow$	Final Acc. $\uparrow$	Correction Uplift $\uparrow$
Baseline	7.60	3.14	10.00	3.57	7.40	6.09	45.12	17.05
ROSA	9.80 (+2.20)	5.65 (+2.51)	16.67 (+6.67)	16.67 (+13.10)	22.20 (+14.80)	18.62 (+12.53)	51.22 (+6.10)	33.75 (+16.70)

Appendix

Table 19: Performance of the DeepSeek-R1-Distill-Qwen-8B model on mathematical reasoning and code generation datasets. The values in red indicate the absolute improvement of ROSA over the baseline.

		Mathematical Reasoning	General Reasoning	Multilingual Reasoning
Model	Method	AIME25	GPQA-diamond	M-IMO
Qwen2.5-0.5B -Instruct	Baseline	3.33	3.54	1.99
	ROSA (+LM + R)	6.67 (+3.34)	7.07 (+3.53)	2.09 (+0.10)
	ROSA (+HS + R)	6.67 (+3.34)	8.53 (+4.99)	3.20 (+1.21)
	ROSA (+LM + M)	6.67 (+3.34)	10.27 (+6.73)	4.71 (+2.72)
Qwen3-0.6B	Baseline	10.00	12.20	5.20

Appendix

Table 20: Supplementary performance results on additional benchmarks, reporting accuracy (%). The values in red indicate the absolute improvement of ROSA variants over the baseline.

Method	Final Acc. $\uparrow$	Correction Uplift $\uparrow$
Baseline	55.80	23.00
SFT Training	63.80	39.24
RL Training	66.20	40.45
ROSA	65.80	40.42

Appendix

Table 21: Comparison of ROSA with training-based methods on the MATH dataset for the Qwen3-8B model. Our test-time method achieves performance comparable to full Reinforcement Learning (RL) training and surpasses Supervised Fine-Tuning (SFT), with zero additional offline training cost.

Figure 6: Detailed distribution of newly solved problems per round on the MATH dataset across various models.

Adaptation scope	Feedback used	Success (%)
No parameter updates	None	3
Instance-specific, reset between tasks	Negative only	14
Persistent across tasks	Negative only	16
Persistent across tasks	Positive and negative	19

Appendix

Table 22: Learning from successful trajectories on BigCodeBench-Hard.

Figure 7: Preference alignment on ALOE using Qwen3-8B. Mean Alignment Level (1–5) across 100 personas is shown at each interaction turn. ROSA improves alignment more rapidly than standard multi-turn interaction, reaching 4.99 versus 4.70 at turn 10. Both methods use the same judging protocol; details are in Appendix E.12 .

Method	1	2	3	4	5	6	7	8	9	10	Mean
Baseline	2.93	2.99	2.94	3.21	3.53	4.01	4.27	4.57	4.68	4.70	3.78
ROSA	2.93	3.51	3.95	4.33	4.68	4.74	4.89	4.95	4.98	4.99	4.40

Appendix

Table 23: ALOE preference-satisfaction scores (1–5) at each turn. The mean is over ten turns.

Model	Method	MATH	MATH-500	AIME25	HumanEval
Qwen3-0.6B	RL	26.20	28.80	10.00	42.68
Qwen3-0.6B	ROSA	50.20 (+24.00)	51.60 (+22.80)	16.67 (+6.67)	45.73 (+3.05)
Qwen3-8B	RL	59.60	63.60	16.67	79.27
Qwen3-8B	ROSA	65.80 (+6.20)	72.80 (+9.20)	30.00 (+13.33)	81.71 (+2.44)
DeepSeek-R1-Distill -Llama-8B	RL	6.20	8.40	10.00	28.05
DeepSeek-R1-Distill -Llama-8B	ROSA	7.80 (+1.60)	18.40 (+10.00)	16.67 (+6.67)	39.02 (+10.97)

Appendix

Table 24: Comparison of Accuracy (%) on mathematical reasoning datasets with RL and ROSA.

Model	Method	MATH	MATH-500	AIME25	HumanEval
Qwen3-0.6B	RL	18.54	20.00	6.90	22.31
Qwen3-0.6B	ROSA	48.87 (+30.33)	51.31 (+31.31)	16.67 (+9.77)	31.01 (+8.70)
Qwen3-8B	RL	29.37	31.58	10.71	50.00
Qwen3-8B	ROSA	40.42 (+11.05)	52.94 (+21.36)	27.59 (+16.88)	62.50 (+12.50)
DeepSeek-R1-Distill -Llama-8B	RL	4.87	7.29	6.90	16.31
DeepSeek-R1-Distill -Llama-8B	ROSA	6.30 (+1.43)	17.41 (+10.12)	13.79 (+6.89)	31.97 (+15.66)

Appendix

Table 25: Comparison of Correction Uplift (%) on mathematical reasoning datasets with RL and ROSA.

Figure 8: Ablation study of the hyperparameter

\beta

on the MATH dataset. The figure illustrates the cumulative accuracy over 10 conversational turns for different values of

\beta

, ranging from 0.25 to 1.75.

Inference-Time Adaptation of Language Models

See all 33 papers

Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation
Mar 2, 2026Chenxing Wei, Hong Wang, Ying He +4Inference-Time Adaptation of Language Models Inference-Time Language Model Alignment
Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Bytedance +4
Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
Mar 21, 2026Haoyu Wang, Yuxin Chen, Liang Luo +3RL for Language Models Process Reward Models
Georgia Institute of Technology · Meta AI
Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction
Mar 5, 2026Xingwu Chen, Zhanqiu Zhang, Yiwen Guo +1Multi-Turn Language Model Reasoning RL for Language Model Reasoning
School of Computing and Data Science, The University of Hong Kong · LIGHTSPEED · Independent Researcher

Mar 2, 2026J/K move · Enter open · S save

Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation

Chenxing Wei, Hong Wang, Ying He, Zhongxiang Dai+3

Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Bytedance+4

Inference-Time Adaptation of Language Models Inference-Time Language Model Alignment Prompt Optimization

Open paper AskPDF arXiv

Test-time policy adaptation for multi-turn interactions (T2PAM) is essential for aligning Large Language Models (LLMs) with dynamic user needs during inference time. However, existing paradigms commonly treat test-time adaptation as a single-axis problem, either purely refining instructions (Prompt Engineering) or only adjusting weights (Test-Time Training), ignoring that interaction failures stem from a coupled mix of ambiguity and incapacity. We argue that these two optimization paths are not merely additive but synergistic: semantic clarity acts as a pre-conditioner for effective parameter updates. To this end, we propose ROSA2, a framework that reformulates interaction as a joint optimization problem over the heterogeneous space of Words and Weights. By mathematically decomposing the error signal, ROSA2 utilizes textual gradients to rectify intent ambiguity and parameter updates to bridge capability gaps. Theoretically, we prove that this co-adaptation strictly reduces the required parameter shift for convergence. Empirically, ROSA2 outperforms state-of-the-art baselines by 30% on MATH while reducing interaction turns by 40%, demonstrating that refining the context unlocks the true potential of parameter updates.

Abstract

Inference-Time Adaptation of Language Models

Explore similar work

Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction