Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Haoyu Wang; Yuxin Chen; Liang Luo; Buyun Zhang; Ellie Dingqiao Wen; Pan Li

cs.LGMar 21, 2026

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Authors: Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li

Organizations: Georgia Institute of Technology · Meta AI

Abstract

Multi-turn human-AI collaboration is fundamental to deploying interactive services such as adaptive tutoring, conversational recommendation, and professional consultation. However, optimizing these interactions via reinforcement learning is hindered by the sparsity of verifiable intermediate rewards and the high stochasticity of user responses. To address these challenges, we introduce Implicit Turn-wise Policy Optimization (ITPO). ITPO leverages an implicit process reward model to derive fine-grained, turn-wise process rewards from sparse outcome signals. Unlike volatile token-level rewards, these turn-level signals exhibit superior robustness and may utilize a normalization mechanism to further enhance training stability. We evaluate ITPO across three representative multi-turn collaborative tasks: math tutoring, document writing, and medical recommendation. Empirical results demonstrate that ITPO, when combined with PPO, GRPO, or RLOO, consistently achieves improved convergence than existing baselines. Elaborate trajectory analysis confirms that ITPO infers turn-wise preferences that are semantically aligned with human judgment. Code is publicly available at https://github.com/Graph-COM/ITPO.

Figures & tables

Figure 1: The overall framework of ITPOfor User-LLM interactions with a real case example in Medical Recommendation. The optimization loop consists: of (1) Multi-turn online roll-out, (2) implicit PRM update with outcome rewards and token-level process reward estimation, (3) Turn-wise process reward aggregation and normalization, (4) Advantage estimation and policy optimization.

Figure 2: Dynamics of the relative implicit rewards on a real Math Tutoring case. Turn-wise rewards of Response Turn #1 assigned by ITPOconsistently rank the highest from step $20$ , meanwhile reflecting the importance of resolving ambiguity early in the conversation. By contrast, the token-level rewards show high variance, with the signals for identical tokens fluctuating across training steps (red boxes).

Figure 3: Spearman correlation of the token-level and turn-wise implicit rewards against their respective converged baselines (the averaged rewards over the final

70

optimization steps) on Math Tutoring. Turn-wise preference stabilizes quickly, while token-level rankings show slower convergence due to optimization difficulty.

Advantage Estimator	Credit Assignment	Math Tutoring			Doc Writing			Medical Recommendation
Advantage Estimator	Credit Assignment	Accracy $\uparrow$	Token(k) $\downarrow$	Score $\uparrow$	Bleu $\uparrow$	Token(k) $\downarrow$	Score $\uparrow$	Diagnosis $\uparrow$	Token(k) $\downarrow$	Score $\uparrow$
Vanilla		25.81	1.70	17.29	24.97	2.40	12.93	25.52	0.28	24.08
PPO	VM Only	2 6.81	1.36	20.06	40.57	0.84	36.37	5 8.41	0 .72	54.77
	VM+PRIME	22.63	1.54	14.92	41.87	2.42	29.77	52.07	0.48	49.73
	VM+ITPO (Ours)	26.38	0 .94	2 1.51	4 2.52	1.02	3 7.44	58.35	0.68	5 4.87
	VM+Norm-ITPO (Ours)	27.15	0.74	23.44	45.16	0 .96	40.12	62.44	0.60	59.44

Table 1: Evaluation results of fine-grained reward attribution strategies, with task-specific metrics (Accuracy, BLEU, Diagnosis Score), generation cost (Tokens), and overall utility scores. Bold values denote the best performance within each group, Underlined denotes the second. ‘Vanilla’ denotes the unaligned backbone policy, ‘VM’ (Value Model) denotes the inclusion of a parameterized value model.

Panel of Figure 4 — Figure 4: The training curve of reward attribution methods with the RLOO advantage estimator.

Panel of Figure 5 — Figure 5: Kendall- $\tau$ between PRM and outcome reward w/ RLOO.

Panel of Figure 6 — Figure 6: Left tracks the Spearman correlation of turn-wise implicit rewards between each training step and the converged reward allocation (calculated as the average over the last $70$ steps). Additionally, the mutual Spearman correlation between ITPOand Norm -ITPOdemonstrates that both methods learn a highly consistent preference structure independent of initialization. Right measures the correlation between adjacent training steps.

Figure 7: Slopes of the mapping between trajectory-level implicit

\mathcal{R}_{\phi}(\tau)

and outcome

\mathbf{R}

on fixed sets of roll-outs trajectories from the validation set in Doc Writing task. Each curve represents the slope of a roll-out group with same initial prompts during training.

Panel of Figure 8 — Figure 8: Qwen3-4B (left) and Qwen2.5-7B (right) on Doc Writing.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Comparison of the baselines adopted in the paper.

	Math Tutoring			Doc Writing			Medical Recommendation
	$\beta$	$\eta$	$w(\mathcal{R}_{\phi})/w(\mathbf{R})$	$\beta$	$\eta$	$w(\mathcal{R}_{\phi}/w(\mathbf{R})$	$\beta$	$\eta$	$w(\mathcal{R}_{\phi})/w(\mathbf{R})$
PRIME	5e-2	-	5/5	5e-3	-	5/5	5e-3	-	5/5
ITPO	5e-2	-	5/5	5e-3	-	5/5	1e-2	-	5/5
Norm-ITPO	1e-2	0.4	5/0	5e-3	0.4	5/0	5e-3	0.4	5/0

Appendix

Table 2: Hyperparameters of implicit process reward models.

w(\mathcal{R}_{\phi})/w(\mathbf{R})

denotes the weight coefficient of advantage results from implicit reward

\mathcal{R}_{\phi}

and outcome

\mathbf{R}

Panel of Figure 10 — Figure 10: The training curve of reward attribution methods with the GRPO advantage estimator.

Panel of Figure 11 — Figure 11: Stability and Consistency on the Math Tutoring and Medical Recommendation tasks.

Panel of Figure 12 — Figure 12: Kendall- $\tau$ between PRM and outcome reward w/ GRPO.

Figure 13: Part #1 of low-performance trajectory #1 in Example #1

Figure 14: Part #2 of low-performance trajectory #1 in Example #1

Figure 15: Part #3 of low-performance trajectory in Example #1

Figure 16: Part #1 of low-performance trajectory #2 in Example #1

Figure 17: Part #2 of low-performance trajectory #2 in Example #1

Figure 18: Part #3 of low-performance trajectory #2 in Example #1

Figure 19: Part #1 of high-performance trajectory #1 in Example #1

Figure 20: Part #2 of high-performance trajectory #1 in Example #1

Figure 21: Part #1 of high-performance trajectory #2 in Example #1

Figure 22: Part #2 of high-performance trajectory #2 in Example #1

Figure 23: Part #3 of high-performance trajectory #2 in Example #1

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 407 papers

InfoPO: Information-Driven Policy Optimization for User-Centric Agents
Feb 28, 2026Fanqi Kong, Jiayi Zhang, Mingyi Deng +3LLM Agent Training RL for LLM Agents
Peking University · DeepWisdom · The Hong Kong University of Science and Technology (Guangzhou) +1
IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
Aug 25, 2026Bo Ren, Yirong Mao, Yi Yang +1LLM Agent Training Credit Assignment in RL
WeChat, Tencent Inc.
A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
May 7, 2026Dingwei Chen, Zefang Zong, Zhipeng Ma +5Credit Assignment in RL Policy Gradient Methods
Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University

Feb 28, 2026J/K move · Enter open · S save

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu+2

Peking University · DeepWisdom · The Hong Kong University of Science and Technology (Guangzhou)+1

LLM Agent Training RL for LLM Agents Active Information Gathering Reward Shaping

Open paper AskPDF arXiv

Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent's subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment-interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent-user collaboration. Code is available at https://github.com/kfq20/InfoPO.

Abstract

RL for Language Models

Explore similar work

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

A2^22TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping