InfoPO: Information-Driven Policy Optimization for User-Centric Agents

Fanqi Kong; Jiayi Zhang; Mingyi Deng; Chenglin Wu; Yuyu Luo; Bang Liu

cs.AIFeb 28, 2026

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

Authors: Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, Bang Liu

Organizations: Peking University · DeepWisdom · The Hong Kong University of Science and Technology (Guangzhou) · Universit´e de Montréal & Mila

Abstract

Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent's subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment-interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent-user collaboration. Code is available at https://github.com/kfq20/InfoPO.

Figures & tables

Figure 1: Standard GRPO vs. InfoPO. Standard GRPO yields zero reward for ”honorable failures” (correct elicitation, failed execution). InfoPO solves this via counterfactual masking to provide dense, turn-level information-gain rewards.

Figure 2: Overview of the InfoPO framework. It extracts a turn-level info-gain signal by counterfactual reasoning and adaptively fuses it with outcome-based advantages to facilitate efficient credit assignment in multi-turn user-centric tasks.

Type	Method	UserGym								ColBench		$\tau^{2}$ -Bench			Avg
Type	Method	Travel	Func.	Pers.	Tau	Turtle	Search	Inten.	Tele.	Pass	Succ.	Telec.	Retail	Air.	Avg
Closed-Source Model
Prompting	Gemini-3-Flash	0.574	0.423	0.695	0.167	0.153	0.968	1.718	0.829	0.515	0.382	0.469	0.619	0.558	0.621
Prompting	GPT-4.1	0.554	0.051	0.599	0.109	0.267	0.480	1.867	0.732	0.529	0.403	0.388	0.544	0.400	0.533
Prompting	GPT-4o-mini	0.596	0.089	0.683	0.091	0.117	0.568	1.277	0.512	0.463	0.342	0.113	0.400	0.175	0.417
Qwen2.5-7B-Instruct

Table 1: Experimental results on UserGym, ColBench, and

\tau^{2}

-Bench. The grey rows represent our proposed method and its ablation versions. Best and second-best results are marked within each model group (Closed-Source/Qwen2.5/Qwen3). Environment abbreviations: Func.: FunctionGym, Pers.: PersuadeGym, Inten.: IntentionGym, Tele.: TelepathyGym, Telec.: Telecom, Air.: Airline. Succ. denotes Success rate.

Panel of Figure 3 — Figure 3: Extrinsic reward curves during training on (a) UserGym, (b) ColBench, and (c) $\tau^{2}$ -Bench. Solid lines and shaded regions represent mean $\pm$ std across three seeds.

Model	UserGym	ColBench	$\tau^{2}$ -Bench
Qwen2.5-7B-Inst.	0.384	0.313	0.763
Qwen3-4B	0.421	0.345	0.812

Table 2: Percentage of rollout groups with zero outcome variance during the initial training phase.

Figure 4: Mechanism diagnostics under ablations (aggregated across tasks).

J_{f}

denotes final extrinsic performance;

\Delta_{\mathrm{bf}}

is the best-to-final drop (late-training instability);

P_{\mathrm{cr}}

is the probability of training collapse;

\bar{T}

and

\bar{L}

are the average interaction turns and response length; and

\rho_{L,r}

is the correlation between response length and extrinsic reward (a proxy for length-based reward hacking). All metrics are converted to “higher-is-better” scores in the figure; formal definitions are in Appendix C .

Panel of Figure 5 — Figure 5: Interaction dynamics and reward signals. (a) Turns vs. response length; (b) Absolute info-gain (solid) and its advantage contribution ratio (dashed). See Appendix D.4 for more results.

Figure 6: Heatmap of turn-level info-gain rewards during training.

Panel of Figure 7 — Figure 7: Success rates on environment-interactive tasks. InfoPO maintains a stable upward learning trend on Sokoban and WebShop, demonstrating its robust generalization beyond user-centric scenarios.

User	UserGym	ColBench	$\tau^{2}$ -Bench
Base	0.552	0.480	0.173
Optim. Prompt (OP)	0.563	0.483	0.203
Optim. Model (OM)	0.558	0.502	0.215

Table 3: Comparison of different simulated users at test time. Base uses GPT-4o-mini with default prompts, OP uses optimized prompts, and OM uses a stronger model (GPT-4.1).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Info Gain Configuration
Parameter	UserGym	ColBench	$\tau^{2}$ -Bench
Intrinsic weight ( $\beta_{0}$ )	0.5	0.3	0.1
KL batch size	4	4	2
Gate temperature	0.5	0.5	0.05
Data Configuration
Train batch size	64	64	32

Appendix

Table 4: Key hyperparameters for training on UserGym, ColBench, and

\tau^{2}

-Bench environments.

Panel of Figure 8 — Figure 8: Robustness analysis of various masking strategies. We evaluate the sensitivity of internal representations (measured by KL divergence) across four placeholder designs: Default (String), Alternative String, Random Tokens, and Fixed Mask Token. (a-b) show that all strategies yield consistent distributions; (c) highlights minimal mean variation with t-test significance ( $p$ ); (d) demonstrates that the maximum relative performance gap remains within an acceptable threshold ( $<10\%$ ), confirming the robustness of our default design.

Panel of Figure 9 — Figure 9: Additional results. (a) Average response length and number of interaction turns on UserGym over training progress. (b) Average response length and number of interaction turns on $\tau^{2}$ -Bench over training progress. (c) Absolute info-gain reward and info-gain ratio on $\tau^{2}$ -Bench over training progress. (d) Sensitivity analysis of the Info-Gain reward weight ( $\beta$ ) varying from 0.1 to 2.0.

Closed-Source Model
Type	Method	Travel22	Travel33	Travel44	Travel233	Travel333	Travel334	Travel444	Travel2222
Prompting	Gemini-3-Flash	0.586	0.572	0.588	0.587	0.600	0.528	0.568	0.565
Prompting	GPT-4.1	0.586	0.543	0.612	0.548	0.555	0.566	0.532	0.494
Prompting	GPT-4o-mini	0.600	0.586	0.579	0.584	0.600	0.617	0.600	0.600
Qwen2.5-7B-Instruct
Prompting	Qwen2.5	0.372	0.457	0.331	0.485	0.475	0.472	0.447	0.494

Appendix

Table 5: Results on Travel subsets. Grey rows denote InfoPO and its variants.

Figure 10: Observation sensitivity grouped by the quality of the realized next action

a_{t+1}

\tau^{2}

-Bench. Good next actions show consistently higher sensitivity than meaningless ones, especially for tool calls, suggesting that the counterfactual signal mainly reflects task-relevant information use rather than arbitrary likelihood shift.

Current action type	Mean $r^{\mathrm{info}}_{t}$	Median $r^{\mathrm{info}}_{t}$
Good	0.380	0.247
Meaningless	0.017	0.012

Appendix

Table 6: Information-gain reward grouped by the quality of the current action

a_{t}

\tau^{2}

-Bench. Good actions receive much larger counterfactual credit than meaningless actions.

Figure 11: Effect of current-action quality on next-step sensitivity and continuation quality. Good current actions induce larger next-step sensitivity and are more likely to lead to good next actions, while meaningless actions mostly lead to low-sensitivity or meaningless continuations.

Method	Top-10% TC	Top-10% GOQ	Mean $r^{\mathrm{info}}$ TC	Mean $r^{\mathrm{info}}$ GOQ	Corr.(Len., $r^{\mathrm{info}}$ )	$\Delta_{\mathrm{LM}}$	Avg. Turns	Avg. Resp. Len.
InfoPO	58.8%	12.4%	0.3589	0.2029	0.157	0.319	10.4	105.3
InfoPO w/o Gate	32.3%	31.2%	0.2412	0.2723	0.352	-0.055	13.5	140.7

Appendix

Table 7: Reward-hacking diagnostics for high information-gain turns. TC denotes targeted clarification and GOQ denotes generic open question.

\Delta_{\mathrm{LM}}

denotes the length-matched information-gain gap between TC and GOQ.

Figure 12: InfoPO Successful Case (UserGym)

Figure 13: InfoPO Successful Case (ColBench)

Figure 14: InfoPO Successful Case ( $\tau^{2}$ -Bench)

Panel of Figure 15 — Figure 15: Optimized system instruction for TravelGym (Planning interaction).

Panel of Figure 16 — Figure 16: Optimized system instruction for IntentionGym (Intent elicitation).

Panel of Figure 17 — Figure 17: Optimized system instruction for TelepathyGym (Iterative guessing).

Panel of Figure 18 — Figure 18: Optimized simulation prompt for collaborative coding in ColBench.

Panel of Figure 19 — Figure 19: Optimized simulation guidelines for Airline and Retail domains in $\tau^{2}$ -Bench.

Panel of Figure 20 — Figure 20: Optimized simulation guidelines for the Telecom domain in $\tau^{2}$ -Bench, addressing tool-usage failure modes.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
Mar 21, 2026Haoyu Wang, Yuxin Chen, Liang Luo +3RL for Language Models Process Reward Models
Georgia Institute of Technology · Meta AI
IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
Aug 25, 2026Bo Ren, Yirong Mao, Yi Yang +1LLM Agent Training Credit Assignment in RL
WeChat, Tencent Inc.
A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
May 7, 2026Dingwei Chen, Zefang Zong, Zhipeng Ma +5Credit Assignment in RL Policy Gradient Methods
Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University

Mar 21, 2026J/K move · Enter open · S save

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang+2

Georgia Institute of Technology · Meta AI

RL for Language Models Process Reward Models Reward Modeling

Open paper AskPDF arXiv

Multi-turn human-AI collaboration is fundamental to deploying interactive services such as adaptive tutoring, conversational recommendation, and professional consultation. However, optimizing these interactions via reinforcement learning is hindered by the sparsity of verifiable intermediate rewards and the high stochasticity of user responses. To address these challenges, we introduce Implicit Turn-wise Policy Optimization (ITPO). ITPO leverages an implicit process reward model to derive fine-grained, turn-wise process rewards from sparse outcome signals. Unlike volatile token-level rewards, these turn-level signals exhibit superior robustness and may utilize a normalization mechanism to further enhance training stability. We evaluate ITPO across three representative multi-turn collaborative tasks: math tutoring, document writing, and medical recommendation. Empirical results demonstrate that ITPO, when combined with PPO, GRPO, or RLOO, consistently achieves improved convergence than existing baselines. Elaborate trajectory analysis confirms that ITPO infers turn-wise preferences that are semantically aligned with human judgment. Code is publicly available at https://github.com/Graph-COM/ITPO.

Abstract

LLM Agent Training

Explore similar work

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

A2^22TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping