IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

Bo Ren; Yirong Mao; Yi Yang; Wenhui Que

cs.LGAug 25, 2026

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

Authors: Bo Ren, Yirong Mao, Yi Yang, Wenhui Que

Organizations: WeChat, Tencent Inc.

Abstract

Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks: ${τ^2}$ -Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.

Figures & tables

Figure 1: Per-action advantage allocation for a successful (left, positive-advantage) and a failed (right, negative-advantage) refund rollout. Refunding to the original card or store balance is correct only if it matches the user’s clarified intent , so an early clarifying turn decides whether later actions are right. GRPO applies the same advantage to every action, whereas IAPO routes credit according to realized support use and error propagation.

Figure 2: Overview of IAPO. (A) Multiple rollouts are sampled per task and scored by the outcome reward. (B) A frozen annotator extracts signed influence edges among trainable actions, using user and tool observations as evidence. (C) The resulting graph features are normalized into bounded per-action weights that replace GRPO’s uniform advantage with sign-conditioned routing.

Model	Method	Think	$\tau^{2}$ (%)	UserBench (%)	AgentChangeBench (%)	BFCL-MT (%)	Avg (%)
GPT-4o-mini	–	–	23.75	28.30	35.50	37.00	31.14
Qwen3.5-397B	Base	w/	84.38	73.92	43.50	64.00	66.45
Qwen3-4B	Base	w/o	19.17	22.04	17.50	31.38	22.52
	Base	w/	31.46	25.33	25.00	30.50	28.07
	GRPO	w/o	34.79 $\pm$ 1.05	23.27 $\pm$ 0.47	21.50 $\pm$ 1.81	31.69 $\pm$ 0.13	27.81
	GiGPO	w/o	37.22 $\pm$ 1.04	25.06 $\pm$ 0.25	24.42 $\pm$ 1.13	32.42 $\pm$ 0.56	29.78

Table 1: Main results (%).

\pm

is the standard deviation over three seeds. † : results re-evaluated using publicly available checkpoint. ‡ : results copied from the published papers. IAPO uses only the 178-task

\tau^{2}

-Bench training split, while Fission-GRPO ( Zhang et al., 2026b ) uses 630 tasks across 11 domains grounded in BFCL characteristics , designed and cleaned with Claude Sonnet 4. ToolACE-2-8B ( Liu et al., 2025a ) and BitAgent-8B ( BitAgent, 2025 ) are specialized function-calling agents.

Benchmark	Split	GRPO	IAPO	$\Delta$
$\tau^{2}$	Airline	$22.78$	$29.72$	$+6.94$
	Retail	$44.31$	$56.88$	$+12.57$
	Telecom	$21.74$	$39.93$	$\mathbf{+18.19}$
UserBench	T22	$20.68$	$25.65$	$+4.97$
	T33	$19.41$	$24.50$	$\mathbf{+5.09}$
	T44	$19.10$	$22.55$	$+3.45$

Table 2: Per-split results at 8B (three-seed mean).

Axis	Setting	$\tau^{2}$	UB	AC
Ref.	GRPO	29.61 $\pm$ 3.56	19.83 $\pm$ 0.19	26.03 $\pm$ 1.79
$\beta_{+}$	$0$	32.50	20.92	27.58
	$.05$ def.	42.18 $\pm$ 2.55	24.44 $\pm$ 0.15	30.44 $\pm$ 2.20
	$.1$	33.96	25.33	30.92
	$.3$	31.39	24.08	28.83
$c$	$.125$	32.92	19.11	30.17

Table 3: Ablation of routing hyperparameters and the frozen annotator on Qwen3-8B (%).

Annotator	Edge agreement			Routing agreement
	F1 e	Jaccard	Cohen’s $\kappa$	$\rho(m^{+})$	$\rho(m^{-})$	Sign agreement
Qwen3-32B (ref.)	0.852	0.822	0.812	–	–	–
Qwen3.5-27B	0.875	0.846	0.833	0.769	1.000	0.858
Gemma4-31B	0.878	0.857	0.849	0.745	0.996	0.858
DeepSeek-V4-Flash	0.879	0.852	0.843	0.758	0.975	0.877
Qwen3-235B-A22B	0.860	0.836	0.809	0.732	0.984	0.848

Table 4: Cross-model annotation consistency. Edge agreement reports F1, Jaccard, and Cohen’s

\kappa

; routing agreement reports Spearman correlations with the Qwen3-32B reference and sign agreement of the routed advantages.

Figure 3: Banking case study with three successive goal shifts. (Left) GRPO anchors on a schema-example identity and propagates the wrong account state. (Right) IAPO grounds the correct customer from supplied credentials and retains the linked state across goals.

Model	Method	Multi- Turn	Base	Multi- Function	Multi- Parameter	Long
4B	Base (w/)	31.38	43.50	27.00	28.50	26.50
	GRPO	31.69	39.61	30.67	28.28	28.22
	GiGPO	32.42	31.83	32.33	32.67	32.83
	InfoPO	29.94	37.94	29.00	26.50	26.33
	IAPO	31.89	40.06	30.78	28.28	28.44
8B	Base (w/)	39.38	49.00	37.50	41.50	29.50

Table 5: BFCL-v4 Multi-Turn sub-split accuracy (%).

Explore similar work

A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
May 7, 2026Dingwei Chen, Zefang Zong, Zhipeng Ma +5Credit Assignment in RL Policy Gradient Methods
Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University
Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context
Aug 18, 2026Yiwen Zhao, Zhihao Wen, Yuchen Mao +5RL for LLM Agents Conversational Agents
Fudan University · Ant International, Ant Group
TCPO: Turn-Level Credit Policy Optimization
Aug 3, 2026Sicong Liao, Zhi Chen, Yaohua TangRL for Language Model Reasoning Credit Assignment in RL
Moore Threads AI

May 7, 2026J/K move · Enter open · S save

A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

Dingwei Chen, Zefang Zong, Zhipeng Ma, Leo Luo+4

Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University

Credit Assignment in RL Policy Gradient Methods RL Post-Training RL for LLM Agents RL for Tool Use

Open paper AskPDF arXiv

Reinforcement learning for agentic large language models (LLMs) typically relies on a sparse, trajectory-level outcome reward, making it difficult to evaluate the contribution of individual tool-calls within multi-turn interactions. Existing approaches to such process credit assignment either depend on separate external process reward models that introduce additional consumption, or tree-based structural rollout that merely redistributes the outcome signal while constraining trajectory diversity. A promising alternative leverages the per-turn change in the policy's predicted probability of the ground-truth, termed Information Gain (IG), as an intrinsic process signal without an external evaluator. However, prior work on leveraging IG signals within the RL training loop faces three systematic challenges: normalizing across turns that face heterogeneous positional contexts can distort the relative standing of individual turns, accumulating a variable number of terms causes advantage magnitudes to drift with trajectory depth, and a fixed clipping range governs policy updates identically for turns with vastly different IG signals. In this paper, we propose A

^2

TGPO (Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping), which retains IG as the intrinsic signal but re-designs how it is normalized, accumulated, and consumed: (i) turn-group normalization: normalizes IG within each (prompt, turn-index) group so that each turn is compared only against peers at the same interaction depth; (ii) variance-rescaled discounted accumulation: divides cumulative normalized IG by square root of accumulated terms to keep advantage magnitudes comparable across turn positions; and (iii) adaptive turn-level clipping: modulates each turn's clipping range based on its normalized IG, widening the update region for informative turns and narrowing it for uninformative ones.

Abstract

Explore similar work

A2^22TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

TCPO: Turn-Level Credit Policy Optimization

A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping