Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

Yiwen Zhao; Zhihao Wen; Yuchen Mao; Mingxuan Jiang; Yihao Hu; Pan Wang; Xin Zhang; Wei Wu

cs.AIAug 18, 2026

Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

Authors: Yiwen Zhao, Zhihao Wen, Yuchen Mao, Mingxuan Jiang, Yihao Hu, Pan Wang, Xin Zhang, Wei Wu

Organizations: Fudan University · Ant International, Ant Group

Abstract

User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce \textbf{F}eedback-\textbf{A}ware \textbf{C}redit \textbf{A}ssignment (\textsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, \textsc{FACA} improves the nine-domain $τ$ -family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

Figures & tables

Figure 1: Multi-turn tool interaction. A user-to-user segment contains all agent responses, tool calls, and tool results between two adjacent user turns; the ellipsis denotes continued interaction.

Figure 2: FACA overview. (a) For each prompt, we sample $K$ multi-turn user-interacting rollouts and obtain a verified final task reward. (b) A U2U segment may contain multiple agent and tool spans; the next user call jointly produces a visible utterance and private reaction strategy, which is aligned with the complete preceding segment. (c) Terminal rewards yield trajectory-level outcome advantages, while aligned reactions yield turn-level process advantages. Their additive combination supplies the final advantage for agent optimization.

Strategy event	Interpretation	$\boldsymbol{f}$
confirm	progress	$+1$
close	completion	$+1$
reveal-piece	requested state elicited	$+1$
be-vague	ambiguous	$0$
invalid	malformed	$0$
ask-clarification	local interaction friction	$-1$

Table 1: User-strategy mapping used by Faca . Polarity represents coarse evidence of interaction progress or friction, not sentiment or verified task correctness.

Model	$\boldsymbol{\tau}$ -bench		$\boldsymbol{\tau}^{\mathbf{2}}$ -bench			$\boldsymbol{\tau}^{\mathbf{3}}$ -bench				Avg.
Model	Air.	Ret.	Air.	Ret.	Tel.	Air.	Ret.	Tel.	Ban.	Avg.
Qwen3-8B	20.0	47.0	14.0	40.4	4.4	14.0	36.0	21.1	3.1	22.2
+ SFT	14.0	39.1	24.0	36.0	2.6	16.0	40.4	28.9	3.1	22.7
+ Interactive GRPO	28.0	52.2	32.0	46.5	30.7	39.3	50.0	29.8	3.4	34.7
+ Faca	37.3	43.8	38.0	54.4	41.2	40.0	53.5	53.5	3.4	40.6
Qwen3-14B	12.0	59.1	26.0	49.1	2.6	14.0	50.0	23.7	3.1	26.6

Table 2: Strict pass@1 (%) across nine domains in the

\tau

-bench family. RL scores average three independently trained step-120 runs per method and scale; each seed is evaluated once under one shared protocol. Base and SFT are fixed references. Avg. weights all nine domains equally. Full nominal task sets are used, and missing cases count as failures. Higher is better. “+” marks a training stage; the two RL rows are alternative continuations of the same SFT. Values use one decimal; bold marks column bests within each scale, including ties.

Condition	$\boldsymbol{\lambda}$	Avg.	Tel. 2
Aligned	0.50	39.37	47.37
Aligned	0.10	36.78	45.61
Aligned	$-0.50$	15.08	17.11
Shifted	0.50	33.92	29.39
Randomized	0.50	32.66	26.32
Outcome-only	0.00	34.70	30.26

Table 4: Single-run controlled 8B credit ablation across the

\tau

-bench family under the shared one-shot evaluation protocol. Avg. is the nine-domain average; Tel. 2 averages

\tau^{2}

and

\tau^{3}

Telecom.

Figure 4: Matched 8B Faca replication under identical training conditions: per-U2U reaction polarity and nonzero process-advantage coverage across 120 steps over three phases.

Figure 5: Change in outcome-homogeneous group rates from the first to the last 20 training steps. Colors distinguish all-correct and all-wrong groups; hollow markers denote steps 1–20 and filled markers steps 101–120 at both model scales.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	T1 Air.	T1 Ret.	T2 Air.	T2 Ret.	T2 Tel.	T3 Air.	T3 Ret.	T3 Tel.	T3 Bank	Pooled	Avg.
Base	10/50	54/115	7/50	46/114	5/114	7/50	41/114	24/114	3/97	197/818	22.20
MUA-SFT	7/50	45/115	12/50	41/114	3/114	8/50	46/114	33/114	3/97	198/818	22.68
Interactive GRPO (s1)	14/50	60/115	16/50	53/114	35/114	20/50	57/114	34/114	3/97	292/818	34.70
Interactive GRPO (s2)	15/50	58/115	15/50	55/114	33/114	20/50	59/114	36/114	3/97	294/818	34.89
Interactive GRPO (s3)	13/50	62/115	17/50	51/114	37/114	19/50	55/114	32/114	4/97	290/818	34.39
Faca (s1)	17/50	49/115	18/50	62/114	47/114	18/50	61/114	61/114	3/97	336/818	39.37

Appendix

Table 5: Raw counts from the independent training runs underlying Table 2 . For each RL method, s1–s3 are distinct training seeds that produce separate step-120 checkpoints, each evaluated once. The seed labels align the two RL methods by training-seed identifier; the main table averages these runs domainwise. Base and SFT are fixed pre-RL references. Pooled counts sum successes across unequal domain sizes and are therefore not the primary metric; Avg. gives each of the nine domains equal weight. All strict scores retain the nominal denominator.

Metric	Count unit	8B (120 steps)	14B (120 steps)
Training coverage	steps $\times$ rollouts/step	120 $\times$ 128	120 $\times$ 128
Complete trajectories	trajectory	15,360	15,360
All-correct groups	group-step	224 (11.67%)	212 (11.04%)
All-wrong groups	group-step	530 (27.60%)	384 (20.00%)
Mixed groups	group-step	1,166 (60.73%)	1,324 (68.96%)
Outcome-homogeneous groups	group-step	754 (39.27%)	596 (31.04%)

Appendix

Table 6: Recoverable telemetry from the step-120 Faca runs. A group-step is one prompt group at one optimizer step (16 groups per step). Singleton statistics count segments assigned to singleton anchors; they are not a nonconstant-anchor rate. Process quantities use

0.5\,\mathbb{E}|A^{\mathrm{p}}|

and

\mathbb{E}|A^{\mathrm{o}}|

before PPO clipping and are advantage-magnitude proxies rather than optimizer-update or gradient mass.

Figure 6: Process-credit share over the common first 120 training steps. Faint lines show per-step values and solid lines a 10-step moving average. The quantity is a pre-optimization advantage L1 proxy, not optimizer-update or gradient mass.

Strategy	Polarity	Steps 1–40	Steps 41–80	Steps 81–120	Overall
reveal_piece	positive	36.30	36.90	36.18	36.46
confirm	positive	26.48	32.55	37.23	31.91
be_vague	neutral	17.23	11.37	8.18	12.41
close	positive	7.17	8.11	9.10	8.10
ask_clarification	negative	5.33	4.11	3.05	4.20
challenge_solution	negative	3.10	2.81	2.39	2.78

Appendix

Table 7: Complete per-U2U strategy distribution (%) in the telemetry-enabled 8B replication. Unlike a terminal representative field, every row here uses the U2U segment as its verified count unit. The largest phasewise changes are confirm (

+10.75

points) and be_vague (

-9.06

points).

Benchmark	Subset	$N$	Faca	Int. GRPO	FACA-only / GRPO-only
$\tau^{2}$ Telecom	Overall	114	96/114	30/114	73 / 7
	Mobile	36	32/36	14/36	20 / 2
	Service	29	23/29	8/29	19 / 4
	MMS	49	41/49	8/49	34 / 1
$\tau^{3}$ Telecom	Overall	114	95/114	58/114	48 / 11
	Mobile	36	33/36	22/36	13 / 2

Appendix

Table 8: Final-14B paired Telecom audit on the s1 run-level evaluation. The two methods share task IDs, evaluation seeds, initial states, criteria, and scenarios. The last column reports discordant task counts. The

\tau^{3}

Service reversal shows that the checkpoint-level gain is not uniform within Telecom.

Explore similar work

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
Aug 25, 2026Bo Ren, Yirong Mao, Yi Yang +1LLM Agent Training Credit Assignment in RL
WeChat, Tencent Inc.
PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems
Aug 28, 2026Hanglong Lv, Dawei Zhu, Lei Li +11User Simulation LLM Agent Training
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core, Xiaomi · The University of Hong Kong +1
ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction
Aug 13, 2026Yongqi Tong, Tan Li Hui Faith, Choy Zhen Wen Marcus +5RL for LLM Agents Reward Design for RL
Ant International

Aug 25, 2026J/K move · Enter open · S save

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

Bo Ren, Yirong Mao, Yi Yang, Wenhui Que

WeChat, Tencent Inc.

LLM Agent Training Credit Assignment in RL LLM Agents RL for LLM Agents Step-Level Credit Assignment in RL

Open paper AskPDF arXiv

Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks:

{τ^2}

-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.

Abstract

Explore similar work

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction