Not Every User Says It All: Training Tool-Using Agents Beyond Ideal Simulated Users

Han Bao; Zheyuan Zhang; Kaiwen Shi; Yanfang Ye

cs.AIOct 16, 2026

Not Every User Says It All: Training Tool-Using Agents Beyond Ideal Simulated Users

Authors: Han Bao, Zheyuan Zhang, Kaiwen Shi, Yanfang Ye

Organizations: University of Notre Dame

Abstract

Tool-using agents are trained and evaluated almost exclusively with strong simulated users who state every detail clearly, yet real users are often weaker. We construct weak users by masking sentences of a simulated user's replies, holding the task and the user's knowledge fixed, and cross-evaluate agents trained with strong and weak users on tau2-bench and ToolWOZ. Weak users lower the success of every policy, and training with them does not help: no policy trained with weak users, through fixed masks, mixtures or a curriculum, serves weak users better than the strong-trained policy beyond run-to-run noise, and prompting the agent to ask more does not help either. We propose StaGPO (State-Grouped Policy Optimization): the agent writes what the task needs, what is missing and how it is being filled, and GRPO uses this state to compare turns of different rollouts that reach the same situation, assigning step-level credit. Within the standard recipe of a small SFT cold start followed by RL, StaGPO makes weak-user training useful: on ToolWOZ, the Qwen3.5-2B policy trained with weak users reaches 76.8% with weak users, the highest of any Qwen3.5-2B policy we trained, 4.8 points above GRPO and 2.8 above the strongest baseline; it is also 5.4 points above GRPO with strong users, and generalises to an unseen mask rate; trained with strong users instead, it serves weak users at least as well as GRPO trained with weak users. On tau2-bench it scores highest with weak users for both models, by a small margin for Qwen3.5-4B.

Figures & tables

Figure 1: Training with strong and weak users on

\uptau^{2}

-bench and ToolWOZ , Qwen3.5-4B and 2B . Top: training success of each arm (moving average); bottom: success of each masked arm minus the unmasked arm, per epoch (shading

\pm

1 SE; grey: the ToolWOZ 2B arms diverge).

Figure 2: StaGPO . (a) The agent writes a hidden state: what the task needs , what is still missing ( gap ) and how it is being filled . (b) Turns of different rollouts with the same state key

\kappa

form an anchor group. (c) Each anchored turn is credited by its return relative to the group mean, added to GRPO ’s episode advantage.

Qwen3.5-4B
	$\uptau^{2}$ -bench					ToolWOZ
Trained with	$p{=}0$	0.2		0.4		$p{=}0$	0.2		0.4
none (untrained)	46.7	40.5	$\downarrow$ 6.2	32.0	$\downarrow$ 14.7	77.3	72.6	$\downarrow$ 4.7	69.0	$\downarrow$ 8.3
strong user	53.9	51.0	$\downarrow$ 2.9	41.2	$\downarrow$ 12.7	85.5	80.5	$\downarrow$ 5.0	74.6	$\downarrow$ 10.9
weak, $p{=}0.2$	45.1	39.2	$\downarrow$ 5.9	40.2	$\downarrow$ 4.9	84.0	82.4	$\downarrow$ 1.6	77.1	$\downarrow$ 6.9
weak, $p{=}0.4$	47.1	42.2	$\downarrow$ 4.9	39.2	$\downarrow$ 7.8	85.9	80.5	$\downarrow$ 5.4	74.8	$\downarrow$ 11.1

Table 1: Cross-evaluation of policies trained with strong and weak users. Success (%) by the user’s mask at evaluation (

p{=}0

: strong user); rows: the user at training; small numbers to the right: change from

p{=}0

. Standard deviations: Table 6 .

Figure 4: Change in success when the user becomes weak. Success at

p{=}0.4

minus success at

p{=}0

(points) for every policy of Table 1 . On

\uptau^{2}

-bench 4B the weak-trained policies lose less because they start lower with the strong user, not because they do better with the weak one.

Qwen3.5-2B
	$\uptau^{2}$ -bench						ToolWOZ
Evaluated at	$p{=}0$		0.2		0.4		$p{=}0$		0.2		0.4
Untrained	41.2		33.0		30.7		49.0		42.6		40.7
GRPO	34.4		30.2		24.0		78.7		74.3		72.0
Hybrid	30.7	$\downarrow$ 3.7	29.7	$\downarrow$ 0.5	27.6	$\uparrow$ 3.6	64.0	$\downarrow$ 14.7	57.1	$\downarrow$ 17.2	52.0	$\downarrow$ 20.0
Curriculum	45.8	$\uparrow$ 11.4	33.3	$\uparrow$ 3.1	33.3	$\uparrow$ 9.3	65.7	$\downarrow$ 13.0	65.1	$\downarrow$ 9.2	62.9	$\downarrow$ 9.1

Table 2: StaGPO against the baselines (trained at

p{=}0.4

). The

p{=}0.4

columns evaluate each policy with its training user (RQ2), the others with a less masked user (RQ3). GRPO starts from the base model, for Qwen3.5-2B with the settings of StaGPO , for Qwen3.5-4B as in Table 1 . Success (%); bold: best per column.

Figure 5: What does not help ( ToolWOZ ; exploratory single runs). (a) Gain over the masked dialogue in counterfactual forks (all 150 prefixes; asking once recovers 25% on the 105 audited ones). (b) Success minus an identical control for repair bonuses (4B) and state-accuracy rewards (2B). Bars:

\pm

1 SE.

Trained and evaluated at	$p{=}0$	0.2	0.4
Qwen3.5-2B
GRPO	74.9	74.8	72.0
UserRL	76.5	69.3	57.9
InfoPO	85.3	72.4	74.0
StaGPO	82.3 $\uparrow$ 7.4	74.8 $\pm$ 0.0	76.8 $\uparrow$ 4.8
Qwen3.5-4B

Table 3: Training mask ( ToolWOZ ). Each policy is trained and evaluated with the same user; GRPO : from the base model, at mask 0.4 for Qwen3.5-2B as in Table 2 , otherwise the policies of Table 1 ; small numbers: change from GRPO ; bold: best per column. Standard deviations: Table 10 .

Figure 6: Interaction with the weak user ( ToolWOZ , Qwen3.5-2B , trained and evaluated at

p{=}0.4

; final checkpoint, step 120; per test dialogue). GRPO starts from the same SFT cold start as StaGPO . Turn limit: share of dialogues that use all 16 agent turns.

Figure 7: Training with the weak user ( ToolWOZ , Qwen3.5-2B ,

p{=}0.4

): rollout success per step, rolling mean over 13 steps; shading:

\pm

1 SE. GRPO here starts from the same SFT cold start as StaGPO .

Figure 8: Transfer across users ( ToolWOZ , test success, %). Each policy is evaluated with a user it was not trained with; bold: best per setting.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Rollout	$t$ / $T$	$R$	$G_{i,t}$	$A^{\text{step}}$	$A$
1: asks for the day	4 / 9	1	0.774	$+$ 0.46	$+$ 1.46
2: books without it	5 / 16	0	0	$-$ 1.09	$-$ 2.09
3: asks for the day	4 / 7	1	0.857	$+$ 0.63	$+$ 1.63
Group mean $\bar{G}_{\mathcal{A}}$			0.544

Appendix

Table 4: A constructed anchor group with key (restaurant, book, {area, food, people}, results): three rollouts reach it at turn

t

T

;

\gamma{=}0.95

\sigma_{q}{=}0.5

\omega{=}1

Training user (epoch)	20b	120b	Mean
	Evaluation user
none (untrained)	35.3 $\,\pm$ 7.8	39.2 $\,\pm$ 7.4	37.3
gpt-oss-20b (4)	41.2 $\,\pm$ 2.9	36.3 $\,\pm$ 9.5	38.7
gpt-oss-120b (4)	37.3 $\,\pm$ 6.8	45.1 $\,\pm$ 7.4	41.2
gpt-oss-20b (8)	43.1 $\,\pm$ 4.5	44.1 $\,\pm$ 2.9	43.6
gpt-oss-120b (8)	46.1 $\,\pm$ 3.4	44.1 $\,\pm$ 5.9	45.1

Appendix

Table 5: Simulator scale ( Qwen3.5-4B ,

\uptau^{2}

-retail , 34 tasks; success, %,

\pm

standard deviation over three evaluation rounds; single training runs).

Qwen3.5-4B
		$\uptau^{2}$ -bench			ToolWOZ
Type	Method	$p{=}0$	0.2	0.4	$p{=}0$	0.2	0.4
Prompting	Untrained	46.7 $\,\pm$ 4.3	40.5 $\,\pm$ 4.6	32.0 $\,\pm$ 5.6	77.3 $\,\pm$ 2.7	72.6 $\,\pm$ 6.1	69.0 $\,\pm$ 4.6
RL, strong user	GRPO ( $p{=}0$ )	53.9 $\,\pm$ 3.4	51.0 $\,\pm$ 10.3	41.2 $\,\pm$ 8.8	85.5 $\,\pm$ 2.9	80.5 $\,\pm$ 1.1	74.6 $\,\pm$ 2.4
RL, weak user	GRPO ( $p{=}0.2$ )	45.1 $\,\pm$ 10.3	39.2 $\,\pm$ 6.1	40.2 $\,\pm$ 3.4	84.0 $\,\pm$ 0.2	82.4 $\,\pm$ 0.6	77.1 $\,\pm$ 1.4
RL, weak user	GRPO ( $p{=}0.4$ )	47.1 $\,\pm$ 10.6	42.2 $\,\pm$ 3.4	39.2 $\,\pm$ 1.7	85.9 $\,\pm$ 2.1	80.5 $\,\pm$ 0.3	74.8 $\,\pm$ 2.7

Appendix

Table 6: Full cross-evaluation ( Table 1 with standard deviations). Success (%)

\pm

standard deviation over evaluation rounds. Columns: the user’s mask at evaluation (

p{=}0

: strong); rows: the user at training.

Figure 9: Mean training reward per epoch, Qwen3.5-4B on

\uptau^{2}

-retail , each arm under its own mask.

Policy	Eval 0	Eval 0.2	Eval 0.4
Untrained ( $\times$ 3)	41.2	33.0	30.7
Strong-trained ( $\times$ 3)	40.2	36.6	33.3
Fixed $p{=}0.2$ ( $\times$ 1)	34.3	29.4	26.5
Fixed $p{=}0.4$ ( $\times$ 3)	44.4	40.5	38.9
0.4 $-$ strong-trained	$+$ 4.2	$+$ 3.9	$+$ 5.6
0.4 $-$ untrained	$+$ 3.3	$+$ 7.5	$+$ 8.2

Appendix

Table 7: Qwen3.5-2B on

\uptau^{2}

-retail , epoch-8 checkpoints; (

\times k

): evaluations pooled.

	gpt-oss-20b		gpt-oss-120b
	weak	strong	weak	strong
A $-$ control	$+$ 1.4 $\,\pm$ 2.4	$+$ 0.7 $\,\pm$ 2.7	$-$ 0.9 $\,\pm$ 2.0	$+$ 1.3 $\,\pm$ 2.4
B $-$ control	$+$ 0.9 $\,\pm$ 2.2	$+$ 1.8 $\,\pm$ 2.9	$-$ 1.4 $\,\pm$ 2.9	$+$ 0.5 $\,\pm$ 2.9

Appendix

Table 8: Repair bonus minus control ( Qwen3.5-4B ; points; task-paired,

\pm

half-width of an approximate 95% interval) on 100 held-out ToolWOZ tasks, by evaluation user; weak:

p{=}0.4

, 8 repetitions (B with gpt-oss-120b : 4); strong:

p{=}0

, 4 (2).

From the SFT cold start
	$\uptau^{2}$ -bench			ToolWOZ
Evaluated at	$p{=}0$	0.2	0.4	$p{=}0$	0.2	0.4
Untrained	41.2 $\,\pm$ 5.3	33.0 $\,\pm$ 8.2	30.7 $\,\pm$ 4.9	49.0 $\,\pm$ 2.4	42.6 $\,\pm$ 1.8	40.7 $\,\pm$ 3.9
GRPO	38.0 $\,\pm$ 3.5	34.9 $\,\pm$ 10.9	39.6 $\,\pm$ 7.0	70.5 $\,\pm$ 4.5	69.7 $\,\pm$ 5.5	62.5 $\,\pm$ 6.6
UserRL	31.2 $\,\pm$ 5.4	27.1 $\,\pm$ 5.4	28.1 $\,\pm$ 5.2	61.4 $\,\pm$ 0.6	60.4 $\,\pm$ 2.4	57.9 $\,\pm$ 0.3
StaGPO	45.8 $\,\pm$ 4.3	36.5 $\,\pm$ 3.0	40.6 $\,\pm$ 2.2	84.1 $\,\pm$ 3.2	81.5 $\,\pm$ 3.8	76.8 $\,\pm$ 1.7

Appendix

Table 9: Table 2 with standard deviations over evaluation rounds ( Qwen3.5-2B , trained at

p{=}0.4

Qwen3.5-2B
Trained and evaluated at	$p{=}0$	0.2	0.4
GRPO (base)	74.9 $\,\pm$ 0.2	74.8 $\,\pm$ 6.3	72.0 $\,\pm$ 4.3
SFT, GRPO	75.8 $\,\pm$ 5.5	74.3 $\,\pm$ 3.5	62.5 $\,\pm$ 6.6
UserRL (SFT)	76.5 $\,\pm$ 3.5	69.3 $\,\pm$ 0.8	57.9 $\,\pm$ 0.3
InfoPO (base)	85.3 $\,\pm$ 1.3	72.4 $\,\pm$ 2.4	74.0 $\,\pm$ 1.5
SFT, StaGPO	82.3 $\,\pm$ 7.0	74.8 $\,\pm$ 1.7	76.8 $\,\pm$ 1.7

Appendix

Table 10: Rows of Table 3 with standard deviations over evaluation rounds ( ToolWOZ ).

Qwen3.5-2B
			Test at selected step
Method	Train	Val	$p{=}0$	0.2	0.4
GRPO (from base)	0.4	45.0	34.4	30.2	24.0
SFT, GRPO	0.4	57.5	38.0	34.9	39.6
InfoPO (from base)	0.4	45.0	35.4	37.5	30.2
UserRL (SFT)	0.4	35.0	31.2	27.1	28.1

Appendix

Table 11:

\uptau^{2}

-retail : training runs, as in Table 12 ; validation on the 10 development tasks, test on the 24 test tasks

\times

4 seeds; Qwen3.5-4B runs compared at steps 20 and 40, except StaGPO at mask 0.4 (steps 20–60) and SFT, GRPO (steps 20–80).

Qwen3.5-2B
			Test at selected step
Method	Train	Val	$p{=}0$	0.2	0.4
GRPO (from base)	0.4	66.8	82.3	74.7	75.2
GRPO (from base)	0.4	67.5	76.2	75.7	69.1
GRPO (from base)	0.4	68.5	77.6	72.6	71.8
InfoPO (from base)	0	76.9	85.3	78.3	74.8

Appendix

Table 12: ToolWOZ: training runs. Val: validation score (30 tasks, training mask) of the checkpoint selected among steps 30/60/90/120; rows with the same method and mask are separate training runs; test: 70 tasks

\times

2 under each evaluation mask.

	$\uptau^{2}$ -bench			ToolWOZ
Eval. at	$p{=}0$	0.2	0.4	$p{=}0$	0.2	0.4
GRPO	34.4	30.2	24.0	78.7	74.3	72.0
InfoPO	35.4	37.5	30.2	78.3	75.7	74.0
StaGPO	32.3	31.8	35.4	74.8	72.2	67.1
StaGPO (SFT)	45.8	36.5	40.6	84.1	81.5	76.8

Appendix

Table 13: Without the cold start ( Qwen3.5-2B , trained from the base model with the weak user,

p{=}0.4

, as in Table 2 ); last row: StaGPO with the cold start, from Table 2 . Standard deviations: Table 9 .

	Succ.	Agent	User	Turn	Ques-
	(%) $\uparrow$	turns $\downarrow$	replies $\downarrow$	limit $\downarrow$	tions
GRPO	72.0	12.3	5.8	38	3.5
InfoPO	74.0	11.6	4.1	22	3.9
StaGPO	67.1	12.9	6.3	49	2.9

Appendix

Table 14: Interaction with the weak user without the cold start ( ToolWOZ , Qwen3.5-2B , trained and evaluated at

p{=}0.4

), as in Figure 6 .

	ToolWOZ		$\uptau^{2}$ -bench
Cold start	none	SFT	none	SFT
GRPO	72.0	62.5	24.0	39.6
StaGPO	67.1	76.8	35.4	40.6
Anchored, first 20 steps	24%	30%	28%	35%
Anchored, last 20 steps	19%	48%	20%	48%

Appendix

Table 15: Effect of the cold start ( Qwen3.5-2B , trained and evaluated at

p{=}0.4

). Anchored turns: share of StaGPO training turns that fall into an anchor group (SFT start: the higher of the two runs with complete logs).

State	Grouping	$p{=}0$	0.2	0.4
written	no ( $\omega{=}0$ )	69.0 $\,\pm$ 5.6	66.6 $\,\pm$ 2.1	60.1 $\,\pm$ 3.6
written	yes ( StaGPO )	84.1 $\,\pm$ 2.5	81.5 $\,\pm$ 2.6	76.8 $\,\pm$ 1.0
none	no ( GRPO )	70.5 $\,\pm$ 3.8	69.7 $\,\pm$ 6.0	62.5 $\,\pm$ 6.9

Appendix

Table 16: Grouping versus writing the state ( ToolWOZ , Qwen3.5-2B , SFT cold start, trained at

p{=}0.4

, 70 test tasks). Mean

\pm

standard deviation over three training runs.

Figure 10: Training curves of StaGPO and GRPO ( ToolWOZ , Qwen3.5-2B , both from the SFT cold start, trained at

p{=}0.4

). (a) Training reward and (b) KL divergence to the reference policy per training step (thin: mean over runs, bold: smoothed); (c) share of training dialogues that use all 16 agent turns, in bins of 10 steps. Shading: range over runs.

Figure 11: Test success at

p{=}0.4

at every candidate checkpoint ( ToolWOZ , Qwen3.5-2B , both from the SFT cold start; band: range over three runs).

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents
May 12, 2026Xiaolin Zhou, Aojie Yuan, Zheng Luo +12Domain Randomization AI Agent Benchmarks
Arizona State University · University of Southern California · University of Pennsylvania +2
UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
Sep 29, 2026Ashish Jain, Armaan SandhuAI Agent Benchmarks User Simulation
Sarvam AI
Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation
Oct 26, 2026Atri Vivek Sharma, Brian Formento, Alessio LomuscioLLM Agent Evaluation User Simulation
Imperial College London · Safe Intelligence

May 12, 2026J/K move · Enter open · S save

When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents

Xiaolin Zhou, Aojie Yuan, Zheng Luo, Zipeng Ling+11

Arizona State University · University of Southern California · University of Pennsylvania+2

Domain Randomization AI Agent Benchmarks Tool-Using Agents Robust RL RL for Tool Use

Open paper AskPDF arXiv

Tool-use language agents are evaluated on benchmarks that assume clean inputs, unambiguous tool registries, and reliable APIs. Real deployments violate all these assumptions: user typos propagate into hallucinated tool names, a misconfigured request timeout can stall an agent indefinitely, and duplicate tool names across servers can freeze an SDK. We study these failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP), where deployment noise enters through the observation, action space, reward-relevant metadata, or transition dynamics. We introduce RobustBench-TC, a benchmark with 22 perturbation types organized by these four POMDP components, each grounded in a verified GitHub issue or documented tool-calling failure. Across 21 models from 1.5B to 32B parameters (including the closed-source o4-mini), the robustness profile is sharply uneven: observation perturbations reduce accuracy by less than 5%, while reward-relevant and transition perturbations reduce accuracy by roughly 40% and 30%, respectively; scale alone does not close these gaps. We then propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories spanning the three statically encodable POMDP components. On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate perturbed accuracy comparable to open-source 14B function-calling baselines while substantially narrowing the gap to o4-mini. It closes approximately 27% of the Transition gap despite never seeing transition perturbations in training, suggesting that RL on adversarial static tool-use inputs induces a more persistent retry policy that transfers to unseen runtime failures. The dataset, code and benchmark leaderboard are publicly available.

Abstract

Tool-Using Agents

Explore similar work

When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation