Tool-using agents are trained and evaluated almost exclusively with strong simulated users who state every detail clearly, yet real users are often weaker. We construct weak users by masking sentences of a simulated user's replies, holding the task and the user's knowledge fixed, and cross-evaluate agents trained with strong and weak users on tau2-bench and ToolWOZ. Weak users lower the success of every policy, and training with them does not help: no policy trained with weak users, through fixed masks, mixtures or a curriculum, serves weak users better than the strong-trained policy beyond run-to-run noise, and prompting the agent to ask more does not help either. We propose StaGPO (State-Grouped Policy Optimization): the agent writes what the task needs, what is missing and how it is being filled, and GRPO uses this state to compare turns of different rollouts that reach the same situation, assigning step-level credit. Within the standard recipe of a small SFT cold start followed by RL, StaGPO makes weak-user training useful: on ToolWOZ, the Qwen3.5-2B policy trained with weak users reaches 76.8% with weak users, the highest of any Qwen3.5-2B policy we trained, 4.8 points above GRPO and 2.8 above the strongest baseline; it is also 5.4 points above GRPO with strong users, and generalises to an unseen mask rate; trained with strong users instead, it serves weak users at least as well as GRPO trained with weak users. On tau2-bench it scores highest with weak users for both models, by a small margin for Qwen3.5-4B.
Figures & tables
Figure 1: Training with strong and weak users on \uptau2 -bench and ToolWOZ , Qwen3.5-4B and 2B . Top: training success of each arm (moving average); bottom: success of each masked arm minus the unmasked arm, per epoch (shading ± 1 SE; grey: the ToolWOZ 2B arms diverge).
Figure 2: StaGPO . (a) The agent writes a hidden state: what the task needs , what is still missing ( gap ) and how it is being filled . (b) Turns of different rollouts with the same state key κ form an anchor group. (c) Each anchored turn is credited by its return relative to the group mean, added to GRPO ’s episode advantage.
\uptau2 -bench
ToolWOZ
Trained with
p=0
0.2
0.4
p=0
0.2
0.4
Qwen3.5-4B
none (untrained)
46.7
40.5
↓ 6.2
32.0
↓ 14.7
77.3
72.6
↓ 4.7
69.0
↓ 8.3
strong user
53.9
51.0
↓ 2.9
41.2
↓ 12.7
85.5
80.5
↓ 5.0
74.6
↓ 10.9
weak, p=0.2
45.1
39.2
↓ 5.9
40.2
↓ 4.9
84.0
82.4
↓ 1.6
77.1
↓ 6.9
weak, p=0.4
47.1
42.2
↓ 4.9
39.2
↓ 7.8
85.9
80.5
↓ 5.4
74.8
↓ 11.1
Table 1: Cross-evaluation of policies trained with strong and weak users. Success (%) by the user’s mask at evaluation ( p=0 : strong user); rows: the user at training; small numbers to the right: change from p=0 . Standard deviations: Table 6 .
Figure 4: Change in success when the user becomes weak. Success at p=0.4 minus success at p=0 (points) for every policy of Table 1 . On \uptau2 -bench 4B the weak-trained policies lose less because they start lower with the strong user, not because they do better with the weak one.
\uptau2 -bench
ToolWOZ
Evaluated at
p=0
0.2
0.4
p=0
0.2
0.4
Qwen3.5-2B
Untrained
41.2
33.0
30.7
49.0
42.6
40.7
GRPO
34.4
30.2
24.0
78.7
74.3
72.0
Hybrid
30.7
↓ 3.7
29.7
↓ 0.5
27.6
↑ 3.6
64.0
↓ 14.7
57.1
↓ 17.2
52.0
↓ 20.0
Curriculum
45.8
↑ 11.4
33.3
↑ 3.1
33.3
↑ 9.3
65.7
↓ 13.0
65.1
↓ 9.2
62.9
↓ 9.1
Table 2: StaGPO against the baselines (trained at p=0.4 ). The p=0.4 columns evaluate each policy with its training user (RQ2), the others with a less masked user (RQ3). GRPO starts from the base model, for Qwen3.5-2B with the settings of StaGPO , for Qwen3.5-4B as in Table 1 . Success (%); bold: best per column.
Figure 5: What does not help ( ToolWOZ ; exploratory single runs). (a) Gain over the masked dialogue in counterfactual forks (all 150 prefixes; asking once recovers 25% on the 105 audited ones). (b) Success minus an identical control for repair bonuses (4B) and state-accuracy rewards (2B). Bars: ± 1 SE.
Trained and evaluated at
p=0
0.2
0.4
Qwen3.5-2B
GRPO
74.9
74.8
72.0
UserRL
76.5
69.3
57.9
InfoPO
85.3
72.4
74.0
StaGPO
82.3 ↑ 7.4
74.8 ± 0.0
76.8 ↑ 4.8
Qwen3.5-4B
Table 3: Training mask ( ToolWOZ ). Each policy is trained and evaluated with the same user; GRPO : from the base model, at mask 0.4 for Qwen3.5-2B as in Table 2 , otherwise the policies of Table 1 ; small numbers: change from GRPO ; bold: best per column. Standard deviations: Table 10 .
Figure 6: Interaction with the weak user ( ToolWOZ , Qwen3.5-2B , trained and evaluated at p=0.4 ; final checkpoint, step 120; per test dialogue). GRPO starts from the same SFT cold start as StaGPO . Turn limit: share of dialogues that use all 16 agent turns.
Figure 7: Training with the weak user ( ToolWOZ , Qwen3.5-2B , p=0.4 ): rollout success per step, rolling mean over 13 steps; shading: ± 1 SE. GRPO here starts from the same SFT cold start as StaGPO .
Figure 8: Transfer across users ( ToolWOZ , test success, %). Each policy is evaluated with a user it was not trained with; bold: best per setting.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Rollout
t / T
R
Gi,t
Astep
A
1: asks for the day
4 / 9
1
0.774
+ 0.46
+ 1.46
2: books without it
5 / 16
0
0
− 1.09
− 2.09
3: asks for the day
4 / 7
1
0.857
+ 0.63
+ 1.63
Group mean GˉA
0.544
Appendix
Table 4: A constructed anchor group with key (restaurant, book, {area, food, people}, results): three rollouts reach it at turn t of T ; γ=0.95 , σq=0.5 , ω=1 .
Evaluation user
Training user (epoch)
20b
120b
Mean
none (untrained)
35.3 ± 7.8
39.2 ± 7.4
37.3
gpt-oss-20b (4)
41.2 ± 2.9
36.3 ± 9.5
38.7
gpt-oss-120b (4)
37.3 ± 6.8
45.1 ± 7.4
41.2
gpt-oss-20b (8)
43.1 ± 4.5
44.1 ± 2.9
43.6
gpt-oss-120b (8)
46.1 ± 3.4
44.1 ± 5.9
45.1
Appendix
Table 5: Simulator scale ( Qwen3.5-4B , \uptau2 -retail , 34 tasks; success, %, ± standard deviation over three evaluation rounds; single training runs).
\uptau2 -bench
ToolWOZ
Type
Method
p=0
0.2
0.4
p=0
0.2
0.4
Qwen3.5-4B
Prompting
Untrained
46.7 ± 4.3
40.5 ± 4.6
32.0 ± 5.6
77.3 ± 2.7
72.6 ± 6.1
69.0 ± 4.6
RL, strong user
GRPO ( p=0 )
53.9 ± 3.4
51.0 ± 10.3
41.2 ± 8.8
85.5 ± 2.9
80.5 ± 1.1
74.6 ± 2.4
RL, weak user
GRPO ( p=0.2 )
45.1 ± 10.3
39.2 ± 6.1
40.2 ± 3.4
84.0 ± 0.2
82.4 ± 0.6
77.1 ± 1.4
RL, weak user
GRPO ( p=0.4 )
47.1 ± 10.6
42.2 ± 3.4
39.2 ± 1.7
85.9 ± 2.1
80.5 ± 0.3
74.8 ± 2.7
Appendix
Table 6: Full cross-evaluation ( Table 1 with standard deviations). Success (%) ± standard deviation over evaluation rounds. Columns: the user’s mask at evaluation ( p=0 : strong); rows: the user at training.
Figure 9: Mean training reward per epoch, Qwen3.5-4B on \uptau2 -retail , each arm under its own mask.
Table 8: Repair bonus minus control ( Qwen3.5-4B ; points; task-paired, ± half-width of an approximate 95% interval) on 100 held-out ToolWOZ tasks, by evaluation user; weak: p=0.4 , 8 repetitions (B with gpt-oss-120b : 4); strong: p=0 , 4 (2).
\uptau2 -bench
ToolWOZ
Evaluated at
p=0
0.2
0.4
p=0
0.2
0.4
Untrained
41.2 ± 5.3
33.0 ± 8.2
30.7 ± 4.9
49.0 ± 2.4
42.6 ± 1.8
40.7 ± 3.9
From the SFT cold start
GRPO
38.0 ± 3.5
34.9 ± 10.9
39.6 ± 7.0
70.5 ± 4.5
69.7 ± 5.5
62.5 ± 6.6
UserRL
31.2 ± 5.4
27.1 ± 5.4
28.1 ± 5.2
61.4 ± 0.6
60.4 ± 2.4
57.9 ± 0.3
StaGPO
45.8 ± 4.3
36.5 ± 3.0
40.6 ± 2.2
84.1 ± 3.2
81.5 ± 3.8
76.8 ± 1.7
Appendix
Table 9: Table 2 with standard deviations over evaluation rounds ( Qwen3.5-2B , trained at p=0.4 ).
Trained and evaluated at
p=0
0.2
0.4
Qwen3.5-2B
GRPO (base)
74.9 ± 0.2
74.8 ± 6.3
72.0 ± 4.3
SFT, GRPO
75.8 ± 5.5
74.3 ± 3.5
62.5 ± 6.6
UserRL (SFT)
76.5 ± 3.5
69.3 ± 0.8
57.9 ± 0.3
InfoPO (base)
85.3 ± 1.3
72.4 ± 2.4
74.0 ± 1.5
SFT, StaGPO
82.3 ± 7.0
74.8 ± 1.7
76.8 ± 1.7
Appendix
Table 10: Rows of Table 3 with standard deviations over evaluation rounds ( ToolWOZ ).
Test at selected step
Method
Train
Val
p=0
0.2
0.4
Qwen3.5-2B
GRPO (from base)
0.4
45.0
34.4
30.2
24.0
SFT, GRPO
0.4
57.5
38.0
34.9
39.6
InfoPO (from base)
0.4
45.0
35.4
37.5
30.2
UserRL (SFT)
0.4
35.0
31.2
27.1
28.1
Appendix
Table 11: \uptau2 -retail : training runs, as in Table 12 ; validation on the 10 development tasks, test on the 24 test tasks × 4 seeds; Qwen3.5-4B runs compared at steps 20 and 40, except StaGPO at mask 0.4 (steps 20–60) and SFT, GRPO (steps 20–80).
Test at selected step
Method
Train
Val
p=0
0.2
0.4
Qwen3.5-2B
GRPO (from base)
0.4
66.8
82.3
74.7
75.2
GRPO (from base)
0.4
67.5
76.2
75.7
69.1
GRPO (from base)
0.4
68.5
77.6
72.6
71.8
InfoPO (from base)
0
76.9
85.3
78.3
74.8
Appendix
Table 12: ToolWOZ: training runs. Val: validation score (30 tasks, training mask) of the checkpoint selected among steps 30/60/90/120; rows with the same method and mask are separate training runs; test: 70 tasks × 2 under each evaluation mask.
\uptau2 -bench
ToolWOZ
Eval. at
p=0
0.2
0.4
p=0
0.2
0.4
GRPO
34.4
30.2
24.0
78.7
74.3
72.0
InfoPO
35.4
37.5
30.2
78.3
75.7
74.0
StaGPO
32.3
31.8
35.4
74.8
72.2
67.1
StaGPO (SFT)
45.8
36.5
40.6
84.1
81.5
76.8
Appendix
Table 13: Without the cold start ( Qwen3.5-2B , trained from the base model with the weak user, p=0.4 , as in Table 2 ); last row: StaGPO with the cold start, from Table 2 . Standard deviations: Table 9 .
Succ.
Agent
User
Turn
Ques-
(%) ↑
turns ↓
replies ↓
limit ↓
tions
GRPO
72.0
12.3
5.8
38
3.5
InfoPO
74.0
11.6
4.1
22
3.9
StaGPO
67.1
12.9
6.3
49
2.9
Appendix
Table 14: Interaction with the weak user without the cold start ( ToolWOZ , Qwen3.5-2B , trained and evaluated at p=0.4 ), as in Figure 6 .
ToolWOZ
\uptau2 -bench
Cold start
none
SFT
none
SFT
GRPO
72.0
62.5
24.0
39.6
StaGPO
67.1
76.8
35.4
40.6
Anchored, first 20 steps
24%
30%
28%
35%
Anchored, last 20 steps
19%
48%
20%
48%
Appendix
Table 15: Effect of the cold start ( Qwen3.5-2B , trained and evaluated at p=0.4 ). Anchored turns: share of StaGPO training turns that fall into an anchor group (SFT start: the higher of the two runs with complete logs).
State
Grouping
p=0
0.2
0.4
written
no ( ω=0 )
69.0 ± 5.6
66.6 ± 2.1
60.1 ± 3.6
written
yes ( StaGPO )
84.1 ± 2.5
81.5 ± 2.6
76.8 ± 1.0
none
no ( GRPO )
70.5 ± 3.8
69.7 ± 6.0
62.5 ± 6.9
Appendix
Table 16: Grouping versus writing the state ( ToolWOZ , Qwen3.5-2B , SFT cold start, trained at p=0.4 , 70 test tasks). Mean ± standard deviation over three training runs.
Figure 10: Training curves of StaGPO and GRPO ( ToolWOZ , Qwen3.5-2B , both from the SFT cold start, trained at p=0.4 ). (a) Training reward and (b) KL divergence to the reference policy per training step (thin: mean over runs, bold: smoothed); (c) share of training dialogues that use all 16 agent turns, in bins of 10 steps. Shading: range over runs.
Figure 11: Test success at p=0.4 at every candidate checkpoint ( ToolWOZ , Qwen3.5-2B , both from the SFT cold start; band: range over three runs).
Tool-use language agents are evaluated on benchmarks that assume clean inputs, unambiguous tool registries, and reliable APIs. Real deployments violate all these assumptions: user typos propagate into hallucinated tool names, a misconfigured request timeout can stall an agent indefinitely, and duplicate tool names across servers can freeze an SDK. We study these failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP), where deployment noise enters through the observation, action space, reward-relevant metadata, or transition dynamics. We introduce RobustBench-TC, a benchmark with 22 perturbation types organized by these four POMDP components, each grounded in a verified GitHub issue or documented tool-calling failure. Across 21 models from 1.5B to 32B parameters (including the closed-source o4-mini), the robustness profile is sharply uneven: observation perturbations reduce accuracy by less than 5%, while reward-relevant and transition perturbations reduce accuracy by roughly 40% and 30%, respectively; scale alone does not close these gaps. We then propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories spanning the three statically encodable POMDP components. On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate perturbed accuracy comparable to open-source 14B function-calling baselines while substantially narrowing the gap to o4-mini. It closes approximately 27% of the Transition gap despite never seeing transition perturbations in training, suggesting that RL on adversarial static tool-use inputs induces a more persistent retry policy that transfers to unseen runtime failures. The dataset, code and benchmark leaderboard are publicly available.
Xiaolin Zhou, Aojie Yuan, Zheng Luo +12
Arizona State University · University of Southern California · University of Pennsylvania +2
Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated user controls what information the agent receives and when, yet current benchmarks score only the agent and do not directly measure whether the user correctly executed its assigned role. We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success. Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks changes mean task reward by 15.2 points, while 24.4% of successful episodes contain a user-specification violation. The dominant failure is premature disclosure: users provide information before it is requested. This behavior has little effect on task reward, yet among successful episodes it causes the agent to make 1.06 fewer tool calls on average, changing the interaction being evaluated while preserving the reward. Finally, across seven proxies we identify an empirical cost-fidelity frontier, enabling practitioners to select the least expensive simulator that satisfies a required fidelity level.
Multi-turn agents are commonly evaluated with LLM-based simulated users, which can be overly cooperative and can therefore overestimate an agent's performance. We here introduce StressSim, an intent-preserving adversarial user simulator framework for stress-testing agents with challenging yet valid user interactions. StressSim optimises the simulator's high-level adversarial plan for each task instance. Candidate planner prompts are evolved with a genetic prompt-optimisation loop that uses rollout traces, granular task-completion signals, and validity feedback to favour conversations that reduce task success without changing the user's underlying goal. We evaluate the approach on the MultiWOZ dataset and the retail and airline domains from tau-bench with three target LLM agents, comparing against standard simulators and hand-designed non-collaborative user behaviours. We demonstrate the effectiveness of StressSim in exposing robustness gaps across frontier LLMs, with tau-bench pass^4 scores decreasing by up to 69% on gemini-3.8-flash and 62% on gpt-5.6-sol. StressSim maintains goal preservation performance (which decreases by at most 10%), using an optimisation budget of 20 rollouts per task. Sensitivity analysis suggests that increasing the budget further decreases task success. Overall our results demonstrate that StressSim exposes robustness gaps missed by standard and hand-designed user simulators while maintaining conversation realism. This is achieved through lightweight prompt optimisation alone, without model fine-tuning.
Atri Vivek Sharma, Brian Formento, Alessio Lomuscio