Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging due to sparse outcome rewards and difficult credit assignment across conversation turns. We present the first application of MT-GRPO (Multi-Turn Group Relative Policy Optimization) combined with GTPO (Generalized Token-level Policy Optimization) for training a tool-calling agent on realistic customer service tasks with an LLM-based user simulator. Through systematic analysis of training rollouts, we discover that naively designed dense per-turn rewards degrade performance by up to 14 percentage points due to misalignment between reward discriminativeness and advantage direction. We introduce Iterative Reward Calibration, a methodology for designing per-turn rewards using empirical discriminative analysis of rollout data, and show that our GTPO hybrid advantage formulation eliminates the advantage misalignment problem. Applied to the Tau-Bench airline benchmark, our approach improves Qwen3.5-4B from 63.8 percent to 66.7 percent (+2.9pp) and Qwen3-30B-A3B from 58.0 percent to 69.5 percent (+11.5pp) -- with the trained 4B model exceeding GPT-4.1 (49.4 percent) and GPT-4o (42.8 percent) despite being 50 times smaller, and the 30.5B MoE model approaching Claude Sonnet 4.5 (70.0 percent). To our knowledge, these are the first published RL training results on Tau-Bench. We release our code, reward calibration analysis, and training recipes.
Figures & tables
Figure 1: Comparison of reward-to-advantage signal across three approaches, shown for a failing rollout ( R=0 ) with 5 turns. Top row : per-turn reward values; bottom row : resulting training advantage (green = reinforce, red = suppress, gray = zero gradient). (a) GRPO uses outcome-only reward—all turns get the same uniform advantage, providing no credit assignment. (b) MT-GRPO with naïve dense rewards (e.g., read-only = 0.3) suffers from advantage misalignment: the outcome advantage AO overwhelms small per-turn advantages, causing necessary read-only turns to be suppressed (red box). (c) Our IRC method calibrates rewards using discriminative analysis (right panel): read-only gets r=0 (non-discriminative), focusing gradient entirely on gold actions. GTPO hybrid dampens AO via λ=0.3 , eliminating all advantage mismatches.
Tier
AI
AI+AO
Aligned?
Gold exact
+1.22
+1.22
✓
Soft match
+0.11
− 0.11
×
Read-only
+0.05
− 0.65
×
State-change
+0.03
− 1.45
✓ *
Error
− 0.15
− 0.15
✓
Table 1: Advantage direction analysis under standard MT-GRPO with dense rewards. Read-only and soft match show misalignment: AI reinforces but AI+AO suppresses. * State-change suppression is correct (98.5% occur in failing rollouts).
Method
Mis-
Corr
Dead
matches
(adv,out)
turns
MT-GRPO (V5)
2
0.836
11.0%
GTPO γ =0.9
0
0.414
1.1%
Hybrid γ =0.9, λ =0.3
0
0.489
1.4%
Table 2: Advantage formulation comparison on 5,952 V5 rollouts. The hybrid combines zero mismatches (from GTPO) with reasonable outcome correlation (from λ -dampened AO ).
Gradient Target
Sparse
Dense
Gold + Soft (useful)
86.4%
47.5%
Read + State (noisy)
0.0%
26.5%
Dead (zero gradient)
27.5%
11.0%
Table 3: Gradient allocation by target type. Sparse rewards naturally focus gradient on outcome-relevant turns.
Tier
Pass%
Fail%
Gap
Action
Gold exact
68.4
1.3
+67.1
Keep 1.0
Soft match
54.2
45.8
+8.4
Keep 0.5+
Read-only
50.1
50.0
+0.1
0.3 → 0.0
State-chg
1.0
2.6
− 1.6
0.1 →− 0.1
Error
12.0
88.0
− 76.0
Keep − 0.1
Table 4: Discriminative power of each reward tier. Gap = frequency in passing − failing rollouts. Read-only has near-zero discriminative power (+0.1pp) and was reduced to 0.0. State-change was flipped to − 0.1.
Model
Size
Pass%
Δ
Frontier models (proprietary)
GPT-4.1 nano
—
14.0
—
Claude 3.5 Haiku
—
22.8
—
GPT-4o
—
42.8
—
GPT-4.1
—
49.4
—
Claude Sonnet 4.5
—
70.0
—
Table 5: Tau-Bench airline pass rates. Both MT-GRPO and IRC consistently improve both model scales, with IRC providing additional gains over MT-GRPO alone: +2.9pp for 4B and +11.5pp for 30.5B MoE. The trained 4B model exceeds GPT-4.1 and GPT-4o despite being ∼ 50 times smaller.
Version
Reward Design
LR
KL
Steps
Tau2 Pass
Key Finding
Base
No training
—
—
0
63.8%
Strong base model
MT-GRPO
Sparse (outcome only)
2e-6
0.05
60
64.6%
Sparse rewards improve +0.8pp then significantly declined after step 70
V5
Dense (read=0.3, state=0.1)
1.5e-6
0.04
116
57.3%
Dense rewards degrade ( − 6.5pp)
V6
IRC (read=0.0, state= − 0.1)
5e-7
0.2
180
59.1–66.7%
Correct rewards, LR too conservative
V7
IRC + higher LR
2e-6
0.05
60
62.0%
Higher LR helps but breaks some tasks
V8
IRC + deep_equal + prompt
2e-6
0.05
430
68.0%
Combined fixes
Table 6: Ablation of reward design variants on Qwen3.5-4B. IRC corrects the discriminative misalignment of V5, recovering and exceeding baseline performance. The gap between V5 (57.3%) and base (63.8%) demonstrates that naïve dense rewards actively harm performance.
Version
Steps
Tau2
Δ
Pass 4
Base (no RL)
0
58.0%
—
—
GRPO V2 (sparse)
480
54.0%
− 4.0
30.0%
MT-GRPO V3 (sparse)
60
68.0%
+10.0
44.0%
MT-GRPO V5 (dense)
187
54.0%
− 4.0
28.0%
V5.2 (GTPO hybrid)
251
56.5%
− 1.5
32.0%
+ IRC
—
69.5%
+11.5
53.0%
Table 7: Qwen3-30B-A3B MoE on Tau2-Bench. Δ is vs. base model (58.0%). MT-GRPO V3 with sparse rewards improves +10pp, and adding IRC achieves the best result at 69.5% (+11.5pp). Naïve GRPO and dense rewards both degrade performance.
Metric
Base
Trained
Conversation turns
56
28
Duration (seconds)
1,633
568
Tool calls
8+
4
Action accuracy
0/2
2/2
Database match
No
Yes
Reward
0.0
1.0
Table 8: Before vs. after training on Task 9 (flight cancellation with user manipulation). Training produces 50% fewer turns, 65% faster completion, and 100% action accuracy.
Domain
Pass Rate
Airline (trained)
69.3%
Retail (trained)
77.4%
Telecom (zero-shot)
32.0%
Table 9: Cross-domain evaluation. The airline-trained model achieves strong zero-shot retail performance (77.4%), while the harder telecom domain (32.0%) shows limited transfer.