When Credit Turns Against Tools: Transition-Aware Credit Assignment in Agentic Reinforcement Learning
Organizations: TMLR Group, Department of Computer Science, Hong Kong Baptist University · Peking University · Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
Abstract
Agentic reinforcement learning with verifiable rewards trains language models through multi-turn tool interactions, yet their reasoning performance and tool-calling ability can decline during training. We trace this degradation to three coupled mechanisms: (1) tool-token imbalance produces negative overall tool credit when incorrect trajectories contain more tool tokens per turn than correct ones; (2) aggregation amplification can further strengthen this negative signal as token-mean aggregation gives longer incorrect trajectories greater weight in policy updates; and (3) failure-driven retries prolong trajectories as models tend to call tools again after execution errors, extending potential negative credit to recovery attempts. To counter this suppression, we propose Transition-Aware Credit Assignment (TACA), which targets the transition from reasoning to tool use through supervision of both call initiation and generation. Specifically, TACA uses Advantage Projection to enforce a nonnegative mean advantage on tool-call tokens within each prompt group through a minimal uniform shift. Additionally, TACA guides tool-use decisions with Decision Contrast, comparing rewards for tool-assisted and direct answers to the same prompt. Through Supervised Replay, it reinforces tool-call generation using calls from previously sampled correct trajectories. Across models and tasks, TACA improves performance and sustains more reliable tool interaction during training, raising mean accuracy over MT-GRPO on six reasoning benchmarks by 4.61 percentage points for Qwen2.5-1.5B and 3.61 for Qwen2.5-3B. Our code is available at https://github.com/tmlr-group/TACA.
Figures & tables
| Methods | Benchmarks | Average | |||||
| AIME24 | AIME25 | MATH500 | AMC23 | OlympiadBench | GSM8K | ||
| Qwen2.5-1.5B-Instruct | |||||||
| Training-free | |||||||
| Base model | 1.09 | 0.35 | 38.83 | 15.56 | 10.52 | 59.42 | 20.96 |
| Agentic prompting | 0.20 | 0.0 | 10.46 | 3.44 | 2.69 | 16.29 | 5.51 |
| Non-agentic training | |||||||
| Method | TACO | WebShop | |||||
| HackerRank | HackerEarth | CodeForces | CodeChef | Avg. | SR (%) | Score | |
| MT-GRPO | 59.80 | 25.00 | 48.13 | 34.98 | 41.98 | 56.80 | 75.80 |
| GiGPO | – | – | – | – | – | 65.00 | 83.10 |
| TACA | 64.71 | 50.00 | 58.98 | 41.12 | 53.70 | 67.20 | 85.80 |
| Variant | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | GSM8K | Avg. | Avg. |
| TACA (full) | 59.14 | 8.43 | 8.12 | 27.42 | 22.20 | 74.15 | 33.24 | – |
| w/o Advantage Projection | 53.74 | 6.55 | 3.49 | 25.98 | 19.16 | 72.71 | 30.27 | |
| w/o Decision Contrast | 53.35 | 6.74 | 5.24 | 26.58 | 18.89 | 72.67 | 30.58 | |
| w/o Supervised Replay | 52.68 | 3.58 | 3.36 | 27.31 | 20.95 | 73.37 | 30.21 |
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | GSM8K | Avg. |
| Qwen2.5-7B-Instruct | |||||||
| MT-GRPO | 68.92 | 11.86 | 8.02 | 44.11 | 33.77 | 88.84 | 42.59 |
| TACA | 70.77 | 18.35 | 13.43 | 47.78 | 36.97 | 88.51 | 45.97 |
| Llama-3.2-1B-Instruct | |||||||
| MT-GRPO | 24.04 | 0.79 | 0.00 | 10.12 | 5.14 | 41.58 | 13.61 |
| TACA | 28.07 | 1.74 | 0.31 | 10.93 | 6.42 | 43.54 | 15.17 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Training dataset | Accuracy (%) | Tool-token gap |
| MATH | 66.41 | |
| DAPO-Math | 30.59 | |
| LIMR | 78.13 | |
| DeepScaleR | 39.45 |
| Retry rate (%) | Mean turns | |||
| Setting | Incorrect | Correct | Incorrect | Correct |
| Mathematical reasoning with Python | ||||
| Qwen2.5-1.5B-Instruct | 31.38 | 12.07 | 2.20 | 1.91 |
| Qwen2.5-3B-Instruct | 60.55 | 35.96 | 3.15 | 2.38 |
| Qwen2.5-7B-Instruct | 36.82 | 21.48 | 1.76 | 1.59 |
| Llama-3.2-1B-Instruct | 88.65 | 50.88 | 4.05 | 3.37 |
| Parameter | Setting |
| Rollout configuration | |
| Prompts per batch / trajectories per prompt | 32 / 8 |
| Training temperature / top-p | 1.0 / 1.0 |
| Maximum input-context length | 7,168 tokens |
| Maximum generated turns | 6 |
| Response tokens per turn | 1,024 (Qwen2.5-7B); 2,048 (other models) |
| Method | MATH500 | AIME24 | AIME25 | AMC23 | Olympiad Bench | GSM8K | Avg. |
| MT-GRPO | 50.70 | 3.26 | 0.97 | 27.50 | 17.88 | 71.49 | 28.63 |
| Dr. GRPO | 51.76 | 4.03 | 0.89 | 27.40 | 17.57 | 72.56 | 29.04 |
| TACA | 59.14 | 8.43 | 8.12 | 27.42 | 22.20 | 74.15 | 33.24 |
| Metric | ReTool | TACA |
| Training accuracy (%) | 49.17 | 48.09 |
| Tool-use rate (%) | 1.49 | 75.76 |
| No-tool failure rate (%) | 49.75 | 18.15 |
| Tool-using correct trajectories (%) | 0.41 | 42.00 |
| Tool use among correct trajectories (%) | 0.84 | 87.34 |
| Steps containing calls | 27 | 100 |
Agentic RL
Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.
See all 245 papers