What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Authors: Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao
Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tsinghua University
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/
Figures & tables
Figure 1: Overview of our findings. Left: Multi-turn agentic RL does not uniformly enhance per-step action quality; rather, it mainly enhances ability of error recovery and long-horizon resilience. Right: The trajectory-level gains of agentic RL depend on a sufficient interaction budget ( T ).
Figure 2: Pass@K across sampling breadth. Sweeping the number of sampled trajectories K .
Figure 3: Pass@T at fixed k=8 . Sweeping the maximum number of interaction turns T .
Figure 4: Step-level action quality. Each point is one matched state; its coordinates are action quality of the base and RL model. Teal points favor the base model, red points favor the RL model.
Figure 5: Step-level action quality on error states. Each point is one error state.
Figure 7: Likelihood analysis. Boxplots show action NLL under base policy (left axis). Curves show cumulative LLR (right axis). Bands indicate 95% confidence.
Figure 8: Training dynamics of the trajectory-level reward Rmulti and step reward Rsingle on DeepCoder-MT.
Figure 9: Effect of process supervision signals on Pass@T .
FrozenLake ( Tmax=70 )
DeepCoder-MT ( Tmax=10 )
BFCL-v3 ( Tmax=24 )
InterCode-SQL ( Tmax=10 )
Method
25%
50%
75%
100%
25%
50%
75%
100%
25%
50%
75%
100%
25%
50%
75%
100%
Avg.
Qwen3-4B
Base
30.3
49.1
58.7
61.6
34.0
43.5
47.3
48.3
3.0
15.6
21.2
26.4
23.1
28.8
31.8
33.5
34.8
GRPO Full
28.7
55.1
68.1
72.4
29.2
48.0
56.0
59.0
2.5
17.6
27.0
30.2
23.1
32.2
37.4
46.0
39.5
GRPO BudgetMix
37.6
61.1
72.0
73.5
38.1
54.3
60.5
61.2
8.5
23.0
30.5
30.8
28.9
37.7
42.9
47.5
44.3
GiGPO Full
38.4
58.4
70.4
76.5
28.8
48.3
55.6
58.8
3.4
18.4
28.8
31.3
23.4
31.5
37.5
45.0
40.9
Table 1: Pass@T evaluation results ( Pass@8 , %). We evaluate models at 25%, 50%, 75%, and 100% of the maximum interaction budget. Bold and underlined values denote the best and second-best results, respectively, within each model size and budget. Avg. gain reports the average improvement of BudgetMix over full-budget RL across GRPO, GiGPO, and PPO, in percentage points.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
SYSTEM PROMPT
You are an agent interacting with the [ENVIRONMENT] task under a finite interaction budget. One interaction turn consists of one action from you followed by one observation from the environment. Maximum number of interaction turns: [MAX_TURNS] . Number of turns already completed: [CURRENT_TURN] . Number of turns remaining: [REMAINING_TURNS] . Follow the task-specific instruction below. At each turn, output exactly one action in the required format. Do not output multiple actions in one turn, and do not assume that additional turns will be available after the budget is exhausted.
TASK-SPECIFIC INSTRUCTION
[TASK_SPECIFIC_PROMPT]
ACTION FORMAT
Output exactly one action in the following format: [TASK_SPECIFIC_ACTION_FORMAT]
Appendix
Table 2: Budget-aware prompt template used during training and evaluation. The turn counters are instantiated before each model action.
Figure 10: Model performance and structural performance upper bounds across interaction budgets. Gray lines indicate the structural upper bounds.
SYSTEM PROMPT
You are evaluating one action taken by an agent in an interactive task. Given the task, interaction history, current observation, and remaining interaction budget, rate the proposed action with an integer from 0 to 10. Evaluate its correctness, relevance, and expected contribution to completing the task efficiently. Necessary intermediate steps can receive full credit. Do not reward verbosity or redundant actions.
INPUT
Task: [TASK] Interaction history: [HISTORY] Current observation: [OBSERVATION] Remaining turns, including this action: [REMAINING_TURNS] Action to evaluate: [ACTION]
OUTPUT FORMAT
Output only one integer from 0 to 10, with no explanation.
Appendix
Table 3: LLM judge prompt for step-level process supervision. The judge scores the current action before execution. Scores are mapped to [0,1] by dividing by 10.
Figure 11: Additional Pass@K results with GiGPO.
Figure 12: Additional Pass@K results with PPO.
Figure 13: Pass@K results for InternVL3.5-8B on FrozenLake and DeepSeek-R1-Distill-Llama-8B on DeepCoder-MT with GRPO.
Figure 14: Additional Pass@K results on Terminal-Bench.
Figure 15: Pass@T results with GiGPO on FrozenLake, DeepCoder-MT, BFCL and InterCode-SQL.
Figure 16: Pass@T results with PPO on FrozenLake, DeepCoder-MT, BFCL and InterCode-SQL.
Figure 17: Pass@T results for InternVL3.5-8B on FrozenLake and DeepSeek-R1-Distill-Llama-8B on DeepCoder-MT.
Figure 18: Pass@T results for the Qwen3 models on Terminal-Bench.
Figure 19: Step-level action quality. Each point is one matched state; its coordinates are action quality of the base and RL model. Teal points favor the base model, red points favor the RL model.
Figure 20: Step-level action quality on error states. Each point is one error state.
Figure 21: Step-level action quality of BudgetMix RL .
Figure 22: Step-level action quality on error states of BudgetMix RL .
Figure 23: More log-likehood analysis of base and RL models. (a) Additional log-likelihood analysis on BFCL. (b) Cumulative LLR after length matching. Error bars indicate 95% CI.
Figure 24: Training dynamics of the trajectory-level reward Rmulti and step reward Rsingle .
Figure 25: A BFCL example in which the RL model corrects an erroneous tool call.
Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation. In long-horizon agentic tasks, such a uniform rollout strategy can waste budget on uninformative dead-end attempts, while promising intermediate states do not receive sufficient exploration. The multi-turn structure of agentic trajectories, with interleaved actions and observations, naturally supports organizing a trajectory group as a tree, where each turn serves as a decision point for exploration. This perspective reframes effective exploration as the problem of deciding where to branch. We propose Process-Scorer Guided Adaptive Tree Rollout (PATR), a quality-aware rollout framework for multi-turn agent RL. PATR uses task-appropriate process feedback to score partial trajectories, selectively branches from promising states, reuses shared prefixes, and conservatively stops degenerate paths to reduce wasted sampling. The resulting rollout groups remain compatible with standard policy optimization while providing more efficient exploration under the same training budget. We evaluate PATR on FrozenLake and the challenging SWE-Bench, which is largely unexplored by prior tree-rollout agent RL methods. Experiments show that PATR improves performance by up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, highlighting process-guided tree rollouts as an effective strategy for scalable multi-turn RL.
Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.
Large Language Model (LLM) agents have shown strong results on multi-turn tool-use tasks, yet they operate in isolation during training, failing to leverage experiences accumulated across episodes. Existing experience-augmented methods address this by organizing trajectories into retrievable libraries, but they retrieve experiences only once based on the initial task description and hold them constant throughout the episode. In multi-turn settings where observations change at every step, this static retrieval becomes increasingly mismatched as episodes progress. We propose SLEA-RL (Step-Level Experience-Augmented Reinforcement Learning), a framework that retrieves relevant experiences at each decision step conditioned on the current observation. SLEA-RL operates through three components: (i) step-level observation clustering that groups structurally equivalent environmental states for efficient cluster-indexed retrieval; (ii) a self-evolving experience library that distills successful strategies and failure patterns through score-based admission and rate-limited extraction; and (iii) policy optimization with step-level credit assignment for fine-grained advantage estimation across multi-turn episodes. The experience library evolves alongside the policy through semantic analysis rather than gradient updates. Experiments on long-horizon multi-turn agent benchmarks demonstrate that SLEA-RL achieves superior performance compared to various reinforcement learning baselines.