What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

Kejian Zhu; Zhuoran Jin; Shangqing Tu; Tianyi Men; Yupu Hao; Jiachun Li; Kang Liu; Jun Zhao

cs.LGNov 4, 2026

What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

Authors: Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao

Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tsinghua University

Abstract

While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth $K$ and interaction depth $T$ via $Pass@(K,T)$ , we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/

Figures & tables

Figure 1: Overview of our findings. Left: Multi-turn agentic RL does not uniformly enhance per-step action quality; rather, it mainly enhances ability of error recovery and long-horizon resilience. Right: The trajectory-level gains of agentic RL depend on a sufficient interaction budget ( $T$ ).

Figure 2:

\mathrm{Pass}@K

across sampling breadth. Sweeping the number of sampled trajectories

K

Figure 3:

\mathrm{Pass}@T

at fixed

k{=}8

. Sweeping the maximum number of interaction turns

T

Figure 4: Step-level action quality. Each point is one matched state; its coordinates are action quality of the base and RL model. Teal points favor the base model, red points favor the RL model.

Figure 5: Step-level action quality on error states. Each point is one error state.

Figure 6: Action quality versus normalized trajectory depth

d_{t}=t/T_{\max}

. Shading denotes 95% CI.

Figure 7: Likelihood analysis. Boxplots show action NLL under base policy (left axis). Curves show cumulative LLR (right axis). Bands indicate 95% confidence.

Figure 8: Training dynamics of the trajectory-level reward

R_{\mathrm{multi}}

and step reward

R_{\mathrm{single}}

on DeepCoder-MT.

Figure 9: Effect of process supervision signals on

\mathrm{Pass}@T

Qwen3-4B
	FrozenLake ( $T_{\max}=70$ )				DeepCoder-MT ( $T_{\max}=10$ )				BFCL-v3 ( $T_{\max}=24$ )				InterCode-SQL ( $T_{\max}=10$ )
Method	25%	50%	75%	100%	25%	50%	75%	100%	25%	50%	75%	100%	25%	50%	75%	100%	Avg.
Base	30.3	49.1	58.7	61.6	34.0	43.5	47.3	48.3	3.0	15.6	21.2	26.4	23.1	28.8	31.8	33.5	34.8
GRPO Full	28.7	55.1	68.1	72.4	29.2	48.0	56.0	59.0	2.5	17.6	27.0	30.2	23.1	32.2	37.4	46.0	39.5
GRPO BudgetMix	37.6	61.1	72.0	73.5	38.1	54.3	60.5	61.2	8.5	23.0	30.5	30.8	28.9	37.7	42.9	47.5	44.3
GiGPO Full	38.4	58.4	70.4	76.5	28.8	48.3	55.6	58.8	3.4	18.4	28.8	31.3	23.4	31.5	37.5	45.0	40.9

Table 1:

\mathrm{Pass}@T

evaluation results (

\mathrm{Pass}@8

, %). We evaluate models at 25%, 50%, 75%, and 100% of the maximum interaction budget. Bold and underlined values denote the best and second-best results, respectively, within each model size and budget. Avg. gain reports the average improvement of BudgetMix over full-budget RL across GRPO, GiGPO, and PPO, in percentage points.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

SYSTEM PROMPT

You are an agent interacting with the [ENVIRONMENT] task under a finite interaction budget. One interaction turn consists of one action from you followed by one observation from the environment. Maximum number of interaction turns: [MAX_TURNS] . Number of turns already completed: [CURRENT_TURN] . Number of turns remaining: [REMAINING_TURNS] . Follow the task-specific instruction below. At each turn, output exactly one action in the required format. Do not output multiple actions in one turn, and do not assume that additional turns will be available after the budget is exhausted.

TASK-SPECIFIC INSTRUCTION

[TASK_SPECIFIC_PROMPT]

ACTION FORMAT

Output exactly one action in the following format: [TASK_SPECIFIC_ACTION_FORMAT]

Appendix

Table 2: Budget-aware prompt template used during training and evaluation. The turn counters are instantiated before each model action.

Figure 10: Model performance and structural performance upper bounds across interaction budgets. Gray lines indicate the structural upper bounds.

SYSTEM PROMPT

You are evaluating one action taken by an agent in an interactive task. Given the task, interaction history, current observation, and remaining interaction budget, rate the proposed action with an integer from 0 to 10. Evaluate its correctness, relevance, and expected contribution to completing the task efficiently. Necessary intermediate steps can receive full credit. Do not reward verbosity or redundant actions.

INPUT

Task: [TASK] Interaction history: [HISTORY] Current observation: [OBSERVATION] Remaining turns, including this action: [REMAINING_TURNS] Action to evaluate: [ACTION]

OUTPUT FORMAT

Output only one integer from 0 to 10, with no explanation.

Appendix

Table 3: LLM judge prompt for step-level process supervision. The judge scores the current action before execution. Scores are mapped to

[0,1]

by dividing by 10.

Figure 11: Additional

\mathrm{Pass}@K

results with GiGPO.

Figure 12: Additional

\mathrm{Pass}@K

results with PPO.

Figure 13:

\mathrm{Pass}@K

results for InternVL3.5-8B on FrozenLake and DeepSeek-R1-Distill-Llama-8B on DeepCoder-MT with GRPO.

Figure 14: Additional

\mathrm{Pass}@K

results on Terminal-Bench.

Figure 15:

\mathrm{Pass}@T

results with GiGPO on FrozenLake, DeepCoder-MT, BFCL and InterCode-SQL.

Figure 16:

\mathrm{Pass}@T

results with PPO on FrozenLake, DeepCoder-MT, BFCL and InterCode-SQL.

Figure 17:

\mathrm{Pass}@T

results for InternVL3.5-8B on FrozenLake and DeepSeek-R1-Distill-Llama-8B on DeepCoder-MT.

Figure 18:

\mathrm{Pass}@T

results for the Qwen3 models on Terminal-Bench.

Figure 19: Step-level action quality. Each point is one matched state; its coordinates are action quality of the base and RL model. Teal points favor the base model, red points favor the RL model.

Figure 20: Step-level action quality on error states. Each point is one error state.

Figure 21: Step-level action quality of BudgetMix RL .

Figure 22: Step-level action quality on error states of BudgetMix RL .

Figure 23: More log-likehood analysis of base and RL models. (a) Additional log-likelihood analysis on BFCL. (b) Cumulative LLR after length matching. Error bars indicate 95% CI.

Figure 24: Training dynamics of the trajectory-level reward

R_{\mathrm{multi}}

and step reward

R_{\mathrm{single}}

Figure 25: A BFCL example in which the RL model corrects an erroneous tool call.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 637 papers

Process Reward Informed Tree Rollout for Effective Multi-Turn RL
Jul 17, 2026Xintong Li, Sha Li, Yuwei Zhang +8Reinforcement Learning RL Exploration
1UC San Diego · 2Amazon · 3MIT Alumni
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Aug 19, 2026Yugu Li, Zehong Cao, Jianglin Qiao +1Agentic RL Reinforcement Learning
School of CSIT · ACFR · School of EECMS
SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training
Mar 18, 2026Prince Zizhuang Wang, Shuli JiangRL for LLM Agents Agentic RL
Carnegie Mellon University

Jul 17, 2026J/K move · Enter open · S save

Process Reward Informed Tree Rollout for Effective Multi-Turn RL

Xintong Li, Sha Li, Yuwei Zhang, Changlong Yu+7

1UC San Diego · 2Amazon · 3MIT Alumni

Reinforcement Learning RL Exploration RL for LLM Agents

Open paper AskPDF arXiv

Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation. In long-horizon agentic tasks, such a uniform rollout strategy can waste budget on uninformative dead-end attempts, while promising intermediate states do not receive sufficient exploration. The multi-turn structure of agentic trajectories, with interleaved actions and observations, naturally supports organizing a trajectory group as a tree, where each turn serves as a decision point for exploration. This perspective reframes effective exploration as the problem of deciding where to branch. We propose Process-Scorer Guided Adaptive Tree Rollout (PATR), a quality-aware rollout framework for multi-turn agent RL. PATR uses task-appropriate process feedback to score partial trajectories, selectively branches from promising states, reuses shared prefixes, and conservatively stops degenerate paths to reduce wasted sampling. The resulting rollout groups remain compatible with standard policy optimization while providing more efficient exploration under the same training budget. We evaluate PATR on FrozenLake and the challenging SWE-Bench, which is largely unexplored by prior tree-rollout agent RL methods. Experiments show that PATR improves performance by up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, highlighting process-guided tree rollouts as an effective strategy for scalable multi-turn RL.

Abstract

RL for LLM Agents

Explore similar work

Process Reward Informed Tree Rollout for Effective Multi-Turn RL

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training