With the rapid development of Large language model (LLM), agent systems enhanced by LLMs show huge potential in being able to deal with complex tasks, especially involving multi-step thinking or interaction with tools. For applying LLM techniques with a well-designed agent paradigm, post-training of LLM in multiple agent scenarios is necessary to achieve better performance. Among the variable post-training techniques, alignment methods such as PPO, DPO, DIL, and GRPO become popular because many papers show a significant positive impact on the model's performance by punishing negative samples while keeping acceptable training complexity. However, most alignment methods address simple single-turn tasks, and there remains room for improvement for complex multi-turn tasks. We propose Turn-level Multiscale Density Ratio Estimation (tlm-DRE), which assigns different weights on corresponding turns and proposes asymmetric token-level training based on the positive-negative space gaps across multiple turns of tasks. The results of the experiment on a wide range of agent benchmarks show that the proposed method performs competitively compared to traditional alignment methods. The proposed training method enables LLMs to perform robustly in multi-turn reasoning tasks with both in-domain and out-of-domain conditions.
Figures & tables
Figure 1: The overall architecture of tlm-DRE in a single iteration. First, the agent initialized with the SFT policy samples trajectory paths and collects failure trajectories. Then, within these failure trajectories, we identify low-confidence turns—i.e., turns where the model exhibits low self-confidence. Finally, we train the agent using Multiscale Density Ratio Estimation, which explicitly upweights these identified failure turns during optimization.
Paradigm
Models
ALFWorld
ScienceWorld
Seen
Unseen
Seen
Unseen
Prompt-based
GPT-3.5 Ouyang et al. (2022)
7.9
10.5
16.5
13.0
GPT-4 Achiam et al. (2023)
42.9
38.1
64.8
64.4
Qwen2.5-7B Team et al. (2024)
25.1
28.4
26.8
25.2
Llama-2-7B-Chat
SFT & RL
SFT Chen et al. (2023)
60.0
67.2
56.8
56.0
Table 1: Performance of different methods on ALFWorld and ScienceWorld, reported as average reward. “Seen” denotes the held-out test set containing task types observed during training, while “Unseen” refers to test tasks with critical unseen variations (e.g., novel objects or goals). “tlm-DRE w/o DRE” denotes linearly turn-weighted DPO training. For a fair comparison, all methods use the same base models: Llama-2-7B and Qwen2.5-7B. All experimental results are averaged over five independent runs with different random seeds.
HotpotQA
Method
EM
F1
SFT Chen et al. (2023)
27.80
36.45
CoH Liu et al. (2023)
28.60
39.53
PPO Trung et al. (2024)
28.20
36.47
DPO Song et al. (2024)
26.40
34.83
NAT Wang et al. (2025)
29.60
42.50
Table 2: Overall results on open-domain question answering tasks. We measure the performance using Exact Match and F1 score. For fair comparison, all methods use the same base models: LLaMA2-7B
Figure 2: Illustration of confidence scores for different turns computed using the SFT policy, shown for both the chosen and rejected trajectories.
Figure 3: Illustration of the reward margin between the chosen and rejected trajectories during training for tlm-DRE and DPO
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Illustration of DRE with Bregman divergence.
ALFWorld
Method
(all)
RLOO Ahmadian et al. (2024)
75.5
GRPO Shao et al. (2024)
77.6
GiGPO w/ std Feng et al. (2025)
90.8
GiGPO w/o std Feng et al. (2025)
90.2
tlm-DRE (ours)
92.4
Appendix
Table 3: For fair comparison, results are evaluated in the GiGPO Feng et al. (2025) test environment (with longer exploration steps and different test categories). All methods use the same base models: Qwen2.5-7B-Instruct
Figure 5: The statistical information of the adequacy/confidence score defined in Equation 9 for each trajectory. The orange dashed vertical line indicates the selected pivot value ppivot .
Figure 6: The searching result of the hyperparameters ωL in the main agentic tasks. The dashed vertical line shows the optimal value of ωL
Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds. Experiments show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement. Code is available at https://github.com/Yydc/TeamTR.
Yi Xie, Siao Liu, Falong Fan +3
Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1
Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.
Large Language Model (LLM) agents have shown strong results on multi-turn tool-use tasks, yet they operate in isolation during training, failing to leverage experiences accumulated across episodes. Existing experience-augmented methods address this by organizing trajectories into retrievable libraries, but they retrieve experiences only once based on the initial task description and hold them constant throughout the episode. In multi-turn settings where observations change at every step, this static retrieval becomes increasingly mismatched as episodes progress. We propose SLEA-RL (Step-Level Experience-Augmented Reinforcement Learning), a framework that retrieves relevant experiences at each decision step conditioned on the current observation. SLEA-RL operates through three components: (i) step-level observation clustering that groups structurally equivalent environmental states for efficient cluster-indexed retrieval; (ii) a self-evolving experience library that distills successful strategies and failure patterns through score-based admission and rate-limited extraction; and (iii) policy optimization with step-level credit assignment for fine-grained advantage estimation across multi-turn episodes. The experience library evolves alongside the policy through semantic analysis rather than gradient updates. Experiments on long-horizon multi-turn agent benchmarks demonstrate that SLEA-RL achieves superior performance compared to various reinforcement learning baselines.