cs.AISep 15, 2026

Turn-level Multiscale Density Ratio Estimation for LLM Agents

Authors: Zishuo Zhao, Kai Chen, Ao Li, Yuan Liu

Organizations: Alibaba Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

With the rapid development of Large language model (LLM), agent systems enhanced by LLMs show huge potential in being able to deal with complex tasks, especially involving multi-step thinking or interaction with tools. For applying LLM techniques with a well-designed agent paradigm, post-training of LLM in multiple agent scenarios is necessary to achieve better performance. Among the variable post-training techniques, alignment methods such as PPO, DPO, DIL, and GRPO become popular because many papers show a significant positive impact on the model's performance by punishing negative samples while keeping acceptable training complexity. However, most alignment methods address simple single-turn tasks, and there remains room for improvement for complex multi-turn tasks. We propose Turn-level Multiscale Density Ratio Estimation (tlm-DRE), which assigns different weights on corresponding turns and proposes asymmetric token-level training based on the positive-negative space gaps across multiple turns of tasks. The results of the experiment on a wide range of agent benchmarks show that the proposed method performs competitively compared to traditional alignment methods. The proposed training method enables LLMs to perform robustly in multi-turn reasoning tasks with both in-domain and out-of-domain conditions.

Figures & tables

Figure 1: The overall architecture of tlm-DRE in a single iteration. First, the agent initialized with the SFT policy samples trajectory paths and collects failure trajectories. Then, within these failure trajectories, we identify low-confidence turns—i.e., turns where the model exhibits low self-confidence. Finally, we train the agent using Multiscale Density Ratio Estimation, which explicitly upweights these identified failure turns during optimization.
ParadigmModelsALFWorldScienceWorld
SeenUnseenSeenUnseen
Prompt-basedGPT-3.5 Ouyang et al. (2022)7.910.516.513.0
GPT-4 Achiam et al. (2023)42.938.164.864.4
Qwen2.5-7B Team et al. (2024)25.128.426.825.2
Llama-2-7B-Chat
SFT & RLSFT Chen et al. (2023)60.067.256.856.0
Table 1: Performance of different methods on ALFWorld and ScienceWorld, reported as average reward. “Seen” denotes the held-out test set containing task types observed during training, while “Unseen” refers to test tasks with critical unseen variations (e.g., novel objects or goals). “tlm-DRE w/o DRE” denotes linearly turn-weighted DPO training. For a fair comparison, all methods use the same base models: Llama-2-7B and Qwen2.5-7B. All experimental results are averaged over five independent runs with different random seeds.
 HotpotQA
MethodEMF1
SFT Chen et al. (2023)27.8036.45
CoH Liu et al. (2023)28.6039.53
PPO Trung et al. (2024)28.2036.47
DPO Song et al. (2024)26.4034.83
NAT Wang et al. (2025)29.6042.50
Table 2: Overall results on open-domain question answering tasks. We measure the performance using Exact Match and F1 score. For fair comparison, all methods use the same base models: LLaMA2-7B
Figure 2: Illustration of confidence scores for different turns computed using the SFT policy, shown for both the chosen and rejected trajectories.
Figure 3: Illustration of the reward margin between the chosen and rejected trajectories during training for tlm-DRE and DPO
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Illustration of DRE with Bregman divergence.
 ALFWorld
Method(all)
RLOO Ahmadian et al. (2024)75.5
GRPO Shao et al. (2024)77.6
GiGPO w/ std Feng et al. (2025)90.8
GiGPO w/o std Feng et al. (2025)90.2
tlm-DRE (ours)92.4
Appendix
Table 3: For fair comparison, results are evaluated in the GiGPO Feng et al. (2025) test environment (with longer exploration steps and different test categories). All methods use the same base models: Qwen2.5-7B-Instruct
Figure 5: The statistical information of the adequacy/confidence score defined in Equation 9 for each trajectory. The orange dashed vertical line indicates the selected pivot value ppivotp_{pivot} .
Figure 6: The searching result of the hyperparameters ωL\omega_{L} in the main agentic tasks. The dashed vertical line shows the optimal value of ωL\omega_{L}
Figure 7: Case analysis across different turns.
Figure 8: Trajectory sample before tlm-DRE .
Figure 9: Trajectory sample after tlm-DRE .

Explore similar work

CardsList

  1. TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

    May 1, 2026Yi Xie, Siao Liu, Falong Fan +3Multi-Agent System OptimizationMulti-Agent Coordination

    Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1

  2. RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

    Aug 19, 2026Yugu Li, Zehong Cao, Jianglin Qiao +1Agentic RLReinforcement Learning

    School of CSIT · ACFR · School of EECMS

  3. SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training

    Mar 18, 2026Prince Zizhuang Wang, Shuli JiangRL for LLM AgentsAgentic RL

    Carnegie Mellon University