Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents

Haojin Yang; Ai Jian; Xinyue Huang; Yiwei Wang; Weipeng Zhang; Ke Zeng; Xunliang Cai; Jingqing Ruan

cs.AIMar 2, 2026

Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents

Authors: Haojin Yang, Ai Jian, Xinyue Huang, Yiwei Wang, Weipeng Zhang, Ke Zeng, Xunliang Cai, Jingqing Ruan

Organizations: Peking University · Beijing University of Posts and Telecommunications · Sun Yat-sen University · University of California at Merced ·

Abstract

Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.

Figures & tables

Figure 1: Comparison of dialogue strategies between SFT, Standard RL, and our proposed DuCA method.

Figure 2: Overview of the DuCA framework. The system generates interaction trajectories via a user simulator conditioned on personas and history. It independently calculates and normalizes: (1) Turn-level advantages from dense heuristic constraints, and (2) Session-level advantages from sparse business outcomes. These decoupled signals are fused to provide a balanced final advantage for robust policy updates.

Method	CVR( $\uparrow$ )	Compliance( $\uparrow$ )	Avg. Turn	Intra-R( $\downarrow$ )	Inter-R( $\downarrow$ )	IDR( $\downarrow$ )	Prefix-R( $\downarrow$ )	Filler( $\downarrow$ )	PATR( $\uparrow$ )
Foundation Models
DeepSeek-R1	22.85%	65.13	9.85	35.19%	1.01%	2.50%	7.69%	59.82%	42.74%
Longcat-Flash-Chat	19.99%	70.17	9.27	3.02%	23.13%	5.92%	39.10%	33.35%	43.65%
Training Methods
SFT (Base)	22.23%	59.11	11.41	9.37%	35.87%	10.59%	52.30%	51.70%	43.59%
REINFORCE++	21.51%	61.32	11.18	8.34%	30.17%	9.99%	50.34%	54.69%	43.62%

Table 1: Main Results on Sales Dialogue Evaluation. CVR and Compliance represent primary business outcomes, while the remaining metrics evaluate fine-grained interaction quality. Best results across all models are bolded .

Method	CVR (%)	Compliance( $\uparrow$ )	Avg. Turn
DuCA (Full)	24.44	66.72	9.91
w/o HIAN	24.13	64.09	10.31
w/o Multi-turn	21.64	60.09	12.02

Table 2: Ablation Study Results.

Panel of Figure 3 — Figure 3: Training dynamics of DuCA compared with baselines over 70 steps.

Metric	Simulator	Human Level
Receiver Identity	4.906	4.948
Decision Subject	4.750	4.824
Merchant Profile	4.899	4.830
Busy State	4.957	4.980
Acceptance Level	4.888	4.830
Basic Judgment	4.910	4.843

Table 3: Consistency Evaluation: Simulator vs. Human Level. Scores (1-5) are generated via GPT-4.1 based on persona and context adherence. Best results are bolded .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Percentage (%)
Correctly identified as Human	36.0%
Indistinguishable / Hard to distinguish	12.0%
Mistakenly identified as Simulator	52.0%

Appendix

Table 4: Human-vs-Simulator Turing Test Results. Results indicate the classification of actual human trajectories by experts.

Variant	Intra-R↓	Inter-R↓	IDR↓	Prefix-R↓	Filler↓	PATR↑
DRF (Full)	4.20%	2.71%	6.11%	34.44%	49.21%	44.61%
w/o Decoupled Norm	1.87%	1.68%	7.35%	36.64%	52.49%	43.83%
w/o Multi-turn	8.94%	41.75%	10.34%	57.41%	45.71%	43.84%

Appendix

Table 5: Detailed Ablation Study on Conversational Quality. "w/o Decoupled Norm" refers to the variant using standard reward summation (rpp_baseline). "w/o Multi-turn" denotes training on single-turn scenarios. Intra-R and Inter-R: Intra-turn and Inter-turn Repetition; IDR: Identity Detection Rate; PATR: Positive Attitude Transfer Rate. Best results are bolded .

Panel of Figure 4 — Figure 4: Training dynamics of DRF compared with ablation variants. (a) Conversion Rate (CVR) evolution. (b) Compliance Score evolution.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 608 papers

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2
HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning
Mar 19, 2026Zhicong Lu, Zichuan Lin, Wei Jia +7RL for LLM Agents Agentic RL
Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent Hunyuan +1
RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
Jul 6, 2026Qiang Liu, Taian Guo, Ruizhi Qiao +1Reinforcement Learning Sparse-Reward RL
Tencent YouTu Lab

Feb 18, 2026J/K move · Enter open · S save

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming+3

University of Minnesota · Northwestern University · Amazon AGI+2

RL for LLM Agents Hierarchical RL Step-Level Credit Assignment in RL Sparse-Reward RL

Open paper AskPDF arXiv

Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4% success on ALFWorld and 83.3% on WebShop with Qwen2.5-7B-Instruct (+6.6% and +8.3% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.

Abstract

RL for LLM Agents

Explore similar work

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning

RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents