MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue

Naifan Zhang; Ruihan Sun; Jinwei Su; Hengjie Yang; Zhengyuan Pan; Zhaohan Chen; Xiaofan Zhang

cs.CLMar 6, 2026

MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue

Authors: Naifan Zhang, Ruihan Sun, Jinwei Su, Hengjie Yang, Zhengyuan Pan, Zhaohan Chen, Xiaofan Zhang

Organizations: NatureSelect · Tsinghua University · South China Normal University · Xiamen University

Abstract

Reinforcement learning (RL) for large language models (LLMs) has shown strong performance in single-turn tasks, but extending it to multi-turn interaction remains challenging due to sparse rewards and poor per-turn credit assignment. In emotional support dialogues, responses shape future user states, so matched-state step-wise comparison is unavailable, while trajectory-level supervision is insufficient. We propose MICA (Multi-granularity Intertemporal Credit Assignment), a critic-free RL framework for multi-turn emotional support tasks. MICA derives both immediate and delayed credit from a shared potential function over the user's structured support state. Incremental Distance Reward measures the per-turn decrease in residual distance to the target state, while its Monte Carlo return captures delayed effects. After scope-specific normalization, the two signals form a mixed advantage for stable per-turn optimization without matched-state comparisons, rollout trees, or a learned critic. On EMPA, EQ-Bench, and EmoBench with Qwen2.5-7B-Instruct and Qwen3-8B/14B/32B, MICA consistently outperforms GRPO and REINFORCE++, achieving up to +42.5 on EMPA, while adding no rollout cost and remaining robust to reward judges. These results show that turn-aware credit assignment enables effective and practical multi-turn RL for interactive LLMs.

Figures & tables

Figure 1: Framework of MICA. The policy model interacts with Actor to collect multi-turn trajectories, and then is optimized via Mixed-Advantage .

Figure 2: MICA Overall Framework. Given an initial prompt, we sample

K

trajectories from the current policy, each consisting of

T

turns. The Turn-level advantage is computed by normalizing returns across samples at the same turn. The Group-level advantage is computed by normalizing rewards over all

K\times T

samples in the group. The final advantage is a convex combination of these two terms, balancing fine-grained credit assignment with global group-level optimization.

Figure 3: Distribution of Monte Carlo returns and immediate rewards across dialogue turns at specific training step. (a) Monte Carlo returns exhibit a clear positive correlation with the turn index; (b) In contrast, immediate rewards show no discernible trend across turns.

Model	Method	EMPA		EmoBench (Acc. %)			EQ-Bench
Model	Method	Pass	Score	EA	EU	Overall	Score
Gemini-2.5-pro		27.0	90.7	74.0	62.0	68.0	86.4
Claude-3.5-sonnet		25.0	85.1	73.0	54.0	63.5	77.0
DeepSeek-V3.2		25.0	78.4	73.0	55.0	64.0	84.9
Qwen3-32B	Base	19.0	68.9	70.0	43.0	56.5	74.0
	GRPO	21.0 ±0.8	73.6 ±0.5	69.7 ±0.5	44.3 ±0.5	57.0 ±0.4	74.6 ±0.2

Table 1: Quantitative results on empathy benchmarks. We compare different base models across five settings: Base (original), GRPO, REINFORCE++ variants (with Traj and IDR-Batch), and MICA. Results are averaged over 3 random seeds. For non-Base models, each result is reported with its standard deviation as a subscript.

Figure 4: Empathy alignment scores across various dimensions. MICA consistently outperforms GRPO and REINFORCE++ variants across all dimensions and model scales, showing greater alignment gains over Base Model.

Trained Model	Judger	EMPA		EmoBench (Acc. %)			EQ-Bench
Trained Model	Judger	Pass	Score	EA	EU	Overall	Score
Qwen2.5-7B-Instruct	Qwen3-235B	8.7 ±0.5	58.2 ±0.7	69.7 ±0.4	38.7 ±0.4	54.2 ±0.4	55.8 ±0.8
	MiniMax-M2.5	8.3 ±0.5	56.8 ±0.8	68.7 ±0.5	39.0 ±0.0	53.8 ±0.2	55.7 ±0.3
	GLM-4.7	9.3 ±0.5	59.5 ±0.6	69.3 ±0.5	38.7 ±0.5	54.0 ±0.0	56.8 ±0.4
Qwen3-8B	Qwen3-235B	8.3 ±0.5	41.5 ±1.8	68.3 ±0.5	37.7 ±0.5	53.0 ±0.4	74.5 ±0.4
	MiniMax-M2.5	8.7 ±0.5	43.5 ±0.2	69.0 ±0.0	38.7 ±0.5	53.8 ±0.2	74.6 ±0.3

Table 2: Quantitative results under different Judgers. Results are averaged over 3 random seeds. MICA yields similar downstream performance when trained with Qwen3-235B, MiniMax-M2.5, or GLM-4.7 as the Judger, suggesting limited sensitivity to the specific Judger choice in this setting.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Reward and gradient norm curves of Qwen3-8B and Qwen2.5-7B-Instruct under various advantages. Mixed Advantage achieves the highest converged reward while maintaining stable gradient norms, demonstrating simultaneous improvements in both reward performance and training stability.

Trained Model	Advantage Level	EMPA		EmoBench (Acc. %)			EQ-Bench
Trained Model	Advantage Level	Pass	Score	EA	EU	Overall	Score
Qwen3-8B	Group-Level	5.7 ±0.5	38.5 ±0.4	67.7 ±0.5	35.3 ±0.5	51.5 ±0.4	73.3 ±0.4
	Turn-Level	5.7 ±0.5	36.1 ±0.5	68.0 ±0	35.3 ±0.5	51.7 ±0.2	72.8 ±0.4
	Mixed	$\mathbf{8.3}_{\pm 0.5}$	$\mathbf{41.5}_{\pm 1.8}$	$\mathbf{68.3}_{\pm 0.5}$	$\mathbf{37.7}_{\pm 0.5}$	$\mathbf{53.0}_{\pm 0.4}$	$\mathbf{74.5}_{\pm 0.3}$
Qwen2.5-7B-Instruct	Group-Level	6.0 ±0.0	50.8 ±0.5	69.0 ±0.0	34.7 ±0.5	51.8 ±0.2	$\mathbf{55.8}_{\pm 0.7}$
	Turn-Level	6.7 ±0.5	54.5 ±1	68.7 ±0.5	33.7 ±0.5	51.2 ±0.2	54.8 ±0.3

Appendix

Table 3: Quantitative results under different Advantage Level. Mixed Advantage consistently achieves the best overall downstream performance compared with Group-Level and Turn-Level Advantage. Results are averaged over 3 random seeds, and each result is reported with its standard deviation as a subscript.

Figure 6: Effect of the Turn-level advantage weight

\alpha

on dialogue strategy and benchmark performance. We sweep

\alpha\!\in\!\{0.0,0.1,\dots,1.0\}

in the mixed advantage

A=\alpha A^{t}+(1-\alpha)A^{g}

on Qwen2.5-7B-Instruct and run

5

independent trials per configuration. Error bars denote one standard deviation across the

5

trials; small dots denote per-trial values. Increasing

\alpha

correlates with longer, more stable EMPA dialogues, while EmoBench and EQ-Bench (single-turn benchmarks) are largely insensitive to

\alpha

Figure 7: Training reward curves under three Judgers for four base models. Each panel reports the per-step reward returned by the Judger during MICA training; transparent lines show raw rewards and bold lines show a centered moving average with window

5

. The dotted horizontal line marks reward

0

. Reward magnitudes are not directly comparable across Judgers due to per-Judger scoring calibration.

Figure 8: Cosine similarity of Judger score directions on fixed EMPA traces. Gemini-2.5-pro generates dialogue trajectories for the $30$ EMPA test cases, and the same response turns are rescored by Qwen3-235B, MiniMax-M2.5, and GLM-4.7. Each entry reports the averaged cosine similarity between per-response score-change vectors $(\Delta x_{t},\Delta y_{t},\Delta z_{t})$ .

Advantage Level	Reward Design	EMPA		EmoBench (Acc. %)			EQ-Bench
Advantage Level	Reward Design	Pass	Score	EA	EU	Overall	Score
Group-level	ADR	1.0 ±0.0	18.2 ±0.4	65.7 ±0.5	30.3 ±0.5	48.0 ±0.0	69.7 ±0.2
Group-level	IDR	5.7 ±0.5	38.5 ±0.4	67.7 ±0.5	35.3 ±0.5	51.5 ±0.4	73.3 ±0.4
Turn-level	ADR	3.3 ±0.5	31.5 ±0.7	68.0 ±0.0	32.3 ±0.5	50.2 ±0.2	72.1 ±0.1
Turn-level	IDR	5.3 ±0.5	34.0 ±0.4	67.7 ±0.5	33.3 ±0.5	50.5 ±0.4	72.3 ±0.4
Mixed	IDR	$\mathbf{8.3}_{\pm 0.5}$	$\mathbf{41.5}_{\pm 1.8}$	$\mathbf{68.3}_{\pm 0.5}$	$\mathbf{37.7}_{\pm 0.5}$	$\mathbf{53.0}_{\pm 0.4}$	$\mathbf{74.5}_{\pm 0.3}$

Appendix

Table 4: Quantitative results under different Reward Design on Qwen3-8B. Results are averaged over 3 random seeds, and each result is reported with its standard deviation as a subscript.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 410 papers

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue
Aug 11, 2026Yi Wei, Shuo Jiang, Huaixia Dou +5Reinforcement Learning with Verifiable Rewards Curriculum Learning
Qwen DianJin Team, Alibaba Cloud Computing · Beihang University · School of Computer Science and Technology, Soochow University
SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
Sep 9, 2026Jianing Wang, Xintao Wang, Aili Chen +7RL for Language Models Reinforcement Learning
Fudan University · Hello Group
HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2

Aug 11, 2026J/K move · Enter open · S save

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue

Yi Wei, Shuo Jiang, Huaixia Dou, Jie Zhu+4

Qwen DianJin Team, Alibaba Cloud Computing · Beihang University · School of Computer Science and Technology, Soochow University

Reinforcement Learning with Verifiable Rewards Curriculum Learning Language Model Self-Improvement Emotion-Aware Dialogue Generation

Open paper AskPDF arXiv

Large language models have demonstrated conversational capabilities, yet empathetic competence remains challenging. Empathetic support is inherently multi-turn and path-dependent: users disclose concerns gradually, emotions evolve over time, and early responses shape trust and receptivity. Reinforcement learning with verifiable emotion rewards provides scalable supervision for long-horizon interactions. However, existing methods evolve the dialogue policy while keeping its training interaction distribution fixed, creating a mismatch between policy competence and training experience. We introduce a dual-loop self-evolution framework driven by verifiable emotion feedback. With the user simulator and verifier frozen, the inner loop optimizes the multi-turn policy using continuous emotion rewards, while the outer loop uses the same outcomes to estimate policy-relative interaction utility and adapt experience. To obtain estimates from sparse, stochastic rollouts, the framework holds the scenario and interaction state constant within each group and prioritizes conditions whose group pass rates lie near the policy's competence boundary. A hierarchical controller shares evidence across support intents, while uncertainty-guided exploration and uniform rehearsal prevent premature exclusion. The resulting distribution generates trajectories, closing both loops without increasing the rollout budget. On SAGE, our framework raises Qwen3-8B Overall from 53.87 to 79.24 and outperforms protocol-matched uniform emotion-reward reinforcement learning by 7.23 points.

Abstract

RL for Language Models

Explore similar work

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue

SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents