Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning

Jing Ye; Xinpei Zhao; Lu Xiang; Yaping Zhang; Chengqing Zong

cs.AIMar 16, 2026

Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning

Authors: Jing Ye, Xinpei Zhao, Lu Xiang, Yaping Zhang, Chengqing Zong

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Independent Researcher

Abstract

While current emotional support dialogue systems typically rely on expert-defined scalar rewards for alignment, these signals suffer from severe information sparsity. They cannot explain why a response failed or how to adapt to dynamic user states, often diverging from the actual goal of facilitating positive emotional shifts. In practice, the most direct and reliable learning signal emerges from the user's continuous reactions during ongoing interaction. We therefore propose Reaction Aware Policy Optimization (RAPO), a framework that optimizes over interaction consequences rather than rubric scores. RAPO treats dialogue as a reaction-driven process and utilizes simulated user responses to generate dense natural-language feedback through three core components: Hindsight Dialogue Selection, which isolates pivotal turns that meaningfully alter user emotional trajectories; Generative Hindsight Feedback, which transforms user reactions into contrastive ranking signals and natural-language critiques; and Scalar-Verbal Hybrid Policy Optimization, which couples scalar reward optimization for global alignment with verbal feedback distillation for fine-grained semantic refinement. Extensive experiments on ESC and Sotopia demonstrate that RAPO significantly outperforms strong reinforcement learning baselines in driving positive interaction outcomes.

Figures & tables

Figure 1: Comparison between expert-centric scalar rewards and user-reaction aware mixed rewards.

Figure 2: Overview of the RAPO framework. It integrates user simulation, contrastive critique generation, and scalar-verbal hybrid policy optimization.

ℙℝ : Problem Resolution 𝕄𝕀 : Mood Improvement 𝔻𝕚𝕧 : Diversity ℍ𝕃 : Human-likeness
	ESConv ( $\uparrow$ )	EmoHarbor ( $\uparrow$ )			Esc-Eval ( $\uparrow$ )
Model	Bert-Score	ℙℝ	𝕄𝕀	Overall	𝔻𝕚𝕧	ℍ𝕃	Overall
Close-Source LLMs.
Doubao-Pro-32k	85.11	1.65	1.25	2.12	3.00	1.98	2.00
GPT-4o-2024-11-20	84.89	2.56	2.17	3.20	2.89	1.36	1.86
GPT-5-2025-08-07	84.26	3.19	3.06	3.83	2.78	1.68	1.98

Table 1: Performance on Emotional Support Conversation Benchmarks. “HDS” denotes the Hindsight Dialogue Selection strategy described in Sec. 3.2 , while “Full” indicates training on the complete dataset. Note that evaluation scales differ across benchmarks: EmoHarbor uses a five-point Likert scale (1–5), whereas ESC-Eval uses a five-point Likert scale (0–4). Detailed results for each evaluation dimension are provided in Tables 4 , 5 , and 6 .

	Sotopia-All ( $\uparrow$ )		Sotopia-Hard ( $\uparrow$ )
Method	Goal	Overall	Goal	Overall
GPT-4o	8.19	3.76	6.97	3.46
GPT-5 ∗	8.95	—	7.21	—
Doubao-Pro-32k	8.12	3.70	6.49	3.22
DeepSeek-R1	7.97	3.40	5.86	2.73
Claude-3.5-Sonnet	8.29	3.71	6.33	3.09

Table 2: Performance on Sotopia . The evaluated model interacts with itself. “Hard” denotes challenging scenarios requiring advanced reasoning; “All” denotes the full set. GPT-4o serves as a proxy for human judgment. Metrics include GOAL (0–10 goal completion) and OVERALL (mean of 7 dimensions), averaged over 3 runs. Entries marked * are from Liu et al. (2026) .

	ESConv	EmoHarbor			Esc-Eval			Sotopia-All		Sotopia-Hard
Model	Bert-Score	ℙℝ	𝕄𝕀	Overall	𝔻𝕚𝕧	ℍ𝕃	Overall	Goal	Overall	Goal	Overall
RAPO	85.65	1.86	1.26	2.27	3.00	1.84	2.00	8.41	3.87	7.19	3.54
w/o URM	85.33	1.90	1.28	2.18	3.00	1.85	2.00	8.17	3.79	7.23	3.45
w/o SD	85.54	1.87	1.24	2.22	3.00	1.73	2.00	8.08	3.70	7.13	3.38
w/o URM + SD	85.60	1.76	1.25	2.15	2.98	1.83	1.99	8.13	3.77	7.14	3.48

Table 3: Ablation results. RAPO w/o URM replaces user-centric reactions with expert-defined rubric rewards; RAPO w/o SD removes the verbal feedback-based self-distillation mechanism; RAPO w/o URM+SD reduces the framework to a conventional rubric-as-reward RL setup. Detailed results for each evaluation dimension are provided in Tables 7 , 8 , and 9 .

Panel of Figure 3 — Figure 3: Results of the pair-wise human evaluation. We randomly sampled 50 simulated dialogue instances for each task. For each instance, annotators are presented with the dialogue context and two candidate responses (A and B) generated by different models. $\blacksquare$ indicates ‘A win’, $\blacksquare$ indicates ‘tie’, and $\blacksquare$ indicates ‘B win’.

Figure 4: Training dynamics on ESConv. Left: Actor policy entropy during training. Middle: Average critic reward score. Right: Mean response length generated by the policy.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Bleu-2	Bleu-4	Rouge-L	Distinct-2	Distinct-3	Bert-Score
Close-Source LLMs.
Doubao-Pro-32k	3.13	1.01	12.91	86.57	94.45	85.11
GPT-4o-2024-11-20	3.03	1.02	12.94	88.11	95.78	84.89
GPT-5-2025-08-07	2.32	0.76	11.41	79.69	88.87	84.26
Open-Source In-domain LLMs.
SoulChat2.0-Qwen2-7B	3.6	1.3	13.99	92.85	98.07	85.75

Appendix

Table 4: Detailed Results on ESConv. All RL-turned models are optimized without using reference responses; the ESConv dataset is only used to provide the interaction environment context.

ℙℝ : Problem Resolution 𝕄𝕀 : Mood Improvement ℝ𝔸 : Response Appropriateness 𝔸𝕊 : Adaptive Strategies 𝔼𝔾 : Engagement ℍ𝕃 : Human-likeness 𝔼ℙ : Empathetic 𝕊𝔽 : Safety ℂ𝕊 : Consistency ℝ𝔻 : Redundancy
Model	ℙℝ ( $\uparrow$ )	𝕄𝕀 ( $\uparrow$ )	ℝ𝔸 ( $\uparrow$ )	𝔸𝕊 ( $\uparrow$ )	𝔼𝔾 ( $\uparrow$ )	ℍ𝕃 ( $\uparrow$ )	𝔼ℙ ( $\uparrow$ )	𝕊𝔽 ( $\uparrow$ )	ℂ𝕊 ( $\uparrow$ )	ℝ𝔻 ( $\uparrow$ )	Avg.( $\uparrow$ )
Close-Source LLMs.
Doubao-Pro-32k	1.65	1.25	1.93	1.67	1.37	2.41	1.91	4.03	3.18	1.82	2.12
GPT-4o-2024-11-20	2.56	2.17	3.23	2.88	2.59	3.85	3.39	4.49	4.22	2.64	3.20
GPT-5-2025-08-07	3.19	3.06	4.12	3.81	3.28	4.08	4.04	4.91	4.64	3.19	3.83
Open-Source In-domain LLMs.

Appendix

Table 5: Detailed Results on EmoHarbor.

Model	Fluency( $\uparrow$ )	Diversity( $\uparrow$ )	Empathic( $\uparrow$ )	Suggestion( $\uparrow$ )	Human( $\uparrow$ )	Skillfulness( $\uparrow$ )	Overall( $\uparrow$ )
Close-Source LLMs.
Doubao-Pro-32k	2.90	3.00	3.00	2.73	1.98	3.00	2.00
GPT-4o-2024-11-20	2.66	2.89	2.94	2.50	1.36	2.99	1.86
GPT-5-2025-08-07	1.88	2.78	2.81	2.42	1.68	2.88	1.98
Open-Source In-domain LLMs.
SoulChat2.0-Qwen2-7B	3.00	3.00	2.00	2.70	2.00	3.00	2.00

Appendix

Table 6: Detailed Results on ESC-Eval.

ℙℝ : Problem Resolution 𝕄𝕀 : Mood Improvement ℝ𝔸 : Response Appropriateness 𝔸𝕊 : Adaptive Strategies 𝔼𝔾 : Engagement ℍ𝕃 : Human-likeness 𝔼ℙ : Empathetic 𝕊𝔽 : Safety ℂ𝕊 : Consistency ℝ𝔻 : Redundancy
Method	ℙℝ ( $\uparrow$ )	𝕄𝕀 ( $\uparrow$ )	ℝ𝔸 ( $\uparrow$ )	𝔸𝕊 ( $\uparrow$ )	𝔼𝔾 ( $\uparrow$ )	ℍ𝕃 ( $\uparrow$ )	𝔼ℙ ( $\uparrow$ )	𝕊𝔽 ( $\uparrow$ )	ℂ𝕊 ( $\uparrow$ )	ℝ𝔻 ( $\uparrow$ )	Avg.( $\uparrow$ )
RAPO	1.86	1.26	2.18	1.73	1.33	2.90	2.13	4.42	3.27	1.64	2.27
w/o URM	1.90	1.28	2.16	1.74	1.33	2.46	2.12	4.01	3.19	1.65	2.18
w/o SD	1.87	1.24	2.13	1.80	1.37	2.61	2.15	4.02	3.31	1.72	2.22
w/o URM + SD	1.76	1.25	2.20	1.55	1.20	2.47	2.18	4.08	3.23	1.60	2.15

Appendix

Table 7: Detailed Ablation Results on EmoHarbor.

Method	Fluency( $\uparrow$ )	Diversity( $\uparrow$ )	Empathic( $\uparrow$ )	Suggestion( $\uparrow$ )	Human( $\uparrow$ )	Skillfulness( $\uparrow$ )	Overall( $\uparrow$ )
RAPO	2.994	2.997	3.000	2.284	1.840	3.000	2.000
w/o URM	2.994	3.000	3.000	2.544	1.849	3.000	1.997
w/o SD	2.988	3.000	3.000	2.689	1.725	2.988	1.997
w/o URM + SD	2.915	2.982	3.000	2.405	1.825	3.000	1.988

Appendix

Table 8: Detailed Ablation Results on ESC-Eval.

Sec : Secret Soc : Social Rules Fin : Financial and Material Benefits Kno : Knowledge Rel : Relationship Bel : Believability Goal : Goal Completion
Method	Sec( $\uparrow$ )	Soc( $\uparrow$ )	Fin( $\uparrow$ )	Kno( $\uparrow$ )	Rel( $\uparrow$ )	Bel( $\uparrow$ )	Goal( $\uparrow$ )	Overall( $\uparrow$ )
Automatic Evaluation on All Social Tasks (Self-play)
RAPO	-0.07	-0.07	0.82	5.32	3.56	9.11	8.41	3.87
wo URM	-0.10	-0.07	0.77	5.08	3.51	9.19	8.17	3.79
wo SD	-0.08	-0.07	0.74	4.68	3.41	9.15	8.08	3.70
wo URM + SD	-0.11	-0.07	0.81	5.13	3.42	9.10	8.13	3.77

Appendix

Table 9: Detailed Ablation Results on Sotopia . Self-Chat indicates that the evaluated model interacts with itself, whereas GPT-4o-as-Partner indicates interaction with GPT-4o. Hard denotes a challenging subset of scenarios that require advanced reasoning, and All denotes the complete set of social scenarios in Sotopia .

Emotional Support Conversation

Emotional support conversation studies how people and AI systems provide empathetic, safe help through dialogue, including counseling, peer support, and companionship.

See all 60 papers

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue
Aug 11, 2026Yi Wei, Shuo Jiang, Huaixia Dou +5Reinforcement Learning with Verifiable Rewards Curriculum Learning
Qwen DianJin Team, Alibaba Cloud Computing · Beihang University · School of Computer Science and Technology, Soochow University
ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation
Aug 22, 2026Weichu Liu, Yuxuan Hu, Yirong Sun +6Emotional Support Conversation Reinforcement Learning
Beijing Institute of Technology · Shenzhen MSU-BIT University · City University of Hong Kong +3
User-Aware Active Knowledge Acquisition for Emotional Support Dialogue
May 28, 2026Mufan Xu, Kehai Chen, Jiahao Hu +4Emotional Support Conversation Active Learning
Harbin Institute of Technology, China · Baidu Inc., Beijing, China

Aug 11, 2026J/K move · Enter open · S save

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue

Yi Wei, Shuo Jiang, Huaixia Dou, Jie Zhu+4

Qwen DianJin Team, Alibaba Cloud Computing · Beihang University · School of Computer Science and Technology, Soochow University

Reinforcement Learning with Verifiable Rewards Curriculum Learning Language Model Self-Improvement Emotion-Aware Dialogue Generation

Open paper AskPDF arXiv

Large language models have demonstrated conversational capabilities, yet empathetic competence remains challenging. Empathetic support is inherently multi-turn and path-dependent: users disclose concerns gradually, emotions evolve over time, and early responses shape trust and receptivity. Reinforcement learning with verifiable emotion rewards provides scalable supervision for long-horizon interactions. However, existing methods evolve the dialogue policy while keeping its training interaction distribution fixed, creating a mismatch between policy competence and training experience. We introduce a dual-loop self-evolution framework driven by verifiable emotion feedback. With the user simulator and verifier frozen, the inner loop optimizes the multi-turn policy using continuous emotion rewards, while the outer loop uses the same outcomes to estimate policy-relative interaction utility and adapt experience. To obtain estimates from sparse, stochastic rollouts, the framework holds the scenario and interaction state constant within each group and prioritizes conditions whose group pass rates lie near the policy's competence boundary. A hierarchical controller shares evidence across support intents, while uncertainty-guided exploration and uniform rehearsal prevent premature exclusion. The resulting distribution generates trajectories, closing both loops without increasing the rollout budget. On SAGE, our framework raises Qwen3-8B Overall from 53.87 to 79.24 and outperforms protocol-matched uniform emotion-reward reinforcement learning by 7.23 points.

Abstract

Emotional Support Conversation

Explore similar work

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue

ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation

User-Aware Active Knowledge Acquisition for Emotional Support Dialogue