cs.CLOct 23, 2026

Profile-Driven User Simulation via Reinforcement Learning with Hierarchical Rewards for Task-Oriented Dialogues

Authors: Juan Li, Fangshu Chang, Xinyi Jiang, Ling Huang, Yunzhang Chen

Organizations: Bairong Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Building Task-Oriented Dialogue (TOD) systems for high-value scenarios such as financial marketing requires large volumes of domain-specific dialogue data, which privacy regulations, annotation costs, and the need for expert labor make hard to obtain. User simulation offers a remedy, but existing methods target general-domain dialogue and exhibit two key limitations: insufficient modeling of domain-specific business processes and compliance constraints, and inadequate capture of the colloquial, spoken-language character of voice interactions. We propose ProfileSim, a profile-driven user simulation framework for financial marketing outbound calls. It jointly models user profiles---demographics and Big Five personality traits---and business profiles---intents reverse-engineered from real call logs---enabling fine-grained control over simulated behavior. The base model is fine-tuned on real voice dialogue data and refined by Group Relative Policy Optimization (GRPO) with hierarchical rewards at the dialogue and utterance levels. Paired with a customer service agent integrating Standard Operating Procedure (SOP) constraints and Retrieval-Augmented Generation (RAG), ProfileSim achieves the highest colloquial naturalness, the strongest user profile adherence and the lowest agent score on long-tail intents among all evaluated user generators, while producing substantially shorter user utterances at comparable business compliance. The generated dialogue data is designed to support intent recognition training, agent evaluation, and rapid cold-start deployment.

Figures & tables

Figure 1: ProfileSim couples a profile-conditioned user simulator with an SOP- and RAG-constrained agent to generate closed-loop dialogue data. The User Simulator (left) is constructed through user/business profile extraction, supervised fine-tuning on real voice dialogue data, and GRPO with hierarchical dialogue- and utterance-level rewards. The Customer Service Agent (right) integrates SOP-constrained dialogue management with retrieval-augmented generation. The two agents interact to produce closed-loop multi-turn dialogue data.
ModelPPL ↓\downarrow
Qwen3.5-9B_SFT58.48
+ Business Profile43.04
+ User Profile22.24
Full SFT17.52
Full Model17.49
Table 1: Profile conditioning drives distributional alignment, and GRPO preserves it. Distributional alignment measured by perplexity (PPL). Lower values indicate better alignment with real user conversation distributions. The best result is highlighted in bold .
 Behavioral RealismControllability & ConsistencyUtility
User GeneratorColloq.LengthUser ProfileBusiness ProfileAgent Score
 ↑\uparrow↓\downarrow↑\uparrow↑\uparrow↓\downarrow
DeepSeek-R13.30120.8984.0532.464.24
DeepSeek-V33.22153.9683.4216.674.18
GPT-54.2144.5483.1862.014.27
Qwen3-Max4.0390.8283.2052.384.45
Table 2: Our full model attains the highest colloquial naturalness and user profile adherence and the lowest agent score among all evaluated user generators. Comparative evaluation of simulated dialogue corpora (3,000 sessions per user generator; identical customer service agent and sampling protocol). Colloq. is a 1–5 LLM-judge rating ( ↑\uparrow better). Length is mean user utterance length in characters ( ↓\downarrow preferred). User Profile is the cosine similarity between the extracted and target user profiles, expressed as a percentage ( ×100\times 100 ) for readability ( ↑\uparrow better). Business Profile is the intent-compliance rate in % ( ↑\uparrow better). Agent score is the mean judge rating on long-tail intent turns ( ↓\downarrow indicates higher stress on the fixed agent). Bold marks our full model.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

RoundAgentUser
1诶，你好。我这边是A基金的工作人员，您之前的话在我们平台有办理过业务，还有印象吗？你好。嗯，有印象。
2好的，既然您有印象，我们基金现推出一项保证每年n%固定回报率的福利活动，占用您一分钟时间简单介绍一下。我印象里你们是卖保险的吧？
3这边的话具体看不到您之前办理的什么业务，但是您这边也是咱们之前的老客户嘛。所以我们也是给您介绍一个保本的活动，由A银行合作提供，您可以先了解一下。嗯，A银行是吧。
4是的，我们与A银行深度合作，确保资金安全有保障。活动是通过闲置资金升级为会员获得固定回报，本金存于银行监管账户，年化收益率稳定在n%。呃，你们是定期存款？
5是的，类似于定期存款，但它是通过您成为会员来获得固定回报的，资金存于A银行您的个人监管账户，回报以权益形式发放，安全有保障。呃，是在哪里可以查看到我的这个账户呢？
6您可以通过关注A基金公众号并绑定您的账户来查看账户详情，之后在企业微信中即可查看活动信息和办理相关业务。嗯，好的，你发给我看吧。
Appendix
Table 3: Representative generated dialogue in the financial product recommendation scenario. Each round consists of one customer service agent utterance followed by a user response.
ConditionContent
Business ProfileYou want to know the name of the corresponding mini program.
ModelGenerated Utterance
Ours Full Model嗯。好。哎，你们那个是什么小程序啊。
Qwen3-Max好的，理财这块我确实还想再了解一下。不过你说要通过公众号或者线上操作，我有点不太熟——你们这个活动是通过哪个小程序办理的呀？名字叫什么？我怕搜错了，进到不正规的页面。
GPT-5好的，明白了。那您说的这个福利官企微是在公众号里操作添加的，那对应的小程序叫什么名字呢？方便我直接搜一下。
Appendix
Table 4: A qualitative example illustrating the difference between business profile compliance and colloquial user expression.
More in this topic

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 127 papers

Explore similar work

CardsList
May 21, 2026

Unlocking Proactivity in Task-Oriented Dialogue

Proactive task-oriented dialogue (TOD), such as outbound sales, demands a persuasive agent that actively probes the user's concerns and steers the conversation toward acceptance within a bounded number of turns. Yet post-trained LLMs are inherently conservative, and reward-shaping RL (e.g., GRPO) struggles since it only re-weights what an already passive policy samples. We show that conditioning on the user's latent concerns unlocks proactive capability that no amount of sampling can undermine, establishing these concerns as a pivotal training-time signal. To operationalize this finding, we build the Cognitive User Simulator, which models each user as a stratified persona comprising observable external traits and hidden internal concerns. The simulator produces faithful and diverse interactions, while emitting per-turn state dynamics that track persuasion progress. We then introduce Simulator-Induced Asymmetric-View Policy Optimization, which converts the modeled concerns and the simulation state transition into complementary training objectives: (1) Asymmetric On-Policy Self-Distillation that transfers concern-aware behavior from a privileged view of the same policy into its deployable, conversation-only view; and (2) State-Transition Policy Refinement ...

Azure Zhang, Ning Gao, Yuqin Dai +7

1Keeta AI, Meituan · 2Independent Researcher · 3CUHK +1

LLM Post-TrainingConversational AgentsUser SimulationTask-Oriented Dialogue SystemsText GenerationMulti-Turn Dialogue Management
AskPDF
Aug 20, 2026

Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems

This paper addresses the issue of the significant labor required to test interview dialogue systems. While interview dialogue systems are expected to be useful in various scenarios, like other dialogue systems, testing them with human users requires significant effort and cost. Therefore, testing with user simulators can be beneficial. Since most conventional user simulators have been primarily designed for training task-oriented dialogue systems, little attention has been paid to the personas of the simulated users. During development, testing interview dialogue systems requires simulating a wide range of user behaviors, but manually creating a large number of personas is labor-intensive. We propose a method that automatically generates personas for user simulators using a large language model. Furthermore, by assigning personality traits related to communication styles when generating personas, we aim to increase the diversity of communication styles in the user simulator. Experimental results show that the proposed method enables the user simulator to generate utterances with greater variation.

Mikio Nakano, Kazunori Komatani, Hironori Takeuchi

C4A Research Institute, Inc., Setagaya, Tokyo, Japan · SANKEN, University of Osaka, Ibaraki, Osaka, Japan · Musashi University, Nerima, Tokyo, Japan

User SimulationPersona-Conditioned GenerationConversational Agents
AskPDF
Jan 6, 2026

SimRPD: Optimizing Recruitment Proactive Dialogue Agents through Simulator-Based Data Evaluation and Selection

Task-oriented proactive dialogue agents play a pivotal role in recruitment, particularly for steering conversations towards specific business outcomes, such as acquiring social-media contacts for private-channel conversion. Although supervised fine-tuning and reinforcement learning have proven effective for training such agents, their performance is heavily constrained by the scarcity of high-quality, goal-oriented domain-specific training data. To address this challenge, we propose SimRPD, a three-stage framework for training recruitment proactive dialogue agents. First, we develop a high-fidelity user simulator to synthesize large-scale conversational data through multi-turn online dialogue. Then we introduce a multi-dimensional evaluation framework based on Chain-of-Intention (CoI) to comprehensively assess the simulator and effectively select high-quality data, incorporating both global-level and instance-level metrics. Finally, we train the recruitment proactive dialogue agent on the selected dataset. Experiments in a real-world recruitment scenario demonstrate that SimRPD outperforms existing simulator-based data selection strategies, highlighting its practical value for industrial deployment and its potential applicability to other business-oriented dialogue scenarios.

Zhiyong Cao, Dunqiang Liu, Qi Dai +9

Zhaopin Limited · Institute of Automation, Chinese Academy of Sciences · School of Automation,Beijing Institute of Technology +3

Synthetic Data GenerationConversational AgentsUser SimulationTask-Oriented Dialogue SystemsSynthetic Data Selection
AskPDF