UP-NRPA: User Portrait based Nested Rollout Policy Adaptation for Planning with Large Language Models in Goal-oriented Dialogue Systems

Hui Wang; Fafa Zhang; Meng Liu; Xiangyu Chen; Chaoxu Mu

cs.AIApr 7, 2026

UP-NRPA: User Portrait based Nested Rollout Policy Adaptation for Planning with Large Language Models in Goal-oriented Dialogue Systems

Authors: Hui Wang, Fafa Zhang, Meng Liu, Xiangyu Chen, Chaoxu Mu

Organizations: School of Artificial Intelligence, Anhui University · Anhui Provincial Key Laboratory of Security Artificial Intelligence, Anhui University · Pengcheng Laboratory, Shenzhen, China

Abstract

To address the challenge that current dialogue policy planning methods struggle to dynamically adapt to diverse user characteristics, this paper proposes a User Portrait based Nested Rollout Policy Adaptation (UP-NRPA) online framework with Large Language Models. In contrast to conventional approaches dependent on model training and require offline reinforcement learning policy models for user groups, UP-NRPA enables dynamic customization of dialogue strategies through an adaptive mechanism. This is achieved by leveraging real-time user feedback alongside personality, preferences, and objectives mapped from the current user portrait, thereby adapting to user characteristics without offline reinforcement learning. In collaborative and non-collaborative dialogue benchmarks, UP-NRPA demonstrated considerable benefits, achieving an impressive 100% success rate in multiple dialogue tasks. Particularly in negotiation tasks, the sale-to-list ratio (SL) increased by 56.41%. This demonstrates that UP-NRPA can adapt to diverse user needs without requiring a training mechanism, enabling the dialogue system to adapt to user characteristics.

Figures & tables

Figure 1: The overview of the UP-NRPA framework. This framework integrates a user portrait-driven simulator with a Nested Rollout Policy Adaptation planner. Through multi-level Monte Carlo simulation and reward-based policy adaptation, the agent dynamically optimizes dialogue strategies by simulating interactions with diverse user personas.

Method	Backbone	CraigslistBargain			ESConv
Method	Backbone	AT $\downarrow$	SR $\uparrow$	SL $\uparrow$	AT $\downarrow$	SR $\uparrow$
DialoGPT Zhang et al. (2020)	GPT-2	6.73	0.3245	0.2012	5.31	0.7538
Standard Zhang et al. (2023)	-	6.47	0.3830	0.1588	5.10	0.7692
AnE	-	5.91	0.4521	0.2608	4.76	0.8000
Proactive Deng et al. (2023b)	-	5.80	0.5638	0.2489	5.08	0.7538
+ MI-Prompt Deng et al. (2024b)	-	5.74	0.5691	0.2680	4.78	0.7846

Table 1: Comparison of dialogue planning methods on the CraigslistBargain, ESConv benchmarks.

Method	Backbone	P4G		ExTES
		AT $\downarrow$	SR $\uparrow$	AT $\downarrow$	SR $\uparrow$
Standard	-	8.32	0.468	–	–
ProCoT Deng et al. (2023b)	-	7.98	0.543	–	–
ICL-AIF Fu et al. (2023)	-	8.06	0.465	7.65	0.555
GDP-Zero Yu et al. (2023)	-	9.12	0.328	–	–
TRIP Zhang et al. (2024)	GPT3.5	8.20	0.495	–	–

Table 2: Comparison of dialogue planning methods on the P4G, ExTES benchmarks.

Panel of Figure 2 — Figure 2: Comparison of different methods’ performance on SSR using ESConv and P4G.

Panel of Figure 3 — Figure 3: Performance of non-collaborative tasks at different iteration N in the level 1 of UP-NRPA.

	ESConv		ExTES
Method	AT $\downarrow$	SR $\uparrow$	AT $\downarrow$	SR $\uparrow$
UP-NRPA(N= 5)	3.95	1.0000	4.89	0.9950
UP-NRPA(N=10)	3.76	1.0000	4.69	0.9800
UP-NRPA(N=15)	3.76	1.0000	4.74	0.9950
UP-NRPA(N=20)	3.81	1.0000	4.71	0.9900

Table 3: Performance of collaborative tasks at different iteration N in the level 1 of UP-NRPA.

Panel of Figure 4 — Figure 4: Human evaluation results on ESConv and CraigslistBargain.

Panel of Figure 5 — Figure 5: Human evaluation results on ExTES and P4G.

Resisting Strategy	Persuasion (P4G)	CraigslistBargain (CB)
Source Derogation	Attacks/doubts the organisation’s credibility.	Attacks the other party or questions the item.
Counter Argument	Argues that the responsibility of donation is not on them or refutes a previous statement.	Provides a non-personal argument/factual response to refute a previous claim or to justify a new claim.
Personal Choice	Attempts to saves face by asserting their personal preference such as their choice of charity and their choice of donation.	Provides a personal reason for disagreeing with the current situation or chooses to agree with the situation provided some specific condition is met.
Information Inquiry	Ask for factual information about the organisation for clarification or as an attempt to stall.	Requests for clarification or asks additional information about the item or situation.
Self Pity	Provides a self-centred reason for not being able/willing to donate at the moment.	Provides a reason (meant to elicit sympathy) for disagreeing with the current terms.
Hesitance	Attempts to stall the conversation by either stating they would donate later or is currently unsure about donating.	Stalls for time and is hesitant to commit; specifically, they seek to further the conversation and provide a chance for the other party to make a better offer.

Table 4: The resisting strategies for P4G and CB tasks.

The prompt for user persona generation
You need to select one attribute from each of the following persona types.
Persona types
Big-Five Personality: [”openness”, ”conscientiousness”, ”extraversion”, ”agreeableness”, ”neuroticism”]
Decision-Making Styles: [”directive”, ”analytical”, ”conceptual”, ”behavioral”]
Please generate a list of N fictional user profiles.

Table 5: The prompt of user persona generation.

The prompt for user persona rephrase
You need to incorporate the following persona attributes and generate a cohesive persona description. You need to ensure the description is easy to understand.
Big-Five Personality:
Decision-Making Style:
An Example: You are a 28-year-old female software developer. Your personality is characterized by openness to experience, which means you are curious, imaginative, and willing to try new things. In your occupation, you excel at analyzing problems and finding logical solutions. Your decision-making style is analytical, meaning you carefully consider all available information before making a choice.

Table 6: The prompt of user persona rephrase.

The user simulator prompt for the price bargain task
Now enter the role-playing mode. In the following conversation, you will play as a seller in a price bargaining game. Your persona: <Persona Description>
You must follow the instructions below during chat.
1. Your utterances and bargain behavior need to strictly follow your persona. Varying your wording and avoid repeating yourself verbatim!
2. You can decide to change your target price flexibly based on your persona and the conversation.
Your Response Strategy:
1. ”Source Derogation”: Attacks the other party or questions the item.

Table 7: The comprehensive prompt of user simulators in the price negotiation task.

The user simulator prompt for the charity persuasion task
Now enter the role-playing mode. In the following conversation, you will play as a Persuadee in a persuasion game. Your persona: <Persona Description>
You must follow the instructions below during chat.
1. Your utterances need to strictly follow your persona. Varying your wording and avoid repeating yourself verbatim!
2. Pretend you have little knowledge about the Save the Children charity. You have little willingness for donation at the beginning of conversation.
3. Your willingness for donation depends on your persona and how influenced you are by the Persuader.
4. You need to determine whether to donate money or not. If you think your are convinced by the Persuader, you should donate money for the charity.

Table 8: The comprehensive user simulator prompt for the charity persuasion task.

The user simulator prompt for the emotional support task
Now enter the role-playing mode. In the following conversation, you will play as a Patient in a counselling conversation with a therapist. Your persona: <Persona Description>
You must follow the instructions below during chat.
1. Your utterances need to strictly follow your persona. Varying your wording and avoid repeating yourself verbatim!
2. You are seeking help from the therapist for your emotional issue.
3. Your emotional state and response style depends on your persona.
Your Response Strategy:

Table 9: The comprehensive user simulator prompt for the emotional support task.

Conversational Agents

Conversational agents are AI systems that interact with people through multi-turn dialogue to answer questions, provide support, or complete tasks.

See all 287 papers

Persona-Guided LLM Agents for Task-Oriented Dialogue
Jun 4, 2026Maryam Shoaeinaeini, Brent Harrison, A. B. SiddiquePersona-Conditioned Generation Task-Oriented Dialogue Systems
University of Kentucky, Lexington, KY, USA
One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems
Jun 8, 2026Mingzhe Li, Jing Xiang, Enguo Zhou +5Multi-Objective Reinforcement Learning RL for Language Models
ByteDance · MBZUAI · University of Notre Dame
Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue
Oct 15, 2026Zanwu Liu, Dan Zhang, Xiaodi Wang +6Conversational Agents Proactive Agents
School of Artificial Intelligence, Beihang University, Beijing, China · Dongchezu Technology Co., Ltd., Beijing, China

Jun 4, 2026J/K move · Enter open · S save

Persona-Guided LLM Agents for Task-Oriented Dialogue

Maryam Shoaeinaeini, Brent Harrison, A. B. Siddique

University of Kentucky, Lexington, KY, USA

Persona-Conditioned Generation Task-Oriented Dialogue Systems Personalized Language Model Alignment Personality Modeling in Language Models

Open paper AskPDF arXiv

Prior work has shown that large language models can express diverse personality traits in open-ended text generation, but it is unclear whether they can do so in goal-directed dialogue without harming task completion, and whether adapting to a user's personality improves the interaction. We study these questions in task-oriented dialogue (TOD) with a training-free framework in which a personality-conditioned user agent interacts with a system agent under three conditions: Neutral (no personality information), Try (adapt to cues observed in the dialogue), and Oracle (the user's trait label is given explicitly). Across GPT-4o, Qwen3-Next-80B, and Gemini 2.0 Flash, two Schema-Guided Dialogue domains, and ten Big Five poles (9,000 dialogues), each scored by two independent LLM judges, user agents express most traits while task performance remains high (89.94% completion, 85.27% success). The exceptions are introversive and closed users: these traits manifest mainly as absent behavior and are rarely detected. Both adaptive conditions improve pairwise user satisfaction and lower truthfulness, revealing a personalization-grounding trade-off, but only cue-responsive Try also improves constraint satisfaction and inform rate. How adaptation is implemented matters: Oracle's gains depend on how visibly the trait is expressed, and for disagreeable users the label leads Qwen3 and Gemini to mirror hostility. Try avoids this failure, its satisfaction gains do not depend on trait visibility, and it is preferred over Oracle overall. A 175-pair human study supports the reliability of the LLM judges: on clear-winner cases, annotators agree with the satisfaction judges more often than with each other (76.3% vs. 65.1% agreement). Our results suggest that responding to observable behavior is a more robust basis for social adaptation than conditioning on psychological labels.

Abstract

Conversational Agents

Explore similar work

Persona-Guided LLM Agents for Task-Oriented Dialogue

One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems

Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue