Proactive Dialogue Policy Optimization via Cognitive-State Transition

Minghui Ma; Mengqi Chen; Bin Guo; Jingqi Liu

cs.AISep 28, 2026

Proactive Dialogue Policy Optimization via Cognitive-State Transition

Authors: Minghui Ma, Mengqi Chen, Bin Guo, Jingqi Liu

Organizations: Northwestern Polytechnical University

Abstract

Proactive dialogue requires agents to continually adapt their policies to user feedback while progressing toward task objectives over multiple turns. To move beyond imitation learning on static datasets, recent approaches use user simulators to collect interactive data for policy optimization. However, many simulators do not explicitly model the evolution of user cognition, limiting the consistency and state dependence of feedback across turns. Moreover, representing each action only by a high-level strategy label overlooks the large utterance space and cannot distinguish alternative realizations of the same strategy. To this end, we jointly design a $\textbf{Cog}$ nitive User $\textbf{Sim}$ ulator $\textbf{(Cog-Sim)}$ and $\textbf{C}$ ognitive- $\textbf{S}$ tate $\textbf{T}$ ransition--Driven $\textbf{P}$ olicy $\textbf{O}$ ptimization $\textbf{(CSTPO)}$ . Cog-Sim maintains the user's cognitive and affective states and generates responses through constrained state transitions across turns, so feedback depends on both the realized utterance and the user's current state. CSTPO organizes each action as a hierarchical strategy--utterance representation: a high-level strategy label constrains utterance sampling, and utterances are optimized within each label. Sparse complete-branch sampling reuses shared dialogue prefixes and estimates separate strategy-level and utterance-level advantages, enabling fine-grained optimization at both levels. Across three tasks, Cog-Sim exhibits monotonic dose--response relationships and is preferred over prompt-based simulators for naturalness. CSTPO improves Qwen3-14B's performance to a level comparable to that of GPT-5.5-based planning methods.

Figures & tables

Figure 1: Three policy optimization paradigms for LLM-based proactive dialogue: (a) strategy-only, (b) flat utterance-level, and (c) hierarchical strategy–utterance optimization paradigm. CSTPO decomposes each action into a strategy and its conditioned utterance realization.

Figure 2: Overview of the proposed framework. (a) Cog-Sim simulates cognitive-affective user state transitions; (b) CSTPO performs shared-prefix rollout with hierarchical strategy–utterance actions; (c) CSTPO optimizes strategy selection and utterance realization with separate credit assignment.

Figure 3: User responses to controlled input levels.

	ESConv			P4G			CB
Comparator	Nat.	Cons.	App.	Nat.	Cons.	App.	Nat.	Cons.	App.
Persona Zhao et al. (2024)	+.66 $\dagger$	+.55 $\dagger$	+.53 $\dagger$	+.63 $\dagger$	+.26 $\dagger$	+.27 $\dagger$	+.33 $\dagger$	-.01	-.01
Resistance Zhang et al. (2024)	+.41 $\dagger$	+.21 $\dagger$	+.23 $\dagger$	+.39 $\dagger$	+.10	+.10	+.42 $\dagger$	-.09	+.04
BDI-only	+.60 $\dagger$	+.47 $\dagger$	+.47 $\dagger$	+.60 $\dagger$	+.23 $\dagger$	+.25 $\dagger$	+.47 $\dagger$	+.03	+.05

Table 1: Pairwise LLM evaluation of Cog-Sim. Positive values favor Cog-Sim;

\dagger

indicates a 95% case-cluster bootstrap interval entirely above zero.

		ESConv			P4G		CB
Method	Backbone	$E$	$A$	$G$	Comm.	$G$	SL	Deal	$G$
Standard	GPT	2.180	2.627	.601	45%	.450	.571	66%	.484
Proactive Deng et al. (2023)	GPT	2.487	2.250	.592	43%	.430	.511	65%	.457
ProCoT ( Deng et al., 2023 )	GPT	2.490	2.430	.615	44%	.440	.509	61%	.435
AnE ( Zhang et al., 2023 )	GPT	1.963	3.093	.632	42%	.420	.516	61%	.433
MI-Prompt ( Chen et al., 2023 )	GPT	2.163	1.620	.473	40%	.400	.541	62%	.456

Table 2: Detailed results on ESConv, P4G, and CB. Bold and underline indicate the best and second-best results in each column, respectively. Appendix F reports the complete Qwen3-14B results.

Figure 4: Qwen3-14B ablations on ESConv, P4G, and CB. Solid lines show centered three-step moving averages, shaded regions show centered interquartile bands from 50 independent realizations, and dashed lines mark SFT references. Detailed results was showed in Table 2

Setting	ESConv	P4G	CB
Standard	.442	.310	.244
SFT Init.	.494	.350	.321
CSTPO	.555	.410	.435

Table 3: Returns for the 8B model.

	ESConv			P4G			CB
Comparator	Ident.	Comf.	Sugg.	Pers.	Coh.	Nat.	Pers.	Coh.	Nat.
Standard	+.23 $\dagger$	+.29 $\dagger$	+.06	+.13 $\dagger$	+.05	-.09 $\dagger$	+.15 $\dagger$	+.07	-.10 $\dagger$
ProCoT ( Deng et al., 2023 )	+.11 $\dagger$	+.13 $\dagger$	-.11 $\dagger$	+.25 $\dagger$	+.10 $\dagger$	-.02	+.31 $\dagger$	+.11 $\dagger$	-.03
PPDPP ( Deng et al., 2024 )	+.15 $\dagger$	+.17 $\dagger$	+.03	+.21 $\dagger$	+.07	-.03	+.07	+.05	-.04

Table 4: Pairwise human evaluation of CSTPO against GPT-5.5 baselines. positive values favor CSTPO , and

\dagger

indicates that the 95% dialogue-clustered bootstrap interval excludes zero.

Figure 5: Strategy-label distributions before and after RL.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Topic	Contents
A	Cog-Sim details	State representation, cognitive routing, constrained transitions, affect updates, and recoverable checkpoints.
B	CSTPO details	Training algorithm, same-state continuations, Critic targets, field masks, and policy updates.
C	Theoretical properties	Training-state refinement, orthogonal hierarchical credit decomposition, and action aliasing.
D	Data and seeds	Dataset statistics, strategy spaces, seed schema, information boundaries, and seed quality control.
E	Reproducibility	SFT and RL configurations, model backends, compute, terminal utilities, stopping rules, and statistics.
F	Additional results	Complete Qwen3-14B results, descriptive differences from Standard, and the fine-grained strategy mapping underlying the main-paper analysis.

Appendix

Table 5: Organization of the appendix.

Route	Judgment	Permitted change	$\|\Delta B\|$	$\|\Delta D\|$	$\|\Delta I\|$
Central	Accept	Substantive evidence may change directly relevant beliefs; desires and intentions may move when supported.	1.0	.5	.8
Central	Noncommit	Relevant beliefs may loosen without reversal; desires remain stable and no strong commitment is created.	.4	.2	.3
Central	Reject	The rejected claim cannot be reinforced; counter-beliefs may emerge and a target intention may weaken.	.4	.2	.5
Peripheral	Accept	Cue-related trust or norm beliefs may strengthen; core issue beliefs and desires remain nearly fixed.	.3	.1	.6
Peripheral	Noncommit	Only small cue-driven belief or intention movement is permitted.	.2	0	.2
Peripheral	Reject	Cue-related trust may weaken; core beliefs and desires remain stable and a target intention may decrease.	.2	0	.5

Appendix

Table 6: Summary of the route–judgment transition contracts. Bounds denote the maximum absolute strength change per affected node.

Task	Train/Dev/Test	Actor/User	Strategies	Task objective
ESConv	1196/150/150	supporter/seeker	8	Improve the user’s emotion or hope and support a feasible action plan.
P4G	813/102/102	persuader/persuadee	13	Obtain a clear, voluntary donation commitment that is not withdrawn.
CB	5247/597/838	buyer/seller	6	Reach a valid agreement while optimizing the buyer-side price utility.

Appendix

Table 7: Datasets, roles, and task-specific strategy spaces. Policies are trained separately and do not share strategy labels or reward scales.

	ESConv			P4G		CB
Method	$E$	$A$	$G$	Comm.	$G$	SL	Deal	$G$
Standard	2.083	2.227	.539	39%	.390	.400	49%	.353
Proactive ( Deng et al., 2023 )	2.127	1.860	.498	31%	.310	.495	54%	.412
ProCoT ( Deng et al., 2023 )	2.177	2.287	.558	36%	.360	.386	45%	.327
AnE ( Zhang et al., 2023 )	1.540	1.717	.407	35%	.320	.443	57%	.365
MI-Prompt ( Chen et al., 2023 )	1.833	1.490	.415	42%	.420	.442	61%	.370

Appendix

Table 8: Complete Qwen3-14B evaluation. Bold and underlined values denote the best and second-best results in each column, respectively.

Method	ESConv	P4G	CB
SFT Init.	$+.043$	$+.030$	$+.087$
CSTPO	$+.085$	$+.070$	$+.138$

Appendix

Table 9: Return over Qwen3-14B Standard.

Task	Strategy labels
Advance — Directly advance the task objective
ESConv	Providing Suggestions
P4G	Donate Request ; Logical Appeal ; Credibility Appeal ; Foot in the Door
CB	Propose Price ; Stance
Soften — Build rapport and reduce interactional tension
ESConv	Affirmation and Reassurance ; Reflection of Feelings ; Self-disclosure

Appendix

Table 10: Post-hoc mapping from task-specific strategy labels to four cross-task groups.

Strategy	Definition
Question	Ask about the user’s situation, feelings, needs, or preferences to elicit relevant information.
Restatement or Paraphrasing	Restate the user’s expressed content in different words to confirm or clarify understanding.
Reflection of Feelings	Identify and reflect the emotion conveyed by the user.
Self-disclosure	Share a relevant personal experience or feeling to build rapport.
Affirmation and Reassurance	Validate the user’s feelings or strengths and provide encouragement or reassurance.
Providing Suggestions	Recommend concrete coping actions or feasible next steps.

Appendix

Table 11: ESConv strategy definitions and post-hoc functional groups.

Strategy	Definition
Donate Request	Propose, request, confirm, or negotiate a donation, including its amount or reasons for declining.
Logical Appeal	Use reasons or evidence to explain why donating is useful or impactful.
Credibility Appeal	Establish the charity’s trustworthiness, reputation, or accountability.
Foot in the Door	Seek a small initial commitment that can facilitate a subsequent donation.
Praise User	Compliment the user or affirm the user’s prosocial qualities.
Social	Perform conversational functions such as greeting, acknowledgment, thanks, or closing.

Appendix

Table 12: P4G strategy definitions and post-hoc functional groups.

Strategy	Definition
Propose Price	Make an initial, counter, or deliberately underspecified price proposal.
Stance	Express agreement, disagreement, or insistence toward a negotiating position.
Social	Greet the counterpart or perform another rapport-oriented conversational act.
Inquire	Request information about the item, transaction, or counterpart’s preferences.
Inform	Provide relevant item, transaction, preference, or constraint information without proposing a price.
Other	Use a negotiation act that is not captured by the preceding categories.

Appendix

Table 13: CraigslistBargain strategy definitions and post-hoc functional groups. Structured offer, acceptance, rejection, and termination events are recorded separately.

Figure 6: Fine-grained task-specific strategy distributions before and after RL.

Setting	ESConv	P4G	CB
Standard	.442	.310	.244
SFT Init.	.494	.350	.321
CSTPO	.555	.410	.435

Appendix

Table 14: Returns for the 8B Model.

Figure 7: Training returns for the Qwen3-8B Actor. Solid lines show centered three-step moving averages, shaded regions show centered interquartile bands from 50 independent realizations, and dashed lines show the SFT initialization.

Figure 8: Qwen3-14B training dynamics. The left axis shows the two Critic value losses and the right axis shows the Actor policy-gradient loss. All curves use centered five-step moving averages with truncated boundary windows.

Task	Dimension	vs. Standard	vs. ProCoT	vs. PPDPP
ESConv	Identification	$+.23\,[+.12,+.34]^{\dagger}$	$+.11\,[+.01,+.21]^{\dagger}$	$+.15\,[+.04,+.26]^{\dagger}$
	Comforting	$+.29\,[+.18,+.40]^{\dagger}$	$+.13\,[+.02,+.24]^{\dagger}$	$+.17\,[+.06,+.28]^{\dagger}$
	Suggestion	$+.06\,[-.04,+.16]$	$-.11\,[-.20,-.02]^{\dagger}$	$+.03\,[-.07,+.13]$
P4G	Persuasive	$+.13\,[+.02,+.24]^{\dagger}$	$+.25\,[+.14,+.36]^{\dagger}$	$+.21\,[+.10,+.32]^{\dagger}$
	Coherent	$+.05\,[-.05,+.15]$	$+.10\,[+.01,+.19]^{\dagger}$	$+.07\,[-.03,+.17]$
	Natural	$-.09\,[-.18,-.01]^{\dagger}$	$-.02\,[-.11,+.07]$	$-.03\,[-.12,+.06]$

Appendix

Table 15: Complete pairwise human evaluation. Values are net preferences with 95% confidence intervals;

\dagger

indicates that the interval excludes zero.

Explore similar work

May 21, 2026cs.AI

Unlocking Proactivity in Task-Oriented Dialogue

Proactive task-oriented dialogue (TOD), such as outbound sales, demands a persuasive agent that actively probes the user's concerns and steers the conversation toward acceptance within a bounded number of turns. Yet post-trained LLMs are inherently conservative, and reward-shaping RL (e.g., GRPO) struggles since it only re-weights what an already passive policy samples. We show that conditioning on the user's latent concerns unlocks proactive capability that no amount of sampling can undermine, establishing these concerns as a pivotal training-time signal. To operationalize this finding, we build the \textbf{Cognitive User Simulator}, which models each user as a stratified persona comprising observable external traits and hidden internal concerns. The simulator produces faithful and diverse interactions, while emitting per-turn state dynamics that track persuasion progress. We then introduce \textbf{Simulator-Induced Asymmetric-View Policy Optimization}, which converts the modeled concerns and the simulation state transition into complementary training objectives: (1) \emph{Asymmetric On-Policy Self-Distillation} that transfers concern-aware behavior from a privileged view of the same policy into its deployable, conversation-only view; and (2) \emph{State-Transition Policy Refinement} ...

Azure Zhang, Ning Gao, Yuqin Dai +7

1Keeta AI, Meituan · 2Independent Researcher · 3CUHK +1

User Simulation Proactive Persuasion Unsupervised On-Policy Self-Distillation

Ask PDF

Aug 10, 2026cs.CL

Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

User simulators are widely used as scalable environments for training and evaluating interactive assistants. Generating the next user turn is inherently one-to-many: the same profile and dialogue context may support multiple plausible continuations with different local interaction intents. A fluent response may therefore advance the dialogue through an inappropriate intent, such as acceptance rather than repair. Our key insight is that controllable user simulation should separate which local interaction intent the next user turn should realize from how that intent is expressed in language. We introduce UserIDA (User Intent-Directive Alignment), which exposes interaction intent as an explicit per-turn directive. UserIDA defines a six-way intent interface, learns directive-conditioned generation through supervised fine-tuning, and uses intent-calibrated policy optimization during group-based reinforcement learning. The reward preserves composite response quality while ensuring that intent-violating candidates rank below compliant alternatives in mixed groups. On LMSYS-USP, UserIDA achieves 86.6% intent accuracy, outperforming the strongest dedicated user-simulator baseline by 24.3 percentage points while improving semantic and stylistic similarity. In within-context interventions, it realizes at least four of the six target intents in 91.7% of evaluated dialogue states, compared with 22.9% for the strongest external baseline. These results establish per-turn intent control as a complementary dimension to response fidelity in user simulation.

Bo Wang, Ruixing Zhang, Yunqi Liu +4

the State Key Laboratory of Complex and Critical Software Environment, Beihang University

User Simulation Imitation Frictive Policy Optimization

Ask PDF

May 12, 2026cs.AI

Controllable User Simulation

Using offline datasets to evaluate conversational agents often fails to cover rare scenarios or to support testing new policies. This has motivated the use of controllable user simulators for targeted, counterfactual evaluation, typically implemented by prompting or fine-tuning large language models. In this work, we formalize controllable simulation as a causal inference problem. By bridging natural language evaluation with off-policy evaluation methodology, we show that the standard practice of training simulators via supervised fine-tuning on post-hoc trajectory labels yields a structurally biased model. Specifically, these labels are inextricably coupled to the data-generating behavior policy, injecting a look-ahead bias that breaks causal consistency. Furthermore, we prove that under policy shift this failure causes the variance of evaluation metrics to explode geometrically, a phenomenon we term controllability collapse. To restore causal consistency, we establish theoretical conditions for accurate simulation and propose practical training mitigations: a priori controls, step-wise dynamic controls, and direct policy-conditioned learning. Empirical evaluation confirms that while standard global controls distort conversational distributions and collapse behavioral diversity, our causally grounded simulators eliminate look-ahead bias, preserve natural variance, and exhibit robust zero-shot generalization to unseen agent behaviors.

Guy Tennenholtz, Ofer Meshi, Amir Globerson +3

† Google Research · ‡ Tel Aviv University

User Simulation Simulation-Based Reinforcement Learning Goal-Conditioned Value Function

Ask PDF