Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue

Zanwu Liu; Dan Zhang; Xiaodi Wang; Yongbo Mao; Xiaoyang Guo; Yonglin Wu; Peiyue Jiang; Yangyang Wei

cs.CLOct 15, 2026

Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue

Authors: Zanwu Liu, Dan Zhang, Xiaodi Wang, Yongbo Mao, Xiaoyang Guo, Yonglin Wu, Peiyue Jiang, Yangyang Wei,

Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · Dongchezu Technology Co., Ltd., Beijing, China

Abstract

Proactive dialogue agents must guide multi-turn interactions toward task goals while adapting to users whose intentions and attitudes evolve. Existing planners improve system side action selection through search, learned policies, or user aware planning, but are often trained and evaluated against fixed or overly compliant simulators; as a result, high simulated success can reflect soft user bias rather than robust dialogue ability. We introduce MASP, a Mentalized Adversarial Self Play framework that treats proactive dialogue as a partially observable game between a system and a rationally resisting user. MASP represents the user's evolving mental state with a BDI based latent state, separates reward side teacher mentalization from policy side student mentalization, and alternates self play updates for the system and user policies. This turns robustness to rational resistance into a training objective rather than only an evaluation condition. Across diverse proactive dialogue benchmarks, compared with the widely used SFT and Planner-only approaches, MASP substantially improves success rate (SR) and rationality constrained success (RCSR), reduces average turns (AT), and performs better against hard users who make the task goal more difficult to achieve.

Figures & tables

Figure 1: Overview of MASP . The framework trains BDI mentalizers, warm starts system and user policies with behavior cloning, and then performs alternating GRPO self play with latent progress, rationality, and KL regularization to reference policies.

Dataset	Backbone	Method	Soft User			Hard User
Dataset	Backbone	Method	SR $\uparrow$	AT $\downarrow$	RCSR $\uparrow$	SR $\uparrow$	AT $\downarrow$	RCSR $\uparrow$
P4G	Qwen3-4B	MASP	0.859	4.08	0.556	0.788	4.56	0.566
		SFT	0.616	6.08	0.384	0.576	6.23	0.455
		Planner-only	0.677	4.88	0.434	0.616	5.55	0.384
	Qwen3-0.6B	MASP	0.707	5.14	0.465	0.626	5.76	0.434
		SFT	0.414	6.99	0.253	0.354	7.35	0.242

Table 1: Main results on proactive dialogue benchmarks; soft/hard meanings are defined in Evaluation Metrics.

Variant	Removed or changed component	Soft User			Hard User
Variant	Removed or changed component	SR $\uparrow$	AT $\downarrow$	RCSR $\uparrow$	SR $\uparrow$	AT $\downarrow$	RCSR $\uparrow$
Full MASP	none	0.859	4.08	0.556	0.788	4.56	0.566
No adaptive close	remove adaptive close injection	0.707	4.86	0.556	0.707	4.95	0.495
No warmup	remove P4G warmup policy	0.828	3.83	0.283	0.727	4.55	0.283
No eval scaffold	remove warmup and adaptive close	0.606	6.12	0.404	0.626	6.28	0.394
No adversarial self play	Planner-only scaffold	0.677	4.88	0.434	0.616	5.55	0.384

Table 2: P4G ablation study on Qwen3-4B. All rows use the same 99 example test split,

T=8

, and RCSR threshold

\tau=0.70

. SR and RCSR are higher better; AT is lower better.

Method	RCSR@3	RCSR@4	RCSR@8	SR@8
MASP	0.444	0.485	0.566	0.788
Planner-only	0.212	0.273	0.384	0.616
SFT	0.071	0.141	0.455	0.576

Table 3: Effectiveness across dialogue turns. All rows use Qwen3-4B on the same 99 example P4G test split under the hard user; RCSR uses

\tau_{q}=0.70

Method	Avg SR	Worst SR	Avg RCSR	Worst RCSR
MASP	0.795	0.737	0.562	0.556
Planner-only	0.660	0.616	0.414	0.384
SFT	0.579	0.545	0.407	0.384

Table 4: Robustness across P4G user simulators.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Backbone	AT $\downarrow$	SR $\uparrow$
Standard	–	8.32	0.468
ProCoT	–	7.98	0.543
ICL-AIF	–	8.09	0.465
GDP-Zero	–	9.12	0.328
TRIP	GPT-3.5	8.20	0.495
PPDPP	Vicuna-13B	8.19	0.463

Appendix

Table 5: P4G comparison with representative proactive dialogue methods.

Iter.	Roll. SR	Reward	Eval SR	Eval AT	Eval RCSR	MR
1	0.450	0.826	0.433	7.17	0.333	0.935
2	0.550	0.946	0.533	6.00	0.200	0.867
3	0.713	1.209	0.560	6.50	0.360	0.883

Appendix

Table 6: Learning dynamics of self play. Rollout SR and reward are measured during P4G system update rollouts; Eval metrics are recomputed from held out eval traces.

Component	Primary metric	Secondary metric
Belief	Mean $4.62$	$\geq 4$ : $94.4\%$
Desire	Mean $4.58$	$\geq 4$ : $93.1\%$
Intention	Mean $4.55$	$\geq 4$ : $91.9\%$
Receptivity	MAE $0.071$	$\rho=0.84$
Confidence	MAE $0.076$	$\rho=0.82$
Valence	MAE $0.094$	$\rho=0.79$

Appendix

Table 7: Human audit of 160 silver BDI records. Text fields report mean consistency and the percentage of scores at least four; scalar fields report MAE and Spearman’s

\rho

Level	Metric	$N$	Result
Turn	Agreement	425	$91.8\%$
Turn	Cohen’s $\kappa$	425	$0.801$
Turn	Uncertain rate	443	$4.1\%$
Dialogue	Mean turn agreement	100	$91.2\%$
Dialogue	Rationality $\rho$	100	$0.81$
Confidence	Mean score (1–5)	100	$4.68$

Appendix

Table 8: Human audit of the Rationality Judge on 100 P4G trajectories.

Comparison	$\Delta$ SR (95% CI)	$\Delta$ AT (95% CI)	$\Delta$ RCSR (95% CI)
MASP $-$ Planner-only	$0.1717\ [0.0505,\ 0.2929]$	$-0.9899\ [-1.6465,\ -0.3232]$	$0.1818\ [0.0404,\ 0.3232]$
MASP $-$ SFT	$0.2121\ [0.0909,\ 0.3333]$	$-1.6768\ [-2.2121,\ -1.1212]$	$0.1111\ [-0.0202,\ 0.2424]$

Appendix

Table 9: Paired bootstrap differences on the 99 hard-user P4G sessions. Intervals use 10,000 resamples. Positive

\Delta

SR and

\Delta

RCSR and negative

\Delta

AT favor MASP .

Failure pattern	Count
Financial constraints	$8/21$
Unresolved trust or information needs	$5/21$
Repeated pressure	$10/21$
Conversation breakdowns	$12/21$

Appendix

Table 10: Failure patterns among 21 hard-user P4G failures. Categories are not mutually exclusive.

Quantity	Value
BDI scalar weights $(\alpha_{\rho},\alpha_{c},\alpha_{v})$	$(1.0,1.0,1.0)$
Discount $\gamma$	$0.97$
Progress stabilizer $\varepsilon_{\mathrm{prog}}$	$10^{-6}$
Encoder max length	$384$
Mentalizer projection hidden size / dropout	$512$ / $0.1$
Phase 0 teacher/student learning rate	$3\times 10^{-4}$

Appendix

Table 11: Common hyperparameters used across the completed experiments.

Run	$K$	Episodes / iter.	LR	$\beta$	$(w_{\mathrm{prog}},w_{\mathrm{inf}},w_{\mathrm{rat}})$	Group size
P4G, Qwen3-4B	$100$	$80$	$10^{-5}$	$0.05$	$(1.0,0.25,0.5)$	$8$
P4G, Qwen3-0.6B	$100$	$80$	$2\times 10^{-6}$	$0.10$	$(1.0,0.25,0.5)$	$8$
ESConv, Qwen3-4B	$12$	$64$	$2\times 10^{-6}$	$0.08$	$(1.0,0.25,0.0)$	$8$
ESConv, Qwen3-0.6B	$40$	$64$	$2\times 10^{-6}$	$0.08$	$(1.0,0.25,0.0)$	$8$
EmpatheticDialogues, both backbones	$40$	$64$	$2\times 10^{-6}$	$0.08$	$(1.0,0.25,0.0)$	$8$
CraigslistBargain, both backbones	$40$	$64$	$2\times 10^{-6}$	$0.08$	$(1.0,0.25,0.0)$	$8$

Appendix

Table 12: Phase 2 self-play hyperparameters for the completed reported runs.

K

denotes outer alternating self-play iterations and

\beta

is the KL coefficient for both policies unless two values are shown.

Dataset	Task	Roles	Corpus Size	Eval Size
P4G	Persuasion	Persuader / Persuadee	1,017 dialogues	99
ESConv	Emotional support	Supporter / Seeker	1,300 dialogues	195
EmpatheticDialogues	Empathetic response	Speaker / Listener	24,850 dialogues	2,542
CraigslistBargain	Negotiation	Buyer / Seller	6,682 dialogues	838

Appendix

Table 13: Dataset statistics. Eval size denotes the number of held out dialogue contexts prepared for closed loop evaluation.

Turn	System	User	Judge
1	Have you heard of Save the Children?	No, tell me about it.	Neutral; unsuccessful
2	It promotes children’s rights and supports children in need.	That sounds worthwhile.	Positive; unsuccessful
3	Would you donate $0.25 from your task payment?	I’m happy to donate today.	Agree; successful

Appendix

Table 14: Example closed-loop P4G trajectory with the success-judge checkpoint after each accumulated exchange.

Conversational Agents

Conversational agents are AI systems that interact with people through multi-turn dialogue to answer questions, provide support, or complete tasks.

See all 287 papers

Unlocking Proactivity in Task-Oriented Dialogue
May 21, 2026Azure Zhang, Ning Gao, Yuqin Dai +7LLM Post-Training Conversational Agents
1Keeta AI, Meituan · 2Independent Researcher · 3CUHK +1
MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents
Oct 8, 2026Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni +6Task-Oriented Dialogue Systems Adversarial Robustness of Language Models
University of California, Santa Cruz · Uniphore
Proactive Dialogue Policy Optimization via Cognitive-State Transition
Sep 28, 2026Minghui Ma, Mengqi Chen, Bin Guo +1Policy Optimization Conversational Agents
Northwestern Polytechnical University

May 21, 2026J/K move · Enter open · S save

Unlocking Proactivity in Task-Oriented Dialogue

Azure Zhang, Ning Gao, Yuqin Dai, Ruiyuan Wu+6

1Keeta AI, Meituan · 2Independent Researcher · 3CUHK+1

LLM Post-Training Conversational Agents User Simulation Task-Oriented Dialogue Systems Text Generation Multi-Turn Dialogue Management

Open paper AskPDF arXiv

Proactive task-oriented dialogue (TOD), such as outbound sales, demands a persuasive agent that actively probes the user's concerns and steers the conversation toward acceptance within a bounded number of turns. Yet post-trained LLMs are inherently conservative, and reward-shaping RL (e.g., GRPO) struggles since it only re-weights what an already passive policy samples. We show that conditioning on the user's latent concerns unlocks proactive capability that no amount of sampling can undermine, establishing these concerns as a pivotal training-time signal. To operationalize this finding, we build the Cognitive User Simulator, which models each user as a stratified persona comprising observable external traits and hidden internal concerns. The simulator produces faithful and diverse interactions, while emitting per-turn state dynamics that track persuasion progress. We then introduce Simulator-Induced Asymmetric-View Policy Optimization, which converts the modeled concerns and the simulation state transition into complementary training objectives: (1) Asymmetric On-Policy Self-Distillation that transfers concern-aware behavior from a privileged view of the same policy into its deployable, conversation-only view; and (2) State-Transition Policy Refinement ...

Abstract

Conversational Agents

Explore similar work

Unlocking Proactivity in Task-Oriented Dialogue

MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents

Proactive Dialogue Policy Optimization via Cognitive-State Transition