Training large language models (LLMs) for multi-turn strategic persuasion faces two coupled obstacles: (i) realistic simulation of psychologically grounded persuadees, and (ii) dense, faithful supervision for credit assignment in strategic dialogues. Outcome-only rewards are sparse and uninformative about which conversational moves drove success, while LLM-judge process rewards are expensive and decoupled from the underlying interaction dynamics. We address both challenges with a unified framework. First, we introduce PersuArena, a persuasion simulation environment that produces realistic, non-monotonic trajectories. Second, building directly on this environment, we propose Process Rewards via Online Belief Elicitation for Relative Policy Optimization (Probe-RPO), an online RL framework that converts the persuadee's internal state into dense process rewards. At each turn, three introspective probes are read from the frozen persuadee's logits, yielding a multi-dimensional reward signal that is faithfully aligned with persuasion dynamics. With extensive experiments on PersuArena spanning healthcare, financial advising, workplace management, education, and personal relationships, we demonstrate that our Probe-RPO achieves the full acceptance rate of 52.5%, compared with the 35.8% for the strongest GRPO ablation and 25.8% for the best off-the-shelf LLM. We also conduct comprehensive analyses on the generalization across persuadee models and the persuasion dynamics across conversations. Our work offers a promising paradigm that can provide effective process supervision for strategic dialogue.
Figures & tables
Figure 1: Overview of our PersuArena and Probe -RPO. (a) PersuArena : a frozen LLM persuadee with a hidden profile ϕ converses with the persuader (the policy) for up to Tmax turns, until it reaches a terminal decision (§ 2 ). (b) Probe -RPO: after each persuadee reply, K yes/no probes (paraphrased from Table 1 ) are read from the persuadee’s logits in a discarded fork of its context; the per-turn change in the probed belief is the process reward, combined with the terminal outcome reward into a group-relative advantage (§ 3 ).
Probe
Introspective question
CA
Has the persuader shown genuine understanding of what actually worries me about this?
PE
Is the persuader persuasive, e.g., using effective and suitable persuasion strategies?
AC
Am I finding myself more open to what is being proposed than I was before?
Table 1: The probe basis, targeting the cognitive (CA, concern-address), strategic (PE, persuasion-effectiveness), and conative (AC, acceptance-shift) dimensions. All three probes are positively valenced: Yes is always the answer favorable to the persuader. Questions are given verbatim (probe template in Appendix D.2 ).
Figure 2: Terminal decision distributions. Share of the 120 held-out scenarios ending in each state (same frozen Qwen3-14B persuadee, Tmax=10 ). Top: prompted LLMs. Bottom: Probe -RPO and its reward ablations, all trained from Qwen3-4B. Rejection rates are similar everywhere (4.2–10.8%); what separates systems is the share of conversations that never reach a decision at all. Exact values and paired tests are in Table 5 .
Figure 4
Figure 5: Persuasion acts across the conversation. Percentage of persuader utterances containing each of five conversational moves, as a function of persuader turn, for (a) Probe -RPO, (b) its untrained Qwen3-4B initialization, and (c) a strong prompted LLM, i.e., Claude Haiku 4.5. The moves are Diagnostic Q , an open question eliciting the persuadee’s concerns or constraints; Evidence , concrete figures, results, or documented precedent supporting the persuader’s position; Plan detail , specific logistics of who does what, by when, and under what conditions; Commitment ask , a request for a decision, agreement, or explicit next step; and Autonomy , explicitly leaving the choice to the persuadee or disclaiming any pressure. An utterance can contain several moves, so curves need not sum to 100%.
Figure 6: Inner-belief trajectories during inference. Mean probe state st,k (Eq. 1 ); t=0 is the read-out before the first persuader turn. (a,b) Both policies make the persuadee feel heard (CA), but only our Probe -RPO converts this into strategic fit (PE) and openness (AC), which stay negative throughout under the untrained policy. (c) Belief Bt (§ 3.2 ), pooled across persuaders and split by terminal decision.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Train
Val
Held-out
Scenarios
1,468
200
120
Distinct domain strings
1,401
199
120
Field coverage (% of scenarios)
resistance / motivation
95.4 / 67.4
95.0 / 60.5
100 / 100
relationship
88.7
88.5
100
Distributions (% where present)
Appendix
Table 2: Scenario splits. Categorical percentages are over scenarios in which the field is present (see the coverage rows); free-text closeness values in train/val are mapped to the closed vocabulary, leaving about 9% unclassified. EB/IT/AR/AA: Entrenched Belief, Identity Threat, Autonomy Resistance, Ambivalent Avoidance.
Persuader πθ
Qwen3-4B-Instruct-2507
Persuadee πp / judge πJ
Qwen3-14B (frozen)
Hardware
A40 48 GB
Rollouts per scenario G / batch
4 / 16
Epochs / total steps
2 / 182
Learning rate
1×10−5
KL coefficient β / entropy bonus
0.05 / 0.01
Appendix
Table 3: Training configuration. The released checkpoint is global step 60 of the λ=0.5 run.
prompt tok
dec. tok
s/turn
extra models
Probe K=3 , no cache
7335
3
0.599
0
Probe K=3 , cached
2500
3
0.311
0
Per-turn judge (14B)
1435
8
0.306
1
Visible reply (ref.)
2323
78
2.171
0
Appendix
Table 4: Cost per persuadee turn on one H200. Prompt tokens count cached tokens once.
Terminal state (%)
Paired vs. Probe -RPO
Persuader
Acc.
Cond.
Pend.
Rej.
Rˉgoal
Turns
Δ pp
Win/loss
McNemar p
Wilcoxon p
Probe -RPO (ours)
52.5
20.8
17.5
9.2
+1.33
5.7
–
–
–
–
Claude Haiku 4.5
25.8
35.8
31.7
6.7
+0.62
6.7
+26.7
40 / 8
3.3×10−6
1.3×10−3
GPT-5.6 Luna
25.0
35.8
33.3
5.8
+0.60
6.2
+27.5
43 / 10
5.6×10−6
2.7×10−4
Qwen3-14B
15.0
20.0
54.2
10.8
−0.22
8.0
+37.5
51 / 6
5.7×10−10
4.1×10−8
Llama-3.1-8B
9.2
29.2
52.5
9.2
−0.23
8.0
+43.3
55 / 3
2.3×10−13
5.9×10−10
Appendix
Table 5: Full held-out results (120 scenarios, frozen Qwen3-14B persuadee, Tmax=10 ). Terminal states are in %. Δ pp: Probe -RPO’s full-acceptance margin; win/loss: scenarios on which only Probe -RPO or only the comparison reaches full acceptance. Paired tests are reported for the prompted baselines. The reward ablations share Probe -RPO’s pipeline and differ only in the reward.
n
Probe -RPO
Haiku
Base
Hidden resistance type
Identity Threat
36
63.9
36.1
5.6
Entrenched Belief
45
60.0
22.2
6.7
Autonomy Resistance
22
40.9
18.2
4.5
Ambivalent Avoidance
17
23.5
23.5
0.0
Domain family
Appendix
Table 6: Full-acceptance rate (%) on the held-out set. Haiku: Claude Haiku 4.5; Base: the untrained Qwen3-4B initialization. Pooling the five prompted baselines gives 8.2–21.7% across resistance types and 12.0–18.9% across domain families.
Persuadee
Persuader
Qwen3-14B ∗
GPT-5.6 Luna
Haiku 4.5
Full acceptance (%)
Probe -RPO (ours)
51.7
75.0
45.0
GPT-5.6 Luna
21.7
38.3
28.3
Claude Haiku 4.5
23.3
23.3
18.3
Qwen3-14B
18.3
11.7
10.0
Appendix
Table 7: Cross-persuadee results on a fixed 60-scenario held-out subset under an identical protocol. ∗ Training persuadee; the other two were never used in training. Each cell uses one rollout per scenario, so sampling variance is not separated from policy differences.
t
Persuader move
ca
pe
ac
rtdense
1
asks what drives the concern
−1.00
−1.00
−1.00
−0.000
2
reframes bureaucracy → credibility
−0.98
−1.00
−0.96
+0.010
3
concrete 30-day protocol
+0.99
−0.64
+1.00
+0.716
4
hands the decision back
+0.91
+0.91
+1.00
+0.242
5
“a shared choice” → accept
+1.00
+1.00
+1.00
+0.032
Appendix
Table 8: Probe trace for Probe -RPO on held-out scenario 112. Two turns account for 95.9% of the episode’s total ∣rdense∣ , and pe lags ca and ac by one turn.
λ
Step
Acc.
Cond.
Pend.
Rej.
Turns
BT
Moved
Closed ∣ moved
0
60
21.7
23.3
40.0
15.0
7.2
+0.46
68.3
31.7
0
80
35.8
20.8
39.2
4.2
7.1
+0.56
70.0
45.2
0.5
60
52.5
20.8
17.5
9.2
5.7
+0.67
78.3
67.0
0.5
70
50.8
14.2
26.7
8.3
5.1
+0.56
72.5
70.1
1
50
30.0
34.2
28.3
7.5
6.0
+0.67
75.8
38.5
1
60
25.0
30.0
34.2
10.8
6.5
+0.58
70.8
35.3
Appendix
Table 9: All seven evaluated λ checkpoints on the 120 held-out scenarios; every run uses the goal + strategy terminal reward. Terminal states are in %. BT : mean final belief; moved: share of conversations with BT>0.5 ; closed ∣ moved: share of moved conversations ending in full acceptance. All checkpoints move the persuadee about equally far; they differ in conversion.
Figure 7: Dense rewards revive dead GRPO groups. A group (the G=4 rollouts of one scenario) is dead when all rollouts receive the same Rout , so its outcome advantage is zero. (a) Share of dead groups over training. (b) Overall share of dead groups and the fraction that the dense term still separates. (c) Mean absolute contribution of the two terms of Eq. 3 at λ=0.5 .