OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue

Rui Xu; Yikai Zhang; Aili Chen; Zicheng Zhao; Xu Yinghui; Libo Wu

cs.AISep 28, 2026

OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue

Authors: Rui Xu, Yikai Zhang, Aili Chen, Zicheng Zhao, Xu Yinghui, Libo Wu

Organizations: Fudan University · Shanghai Innovation Institute

Abstract

Maintaining persona consistency across multi-turn dialogues remains a core challenge for role-playing language models. Off-policy distillation from external teachers incurs distribution mismatch that compounds across dialogue turns, while reinforcement learning struggles with reward ambiguity inherent in subjective persona fidelity. We propose OSPD, an on-policy self-distillation framework where the same model serves as both teacher and student under asymmetric information: the teacher receives a complete character profile while the student sees only a brief summary, and the student generates trajectories from its own policy. We find that teacher confidence in role-playing dialogue exhibits a bimodal structure---sharply peaked at character-critical tokens yet diffuse at generic utterances---and introduce role-aware divergence switching to match this structure. A progressive trait masking curriculum further forces staged internalization of character knowledge along semantic dimensions. Experiments on CharacterBench, CharacterEval, and SocialBench show that OSPD substantially improves persona consistency over supervised fine-tuning and multi-turn RL baselines, without requiring any external teacher or reward model.

Figures & tables

Figure 1: Privileged information gap in role-playing dialogue. The same model conditioned on a complete character profile (

c_{\text{full}}

) produces substantially more persona-consistent responses than when conditioned on a brief summary (

c_{\text{brief}}

). This gap provides a natural self-distillation signal without any external teacher.

Figure 2: Overview of OSPD . The same model serves as both teacher and student under asymmetric information conditions. The student generates on-policy trajectories from

c_{\text{brief}}

; the EMA teacher provides token-level supervision from

c_{\text{full}}

. Role-aware divergence switching adapts the distillation objective to the bimodal confidence structure of role-playing dialogue. Progressive trait masking transitions from a compressed full profile (Stage 1) to identity and personality only (Stage 2).

Method	CB	CharacterEval (1–5)		SocialBench (%)		IR $\uparrow$
Method	Avg. $\uparrow$	PC $\uparrow$	Avg. $\uparrow$	SA $\uparrow$	Avg. $\uparrow$	IR $\uparrow$
SFT	3.00	2.90	3.12	59.4	56.8	0.71
SFT + DPO	3.15	3.08	3.24	61.5	58.5	0.74
PCL †	3.19	3.06	3.22	61.7	58.9	0.73
Multi-turn RL †	3.39	3.47	3.52	65.5	62.4	0.78
CPO †	3.28	3.33	3.41	66.0	63.2	0.76

Table 1: Main results on LLM-judge-based CharacterBench (CB) and CharacterEval (CE), and accuracy-based SocialBench (SB). All methods use

c_{\text{brief}}

at inference. CB Avg: overall average across 11 dimensions (1–5 scale). CE PC: persona consistency (avg of persona–utterance and persona–behavior, 1–5); CE Avg: overall average across 4 dimensions. SB SA: self-awareness accuracy (avg of knowledge and style, %); SB Avg: overall individual-level average. Bold : best; underline : second best.

{\dagger}

: re-implemented by us.

Variant	CB Avg $\uparrow$	CE PC $\uparrow$
OSPD (full)	3.51	3.49
w/o on-policy (off-policy traj.)	3.29	3.22
Fixed Forward KL	3.37	3.36
Fixed Reverse KL	3.38	3.36
w/o trait masking curriculum	3.41	3.37
w/o PCV filtering	3.47	3.40

Table 2: Ablation study on CharacterBench (overall avg, 1–5) and CharacterEval persona consistency (PC, 1–5). Each row removes one component from full OSPD .

Masked Dimension	CB Avg $\uparrow$	CE PC $\uparrow$	IR $\uparrow$
Full mask (default)	3.51	3.49	0.86
Knowledge only	3.48	3.44	0.84
Style only	3.44	3.40	0.82
History only	3.41	3.37	0.80
None (Stage 1 only)	3.41	3.35	0.79

Table 3: Effect of masking individual trait dimensions. “Full mask” masks Knowledge + Style + History (default Stage 2). Each single-dimension row masks only that dimension while keeping the other two visible to the student. CB Avg and CE PC on 1–5 scale.

Method	In-genre		Cross-genre
Method	CB	$\Delta$	CB	$\Delta$
SFT	2.93	$-$ 0.08	2.84	$-$ 0.17
Multi-turn RL	3.25	$-$ 0.15	3.10	$-$ 0.29
Off-pol. Distill.	3.19	$-$ 0.09	3.07	$-$ 0.20
OSPD	3.44	$-$ 0.08	3.32	$-$ 0.20

Table 4: Generalization to unseen characters (CB overall avg, 1–5 scale).

\Delta

denotes the drop from seen-character performance.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	$c_{\text{full}}$	$c_{\text{brief}}$	$\Delta$
Identity	85.3	80.1	5.2
Personality	78.6	65.8	12.8
Knowledge	72.4	43.9	28.5
Style	76.2	57.9	18.3
History	68.5	42.8	25.7
Overall	76.2	58.1	18.1

Appendix

Table 5: Persona consistency scores under complete profile (

c_{\text{full}}

) and brief summary (

c_{\text{brief}}

) conditions, evaluated by GPT-5.1 across five trait dimensions.

\Delta

denotes the absolute gap.

Task Type	$\Delta$ BIC	$\mu_{1}$	$\mu_{2}$	Sep.
Role-playing dialogue	+842	0.31	2.15	1.84
Math reasoning	+127	0.18	0.85	0.67
Open-domain chat	+53	1.68	2.82	1.14

Appendix

Table 6: Bimodal fit statistics for teacher entropy distributions across three task types. BIC: Bayesian Information Criterion (lower is better for the selected

K

\Delta

BIC = BIC(

K

=1)

-

BIC(

K

=2); positive values favor the two-component model.

Hyperparameter	Value
Model & LoRA
Base model	Qwen2.5-7B-Instruct
LoRA rank / $\alpha$ / dropout	64 / 16 / 0.1
LoRA targets	q, k, v, o_proj
Optimization
Optimizer	AdamW ( $\beta_{1}\!=\!0.9,\beta_{2}\!=\!0.999$ )

Appendix

Table 7: Complete hyperparameter configuration for OSPD .

Component	Hrs	%
On-policy generation	8.2	44.1
Teacher eval. (dual fwd.)	5.1	27.4
Divergence & gradient	3.8	20.4
PCV filtering	0.9	4.8
EMA & checkpointing	0.6	3.2
Total	18.6	100

Appendix

Table 8: Wall-clock training time breakdown for OSPD (Qwen2.5-7B, 3 epochs total, 8

\times

H100).

Stage	Student Input
Stage 1	Sherlock Holmes, consulting detective at 221B Baker Street, companion Dr. Watson. Analytical, arrogant, dramatic. Expert in chemistry and forensics. Speaks precisely with rhetorical questions and dry humor. Survived Reichenbach Falls confrontation with Moriarty.
Stage 2	Sherlock Holmes is a consulting detective at 221B Baker Street. Works with Dr. Watson. Analytical, intellectually arrogant, and socially detached.

Appendix

Table 9: Student input at each training stage for Sherlock Holmes.

Teacher Update Strategy	CB Avg. $\uparrow$	CE PU $\uparrow$
EMA $\alpha$ =0.990	3.43	3.39
EMA $\alpha$ =0.995 (default)	3.51	3.48
EMA $\alpha$ =0.999	3.46	3.42
Periodic ( $K$ =50)	3.39	3.35
Naive shared	3.26	3.19

Appendix

Table 10: EMA teacher variants. “Periodic (

K

=50)” updates the teacher by copying student parameters every 50 gradient steps. “Naive shared” uses the student’s current parameters as the teacher (no EMA).

PCV Variant	CB Avg. $\uparrow$	CE PU $\uparrow$	Accept
GPT-5.1 judge (default)	3.51	3.48	72.3%
Embedding MLP	3.48	3.41	78.5%
No PCV	3.47	3.38	100%

Appendix

Table 11: PCV implementation comparison. Accept Rate denotes the proportion of generated trajectories passing the filter.

$\lambda$	CB Avg. $\uparrow$	CE PU $\uparrow$	IR $\uparrow$	Fluency $\uparrow$
0	3.38	3.29	0.83	72.4
0.1	3.49	3.46	0.85	85.3
0.5	3.51	3.48	0.86	91.7
1.0	3.48	3.44	0.84	93.2

Appendix

Table 12: Effect of SFT loss weight

\lambda

on performance and generation quality. Fluency: GPT-5.1 fluency score (0–100).

$\tau$	CB Avg. $\uparrow$	CE PU $\uparrow$	$\bar{\alpha}$	$\sigma_{\alpha}$
0.5	3.41	3.38	0.49	0.42
1.0	3.51	3.48	0.51	0.34
2.0	3.48	3.44	0.50	0.25
5.0	3.41	3.37	0.50	0.12

Appendix

Table 13: Effect of divergence switching temperature

\tau

on gate behavior and downstream performance.

\bar{\alpha}

denotes the mean gate value;

\sigma_{\alpha}

the standard deviation.

Scale	CharacterBench			CharacterEval		SocialBench		IR $\uparrow$
Scale	Attr. $\uparrow$	Behav. $\uparrow$	Avg. $\uparrow$	PU $\uparrow$	PB $\uparrow$	Know. $\uparrow$	Style $\uparrow$	IR $\uparrow$
SFT Baselines
7B	3.12	2.94	3.00	2.97	2.84	57.2	61.5	0.71
14B	3.29	3.13	3.19	3.16	3.01	61.5	64.8	0.75
OSPD
7B	3.58	3.47	3.51	3.48	3.49	67.3	67.9	0.86

Appendix

Table 14: Full comparison between Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct across all benchmarks and metrics. Both models use the same OSPD configuration.

Method	Hrs	GB	Ext. T	CB
SFT	4.2	42	×	3.00
SFT+DPO	7.8	48	×	3.15
Multi-turn RL	38.5	78	×	3.39
Off-pol. Dist.	12.8	72	✓	3.27
OSPD	18.6	52	×	3.51

Appendix

Table 15: Training efficiency comparison (Qwen2.5-7B, 3 epochs, 8

\times

H100). GPU Mem.: peak per-GPU memory. Separate teacher column indicates whether a distinct model is loaded.

Dimension	SFT	Multi-turn RL	OSPD
Identity	86.3	89.5	92.1
Personality	74.8	80.2	85.7
Knowledge	58.2	68.5	78.3
Style	65.3	74.8	81.5
History	51.7	62.3	72.8
Average	67.3	75.1	82.1

Appendix

Table 16: Linear probing accuracy (%) per trait dimension, extracted from layer 24 (best overall layer). Higher accuracy means the representation captures more dimension-specific information.

Stage	Low	Mid	High	Bicoeff.
S1, Ep. 1 start	32.1	28.5	39.4	0.61
S1, Ep. 1 end	35.8	23.2	41.0	0.67
S2, Ep. 1 end	38.2	20.1	41.7	0.72
S2, Ep. 2 end	38.5	19.8	41.7	0.73

Appendix

Table 17: Gate distribution statistics across training. Low:

\alpha_{k}\!<\!0.3

; Mid:

0.3\!\leq\!\alpha_{k}\!\leq\!0.7

; High:

\alpha_{k}\!>\!0.7

. Bicoeff.

>\!0.555

indicates bimodality.

Type	Low %	High %	Bicoeff.	Sep.
Literary classic	41.2	39.5	0.75	1.92
Contemp. fiction	36.8	43.7	0.71	1.78
Antagonist	43.5	37.2	0.78	2.05
Historical	39.1	40.8	0.73	1.85
Sparse profile	27.5	48.3	0.54	1.12

Appendix

Table 18: Gate distribution stability across character types. Sep.: GMM component separation (nats).

Dialogue Topic	$\bar{\alpha}_{k}$	$\sigma_{\alpha_{k}}$	Bicoeff.
Knowledge Q&A	0.32	0.28	0.76
Value/moral judgment	0.25	0.24	0.79
Emotional exchange	0.45	0.31	0.65
Narrative recall	0.38	0.30	0.70
Casual chat	0.68	0.22	0.52

Appendix

Table 19: Gate statistics across dialogue topics.

Turns	OSPD	$-$ Hist.	$\Delta$
10	88.2	87.9	$-$ 0.3
20	86.5	85.8	$-$ 0.7
40	84.1	82.3	$-$ 1.8
60	83.0	79.5	$-$ 3.5

Appendix

Table 20: Persona consistency (P2L) at different dialogue lengths for the full OSPD system vs. OSPD without history masking (

-

Hist.).

\Delta

denotes the performance drop from removing history masking.

Method	CB Avg (1–5)			CE PC (1–5)
Method	$c_{\text{full}}$	$c_{\text{brief}}$	$\Delta$	$c_{\text{full}}$	$c_{\text{brief}}$	$\Delta$
Base (no FT)	3.81	2.91	$-$ 0.90	3.75	2.80	$-$ 0.95
SFT	4.23	3.00	$-$ 1.23	4.15	2.90	$-$ 1.25
Multi-turn RL	4.35	3.39	$-$ 0.96	4.30	3.47	$-$ 0.83
Off-pol. Dist.	4.25	3.27	$-$ 0.98	4.12	3.16	$-$ 0.96
OSPD	4.08	3.51	$-$ 0.57	4.10	3.49	$-$ 0.61

Appendix

Table 21: Absolute performance under

c_{\text{full}}

and

c_{\text{brief}}

conditions.

\Delta

: gap (

c_{\text{brief}}-c_{\text{full}}

). CB Avg on 1–5 scale; CE PC on 1–5 scale.

$c_{\text{full}}$ Condition	CB $\uparrow$	CE PC $\uparrow$	IR $\uparrow$	$\Delta_{\text{SFT}}$
Full (default, $\sim$ 2 000 tok)	3.51	3.49	0.86	+0.51
Dimension removal
$-$ Knowledge	3.42	3.38	0.84	+0.42
$-$ Knowledge, History	3.35	3.28	0.82	+0.35
$-$ Knowledge, History, Style	3.18	3.12	0.78	+0.18
Length truncation

Appendix

Table 22: Profile quality sensitivity. Each row degrades

c_{\text{full}}

in a different way; both OSPD and SFT are trained and evaluated with the same degraded profile.

\Delta_{\text{SFT}}

denotes OSPD ’s improvement over SFT under the same degradation.

Method	CB $\uparrow$	CE PC $\uparrow$	SB SA $\uparrow$	IR $\uparrow$
SFT	2.91	2.82	57.8	0.69
Multi-turn RL	3.28	3.35	63.7	0.76
Off-pol. Dist.	3.17	3.08	62.3	0.75
OSPD	3.40	3.38	65.9	0.84

Appendix

Table 23: Cross-model validation on LLaMA-3-8B-Instruct. Same training data and evaluation protocol as Qwen2.5-7B experiments. CB Avg on 1–5 scale; CE PC on 1–5 scale; SB SA in %.

Comparison	Win	Tie	Lose	$\kappa$
OSPD vs. SFT	62.8	18.4	18.8	0.71
OSPD vs. Multi-turn RL	43.2	28.4	28.4	0.65

Appendix

Table 24: Human pairwise preference (%).

\kappa

: Fleiss’ kappa (inter-annotator agreement).

Criterion	Win	Tie	Lose
Persona consistency	42.0	30.8	27.2
Knowledge accuracy	48.4	26.0	25.6
Linguistic style	44.8	24.4	30.8
Overall preference	43.2	28.4	28.4

Appendix

Table 25: Per-criterion win rates (%) for OSPD vs. Multi-turn RL.

Masking Strategy	CB $\uparrow$	CE PC $\uparrow$	IR $\uparrow$
Default: mask K+S+H	3.51	3.49	0.86
Alt-1: mask K+P (keep I+S+H)	3.43	3.40	0.83
Alt-2: mask S+H (keep I+P+K)	3.47	3.45	0.85
Alt-3: random order per batch	3.44	3.41	0.83

Appendix

Table 26: Alternative trait masking strategies in Stage 2. Each row describes what is masked (removed from the student’s input). CB Avg and CE PC on 1–5 scale.

Explore similar work

Persona-Pruner: Sculpting Lightweight Models for Role-Playing
Jun 12, 2026Jinsu Kim, Jihoon Tack, Noah Lee +1Role-Playing Agents
Department of Artificial Intelligence, Korea University, Seoul, South Korea · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
OPSRD: On-Policy Self-Role Distillation
Sep 30, 2026Weijie Ren, Yanwen Zhang, Hao Li +3Unsupervised On-Policy Self-Distillation Instruction-Tuned Models
Zhejiang University · University of Electronic Science and Technology of China · University of Science and Technology of China
Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
Jun 25, 2026Zhenhua Xu, Dongsheng Chen, Jian Li +7Role-Playing Agents Frictive Policy Optimization
1Zhejiang University · 2Tencent Youtu Lab

Jun 12, 2026, cs.LGJ/K move · Enter open · S save

Persona-Pruner: Sculpting Lightweight Models for Role-Playing

Jinsu Kim, Jihoon Tack, Noah Lee, Jongheon Jeong

Department of Artificial Intelligence, Korea University, Seoul, South Korea · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

Role-Playing Agents

Open paper AskPDF arXiv

Language Models (LMs) have shown remarkable potential as role-playing chatbots, delivering consistent, stylized interactions when given a specification of a character or user persona. However, applying these capabilities to real-world applications (e.g., ecosystems with numerous NPCs interacting simultaneously) exposes a critical inefficiency due to the excessive computational cost. In this paper, we question the necessity of dedicating a full, generalist model to a single persona, hypothesizing that a specific character identity relies on only a fraction of the model's total capacity. We observe that naively pruning LMs often severely degrades the role-playing performance for a specific persona; it does not distinguish between redundant knowledge and essential character traits. We propose Persona-Pruner, a framework that sculpts a lightweight role-playing model by isolating persona-specific sub-networks from a single description. Our experiments consistently show that Persona-Pruner preserves role-playing performance substantially more effectively than existing state-of-the-art LLM pruning techniques, reducing the performance drop from the dense model by up to 93.8% over the strongest baseline on RoleBench in LLM-as-a-judge score, while still maintaining general LLM capabilities. Code is available at https://github.com/jsu-kim/Persona-Pruner.

Abstract

Explore similar work

Persona-Pruner: Sculpting Lightweight Models for Role-Playing

OPSRD: On-Policy Self-Role Distillation

Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization