ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing

Xi Chen; Wei Xue; Yike Guo

cs.SDApr 13, 2026

ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing

Authors: Xi Chen, Wei Xue, Yike Guo

Organizations: The Hong Kong University of Science and Technology

Abstract

Role-playing has garnered rising attention as it provides a strong foundation for human-machine interaction and facilitates sociological research. However, current work is confined to textual modalities, neglecting speech, which plays a predominant role in daily life, thus limiting genuine role-playing. To bridge this gap, we conceptualize and benchmark speech role-playing through ActorMindBench, and we present a corresponding reasoning framework, called ActorMind. Specifically, (1) Speech Role-Playing enables models to deliver spontaneous responses with personalized verbal traits based on their role, the scene, and spoken dialogue. (2) ActorMindBench is a hierarchical benchmark comprises Utterance-Level content with 7,653 utterances, Scene-Level content with 313 scenes, and Role-Level content with 6 roles. (3) ActorMind is an off-the-shelf, multi-agent, chain-of-though style reasoning framework that emulates how human actors perform in theaters. Concretely, ActorMind first reads its assigned role description via Eye Agent, then comprehends emotional cues within contextual spoken dialogues through Ear Agent. Subsequently, Brain Agent generates a descriptive emotional state, and finally, Mouth Agent delivers the scripts infused with corresponding emotion state. Experimental results demonstrate the effectiveness of ActorMind in enhancing speech role-playing.

Figures & tables

Figure 1: Overview of ActorMindBench. ActorMindBench comprises three content levels: Utterance-Level includes speech segments with text content and speaker labels; Scene-Level includes scene boundaries and descriptions; Role-Level includes role profiles.

Figure 2: Overview of ActorMind. ActorMind operates in a multi-agent chain-of-thought reasoning style. Specifically: (1) The Eye Agent grasps the scene descriptions and role profiles. (2) The Ear Agent listens to the tones expressed by others. (3) The Brain Agent brainstorms the emotional state for the next line of dialogue based on what has been seen and heard. (4) Powered by RAG, the Mouth Agent retrieves the most similar speech from its won database with a comparable emotional description, mimics it, and spontaneously delivers it.

	Phoebe	Joey	Chandler	Rachel	Ross	Monica	Average
YourTTS ( Casanova et al., 2022 )	2.90 ± 0.89	2.47 ± 0.90	2.30 ± 1.40	1.80 ± 0.84	2.60 ± 1.19	2.30 ± 0.91	2.39 ± 0.93
F5-TTS ( Chen et al., 2024b )	2.60 ± 1.08	2.33 ± 0.75	3.60 ± 0.65	3.00 ± 1.58	2.90 ± 0.74	2.80 ± 0.45	2.87 ± 0.77
Cosyvoice ( Du et al., 2024 )	2.30 ± 0.76	2.67 ± 0.71	2.10 ± 0.22	1.40 ± 0.55	2.00 ± 0.94	1.80 ± 0.67	2.04 ± 0.45
SparkTTS ( Wang et al., 2025 )	3.40 ± 1.08	2.53 ± 0.80	2.90 ± 0.42	2.20 ± 1.30	3.20 ± 0.91	2.00 ± 0.94	2.71 ± 0.78
IndexTTS ( Deng et al., 2025 )	3.80 ± 0.67	2.20 ± 0.65	3.30 ± 0.27	3.20 ± 0.84	2.60 ± 1.08	3.20 ± 0.76	3.05 ± 0.56
Qwen_Omni ( Xu et al., 2025 )	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00

Table 1: Main Results. Subjective evaluation using RP-MOS for ActorMind and baseline models.

	Phoebe	Joey	Chandler	Rachel	Ross	Monica	ALL
ActorMind + F5-TTS	1.00 ± 0.00	0.75 ± 0.29	0.75 ± 0.50	0.50 ± 0.58	0.88 ± 0.25	0.75 ± 0.29	0.77 ± 0.18
ActorMind + Cosyvoice	0.88 ± 0.25	0.63 ± 0.25	0.75 ± 0.29	0.50 ± 0.58	0.38 ± 0.48	0.63 ± 0.48	0.63 ± 0.16
ActorMind + SparkTTS	0.50 ± 0.00	0.88 ± 0.25	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	0.90 ± 0.04
ActorMind + IndexTTS	0.88 ± 0.25	0.75 ± 0.5	0.25 ± 0.50	0.75 ± 0.50	0.88 ± 0.25	1.00 ± 0.00	0.75 ± 0.17
ActorMind + YourTTS	0.63 ± 0.48	0.50 ± 0.41	0.88 ± 0.25	0.50 ± 0.58	1.00 ± 0.00	0.50 ± 0.58	0.67 ± 0.17

Table 2: Performance improvement over baseline models after applying ActorMind.

		RP-MOS $\uparrow$
(1)	w/o Role Profile (w/o Eye)	-0.37 ± 0.21
(2)	w/o Scene (w/o Eye)	-0.30 ± 0.17
(3)	w/o Context (w/o Eye, w/o Ear)	-0.22 ± 0.14
(4)	w/o Ear	-0.32 ± 0.23
(5)	w/o Brain (w/o All)	-0.51 ± 0.56

Table 3: Ablation study. Relative performance with respect to ActorMind across six roles.

Panel of Figure 3 — Figure 3: Spectrogram Comparison of baselines and ActorMind. All samples are generated for Phoebe performing "…So, um, do you think he’s doing any better than he was this morning?" under the same scene and context.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Prompt for scene captioning: black text indicates the prompt, while brownish-yellow highlights the utterances within a scene.

Figure 6: Prompt for role profile generation: black text indicates the prompt, while brownish-yellow highlights the role content.

Episode	Rachel		Monica		Phoebe		Joey		Chandler		Ross		OTHERS		TOTAL
	num	duration	num	duration	num	duration	num	duration	num	duration	num	duration	num	duration	num	duration
SE01_01	86	0:03:15	82	0:03:08	22	0:00:54	39	0:01:42	33	0:01:23	63	0:02:45	25	0:01:07	350	0:14:13
SE01_02	56	0:02:36	37	0:01:15	21	0:00:44	11	0:00:23	29	0:01:10	101	0:04:16	97	0:03:44	352	0:14:09
SE01_03	32	0:01:12	76	0:02:36	65	0:02:20	28	0:01:04	72	0:02:37	53	0:01:54	39	0:01:31	365	0:13:14
SE01_04	81	0:03:16	65	0:02:27	47	0:01:52	36	0:01:15	49	0:01:55	57	0:02:36	36	0:01:23	371	0:14:44
SE01_05	48	0:02:01	40	0:01:44	29	0:00:57	58	0:02:23	50	0:01:51	73	0:03:30	48	0:02:05	346	0:14:31

Appendix

Table 4: Utterance-level statistics on number of utterances and speech duration by role and episode.

Episode	Scene Num	Avg Utterances per Scene	Avg Roles per Scene
SE01_01	14	17.29	3.93
SE01_02	8	29.75	5.38
SE01_03	13	20.0	4.85
SE01_04	16	15.75	4.19
SE01_05	16	14.94	3.31
SE01_06	9	24.33	4.78

Appendix

Table 5: Scene-level statistics.

Figure 7: Qwen_Omni Prompt for Speech Role-Playing. Black text: prompt template; Brownish-yellow: corresponding speech and text content.

Figure 8: Prompt for the Brain Agent used for role injection, contextual understanding, and emotion rendering. Here, yellow content represents what the Eye Agent saw, blue content represents what the Ear Agent heard, and purple content represents what the Brain Agent inferred.

Large Language Model-Based Role-Play Simulation

Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.

See all 106 papers

RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue
Sep 15, 2026Yuqi Wang, Fengyuan Liu, Haochen Luo +2Large Language Model-Based Role-Play Simulation Speech Generation Evaluation
The University of Hong Kong · Kami AI
DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention
Jun 16, 2026Wenqiu Tang, Zhen Wan, Takahiro Komamizu +1Large Language Model-Based Role-Play Simulation Spoken Dialogue Systems
Nagoya University, Nagoya, Aichi, Japan · National Institute of Informatics, Tokyo, Japan
Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning
Apr 15, 2026Dongjie Fu, Fangming Feng, Xize Cheng +3Audio-Language Model Evaluation Large Language Model-Based Role-Play Simulation
Meituan, Shanghai, China

Sep 15, 2026J/K move · Enter open · S save

RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue

Yuqi Wang, Fengyuan Liu, Haochen Luo, Zhiqi Yu+1

The University of Hong Kong · Kami AI

Large Language Model-Based Role-Play Simulation Speech Generation Evaluation Spoken Dialogue Systems Multi-Turn Dialogue Evaluation Language Model Robustness Emotion-Aware Dialogue Generation

Open paper AskPDF arXiv

Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric and short-horizon. This leaves open whether spoken dialogue models can sustain diverse roles over extended interactions, especially beyond predefined fictional characters. We introduce RoleBreak, an open benchmark for long-horizon role-playing robustness in spoken dialogue. RoleBreak contains 310 character-based and user-centered roles, 6,688 human-verified dialogue turns, and 11,743 fine-grained evaluation criteria, with 1,856 turns carrying expressive emotion targets for evaluating vocal emotion. Its scenarios are designed to stress role consistency, interaction quality, safety, and affect over extended conversations. We evaluate nine configurations spanning full-duplex, omni-modal, and cascaded ASR--LLM--TTS paradigms. We find four key patterns. First, current systems are substantially stronger at semantic role adherence than at vocal emotion. Second, semantic robustness remains brittle over long interactions: even the strongest evaluated system encounters its first persona and safety failures after only 10.4 and 11.6 turns on average. Third, scaling the LLM substantially improves semantic robustness and delays failure, but yields little improvement in vocal emotion. Finally, user vocal emotion affects role-playing behavior even when linguistic content is fixed. These findings highlight persistent gaps in both long-horizon robustness and vocal expressiveness in spoken role-playing systems.

Abstract

Large Language Model-Based Role-Play Simulation

Explore similar work

RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue

DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention

Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning