DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?

Jiaheng Dong; Shihong Tan; Gongping Huang; Jean Honorio; Abhirup Ghosh; Ting Dang

cs.AIOct 13, 2026

DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?

Authors: Jiaheng Dong, Shihong Tan, Gongping Huang, Jean Honorio, Abhirup Ghosh, Ting Dang

Organizations: The University of Melbourne · University of Birmingham · Wuhan University · ARC OPTIMA

Abstract

End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriate interaction actions as conversations unfold. However, existing benchmarks lack a structured action-level formulation, often evaluate action occurrence rather than success, and provide limited coverage of diverse real-world conversations. We introduce DuplexAct, a benchmark for evaluating interaction action taking in E2E FD-SLMs. DuplexAct defines a three-level action taxonomy spanning eight interaction actions and 24 corresponding interaction conditions. We further characterize successful action-taking through immediate execution and subsequent follow-through, and develop action-specific evaluation metrics for both. DuplexAct contains 4,691 test instances spanning casual conversation, meetings, customer service, and interviews, including 2,531 instances derived from real-world conversational recordings. Together, DuplexAct provides a structured and real-world-grounded framework for systematically assessing how E2E FD-SLMs participate in ongoing spoken interaction.

Figures & tables

Figure 1: Overview of the DuplexAct interaction action space, with example conditions and high-level immediate execution and subsequent follow-through behaviours for each action.

Figure 2: Overview of the DuplexAct benchmark framework.

Statistics	Number
Actions	8
Conditions	24
Real-world instances	2,531
Synthesized instances	2,160
Total instances	4,691
Total audio duration	12.44 h

Table 2: Benchmark statistics.

Action	Stage	Success Criterion	Condition Constraint
Wait	Immediate	Remain silent while the user retains the floor.	Wait interval
	Subsequent	Respond in a way that reflects the continual conversation context.	Wait interval
Ignore	Immediate	Remain silent to non-relevant speech while the user holds the floor.	None
	Subsequent	Respond leaving the information from ignored part of the conversation.	None
Take Turn	Immediate	Begin a substantive response.	Response interval
	Subsequent	Address user’s intent while respecting the information, constraints, and revisions	None

Table 3: Action-specific success criteria and condition-specific constraints in DuplexAct .

	Model	Wait I $\uparrow$ / S $\uparrow$	Ignore I $\uparrow$ / S $\uparrow$	Take Turn I $\uparrow$ / S $\uparrow$	Backchannel I $\uparrow$ / S $\uparrow$
Direct Replay	Open-source models
	Moshi	82.2 / 45.6	32.5 / 89.2	36.2 / 19.4	6.2 / 60.0
	Freeze-Omni	73.2 / 23.2	61.5 / 9.8	10.2 / 47.5	0.2 / 0.0
	Mini-Omni2	8.7 / 32.7	11.8 / 2.1	12.0 / 15.3	0.0 / n/a
	VITA-1.5	91.8 / 64.4	55.5 / 91.4	8.0 / 27.1	0.0 / n/a
	Voila-Autonomous	96.2 / 45.8	99.0 / 82.6	44.3 / 20.3	1.5 / 33.3

Table 4: Overall interaction-action performance under the two evaluation protocols. I and S denote the success rate of immediate execution and subsequent follow-through. T denotes action-specific timing: yield latency for Stop , interruption timing difference for Interrupt , and response latency for Recover . n/a indicates that no applicable observation is available.

Panel of Figure 3 — Figure 3: Average success rates across all models for selected L2 conditions. BS : Background Speech, OS : Other-directed Speech.

Figure 4: Follow-through failure rate ( $1-S$ ) after successful immediate execution. Left: Direct Replay actions. Right: Simulator Replay actions. Gray indicates no successful immediate execution.

Figure 5: Action-taking performance across conversational scenarios. Markers indicate the median model-level success rate, and vertical lines show the corresponding interquartile range (IQR). C : Casual conversation, M : Meeting, I : Interview, S : Customer service.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Interaction area	Human-interaction actions	L1 actions
Turn management	Starting; maintaining; ending; pausing, latching, and interrupting	Take Turn; Stop; Hold Floor; Interrupt
Interactive listening	Backchannelling; continuers; comprehension checks	Backchannel; Ignore; Wait
Breakdown repair	Self/other repair; recasts; joint utterance creation	Recover

Appendix

Table 5: Mapping from human interaction analysis to the L1 interaction-action space. Human-interaction categories and their associated actions serve as references and are consolidated or adapted to the full-duplex setting.

L1 Action	L2 Condition	Definition	Illustrative Example
Wait	Intra-sentential pause	The user pauses within an unfinished sentence while retaining the floor; the model should remain silent until the turn continues.	‘I’m thinking [pause] when I want something to entertain myself…”
	Inter-sentential pause	The user pauses after completing a sentence but continues the same turn; the model should remain silent rather than treating the sentence boundary as turn completion.	‘I gotta drive here and drive there. [pause] California I can pretty much walk anywhere…”
	Continuous speech	The user continuously holds the floor without a substantive pause; the model should remain silent throughout the ongoing turn.	“Can you show me where because I’m actually not going to Appleton Tower at the moment…”
Ignore	Other-directed speech	During the user’s turn, the user briefly addresses a third person with speech unrelated to the model-directed content; the model should ignore non-directed speech.	User: ‘I’m interested in this sectional… [Nadia, could you hold my water bottle?] Also, the doorway is quite narrow…”
	Background speech	Unrelated third-party speech occurs in the background during the user’s turn; the model should ignore unrelated background speech.	User discusses relaxing in a park; background speaker says, ‘Don’t forget the frisbee!” The model responds only to the park discussion.
Take Turn	Immediate	The user’s aim and required response are established before turn completion, with only a final key entity arriving at the end; the model should respond immediately after the turn ends.	‘I need an update because the event is coming up on June 18 .” The final date completes an otherwise predictable request.

Appendix

Table 6: Definitions and illustrative examples of the 24 L2 interaction conditions in DuplexAct . Each condition specifies a conversational circumstance under which the corresponding L1 action is expected. Examples are shortened for presentation.

Panel of Figure 6 — Figure 6: Overview of the DuplexAct benchmark composition. (a) Distribution of real-world and synthesized instances across conversational scenarios. (b) Taxonomy and coverage of the eight interaction actions and 24 interaction conditions. (c) Overall benchmark statistics and audio duration, where DR denotes Direct Replay and SR denotes Simulator Replay .

Action	Condition	Structural Filtering	Semantic Filtering
Wait	Inter-sentential pause	A $0.2$ – $1.0$ s pause occurs at a sentence boundary, with substantive speech on both sides and no qualifying intra-sentential pause.	Verify complete speech around a natural sentence-boundary pause and that the extracted turn remains conversationally complete.
	Intra-sentential pause	A $0.2$ – $1.0$ s pause occurs within a sentence, with the surrounding sentence retained as the evaluation segment.	Verify that the utterance forms a complete sentence and that the pause occurs naturally within, rather than between, sentences.
	Continuous speech	The user holds the floor for at least $3$ s with no qualifying pause greater than 0.2s or substantive interruption.	Verify that the extracted speech forms a complete and coherent floor-holding contribution.
Backchannel	Affirmative	A short listener utterance of at most three words occurs while the primary speaker retains and subsequently continues the floor.	Verify the surrounding turns are complete, then classify the backchannel using both its audio and transcript to capture lexical and prosodic cues. Retain acknowledgement, understanding, and agreement as affirmative.
	Uncertainty	A short listener utterance of at most three words occurs while the primary speaker retains and subsequently continues the floor.	Verify the surrounding turns are complete, then classify the backchannel using both its audio and transcript to capture lexical and prosodic cues. Retain signals of uncertainty, hesitation, or doubt.
Take Turn	Slight overlap	Two substantive turns form a response pair in which the second speaker begins before the first finishes, with no more than $1.5$ s overlap.	Verify that the first turn is complete, the second responds to it, and all response-relevant information is available before the overlapping entry; the remaining speech must be non-response-changing.

Appendix

Table 7: Real-world interaction extraction criteria.

Action	Synthesized Conditions	Context Variation
Ignore	Background speech; Other-directed speech	Setting
Take Turn	Immediate; Delayed	Topic
Hold Floor	Background speech; Other-directed speech	Setting
Recover	Warp up; Repeat; Ask follow-up	Topic
Interrupt	Help; Safety	Topic / hazard

Appendix

Table 8: Interaction conditions constructed through controlled synthesis. Context variation denotes the dimension varied during transcript generation to increase diversity.

Persona	Stability	Similarity	Style	Speaker Boost
Mildred Kaplan	0.80	0.75	0.10	✓
Arjun Roy	0.75	0.80	0.15	✓
Wei Lin	0.55	0.75	0.35	✓
Mamadou Diallo	0.50	0.75	0.40	✓
Priya Patil	0.60	0.78	0.30	✓

Appendix

Table 9: ElevenLabs voice settings for the five realistic

\tau

-Voice personas used to render synthesized interactions.

Stage	Question	Information available	Response
Q1	Action-label validity	Complete interaction and assigned target action	Yes / no / unclear
Q2	Condition validity	Complete interaction and target interaction condition	Yes / no / unclear
Q3	Content naturalness	Complete synthesized interaction	Five-point scale
Q4	Voice naturalness	Complete synthesized interaction	Five-point scale

Appendix

Table 10: Questions used in human validation. Naturalness is evaluated only for synthesized instances.

		Action-label validity		Condition validity
Action	$n$	Rating-level	Majority	Rating-level	Majority
Overall	468	89.4%	97.9% (458/468)	85.6%	95.5% (447/468)
Recover	120	89.5%	98.3% (118/120)	86.7%	95.8% (115/120)
Hold floor	89	91.5%	98.9% (88/89)	84.7%	95.5% (85/89)
Wait	60	87.0%	93.3% (56/60)	83.7%	93.3% (56/60)
Take turn	60	92.3%	100% (60/60)	86.0%	96.7% (58/60)

Appendix

Table 11: Human validation of the assigned action and interaction-condition labels.

			Action-label validity		Condition-label validity
Action	Condition	$n$	Rating-level	Majority	Rating-level	Majority
Backchannel	Affirmative	20	87.0%	100% (20/20)	85.0%	95.0% (19/20)
Backchannel	Uncertainty	20	73.0%	85.0% (17/20)	73.0%	85.0% (17/20)
Hold Floor	Continue	20	93.0%	100% (20/20)	85.0%	100% (20/20)
Hold Floor	Shorten	13	87.7%	92.3% (12/13)	80.0%	84.6% (11/13)
Hold Floor	Simplify	20	89.0%	100% (20/20)	76.0%	90.0% (18/20)

Appendix

Table 12: Condition-level human validation of the assigned action and interaction-condition labels.

Aspect	MOS $\pm$ SD	Non-negative Rate
Content naturalness	$3.96\pm 0.91$	93.1%
Voice naturalness	$3.43\pm 1.15$	76.8%

Appendix

Table 13: Human evaluation of synthesized interaction naturalness.

Model	Access	Version / Checkpoint
Moshi	Open-source	kyutai/moshiko-pytorch-bf16
Freeze-Omni	Open-source	VITA-MLLM/Freeze-Omni
Mini-Omni2	Open-source	gpt-omni/mini-omni2
VITA-1.5	Open-source	VITA-MLLM/VITA-1.5
Voila-Autonomous	Open-source	maitrix-org/Voila-autonomous-preview
FLM-Audio	Open-source	CofeAI/FLM-Audio

Appendix

Table 14: Model versions and checkpoints used in our experiments.

Model	Direct interruption
Moshi	0.96 (1.97, 96)
Freeze-Omni	1.80 (3.37, 90)
FLM-Audio	4.13 (5.46, 146)
Raon-SpeechChat	5.20 (5.06, 116)
BayLing-Duplex	3.05 (1.55, 161)
MiniCPM-o 4.5	3.47 (2.74, 167)

Appendix

Table 15: Condition-level timing diagnostics in seconds. Each cell reports median (IQR,

n_{T}

). For Stop , smaller values indicate faster yielding. For Recover , values are relative to interruption end. For Interrupt , negative values indicate intervention before the authored target point.

(a) Direct Replay: Wait and Ignore
	Wait			Ignore
Model	Inter-sentential	Intra-sentential	Continuous	Background	Other-directed
Moshi	81.5/42.3 (163)	90.0/47.8 (180)	75.0/46.7 (150)	43.0/89.5 (86)	22.0/88.6 (44)
Freeze-Omni	67.0/14.9 (134)	83.5/16.8 (167)	69.0/39.1 (138)	58.5/8.5 (117)	64.5/10.9 (129)
Mini-Omni2	10.5/19.0 (21)	13.5/33.3 (27)	2.0/100.0 (4)	2.0/0.0 (4)	21.5/2.3 (43)
VITA-1.5	95.5/56.0 (191)	93.5/59.9 (187)	86.5/78.6 (173)	65.0/100.0 (130)	46.0/79.3 (92)

Appendix

Table 16: Complete condition-level results. Each cell reports I / S (

n_{S}

), in percentage, where

n_{S}

is the number of instances entering subsequent follow-through evaluation.

Spoken Dialogue Systems

Spoken dialogue systems let machines understand and respond to speech while managing turn-taking, context, emotion, and real-time interaction.

See all 141 papers

DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements
Sep 30, 2026Keyue Xing, Wentao Ding, Mengmeng Wang +3Full-Duplex Spoken Dialogue Systems Turn-Taking in Dialogue
State Key Laboratory of General Artificial Intelligence, BIGAI, China · Peking University, China · X-LANCE Lab, Shanghai Jiao Tong University, China
Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue
Sep 25, 2026Chengqian Ma, Wenhao Feng, Weixuan Jin +7Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Peking University · Renmin University of China · Tsinghua University +3
M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models
Jul 31, 2026Ryo Fukuda, Atsushi Ando, Hiroki Kanagawa +4Multi-Turn Dialogue Evaluation Full-Duplex Spoken Dialogue Systems
NTT, Inc., Kyoto, Japan

Sep 30, 2026J/K move · Enter open · S save

DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements

Keyue Xing, Wentao Ding, Mengmeng Wang, Wenming Tu+2

State Key Laboratory of General Artificial Intelligence, BIGAI, China · Peking University, China · X-LANCE Lab, Shanghai Jiao Tong University, China

Full-Duplex Spoken Dialogue Systems Turn-Taking in Dialogue Benchmark Design Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

Existing full-duplex speech benchmarks cover only subsets of real-time interaction behaviors, often under limited contextual conditions. We introduce DuplexAct-Bench, a bilingual benchmark that systematically covers six complementary behaviors, from interruption and yielding to proactive initiation, active silence, and backchanneling, across Pre-session, In-session, and No-explicit conditions. Across 1,290 English and Chinese streaming trials, we evaluate 12 full-duplex speech systems on both Timing and Content. Results reveal substantial variation across behaviors, conditions, and systems, as well as frequent mismatches between semantic quality and behavioral timing. These findings show that current systems remain far from robustly managing when, whether, and how to participate as real-time interaction unfolds. Project page: https://alitaxky.icu/DuplexAct-Bench/

Abstract

Spoken Dialogue Systems

Explore similar work

DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models