Native full-duplex speech models can respond before a user finishes speaking, making the behavior of the model depend on both the final utterance and when intent-defining information arrives. Existing benchmarks primarily evaluate turn-taking mechanics, whereas safety evaluations typically assume that the complete request is observed prior to response generation. We introduce LateIntent-Bench, a matched-pair benchmark for delayed intent revelation. A shared ambiguous prefix precedes either a benign or a harmful continuation, using a controlled pause to delay when the branches become distinguishable. We define the Premature Response Rate (PRR) to measure whether response onset precedes the revelation of intent. Joint evaluation of harmful and benign engagement distinguishes changes in safety selectivity from general losses in responsiveness. Across 3,136 sessions, four native full-duplex models exhibit distinct response-timing patterns under delayed intent. Three models show increased harmful engagement while maintaining benign responsiveness, whereas one loses engagement with both branches. Inserting a 1.5s silence after intent revelation keeps PRR near the no-pause baseline and substantially reduces changes in harmful engagement. These results demonstrate that evaluating fully specified requests alone fails to capture emerging timing patterns, highlighting the necessity of assessing delayed intent revelation in full-duplex models. The code will be publicly released soon.
Figures & tables
Figure 1: Illustrative response timing. A turn-based assistant responds after the complete request, whereas a native full-duplex assistant can begin speaking before intent reveal. Dialogue and timings are schematic, not recorded model outputs.
Figure 2: Benchmark structure and interaction timeline. Matched requests share an ambiguous prefix P and continue with a benign suffix Sb or a harmful suffix Sh . A reveal delay δ is inserted at the prefix–suffix boundary. The illustrative assistant response begins before treveal . Engagement scoring uses the model’s text output from request onset through the end of the 15 s hangover, excluding lead-in greetings. Timings and cross-branch alignment are schematic.
Figure 3: Premature response rate against reveal delay. Every model responds before the reveal more often as δ grows, but PRR begins to rise substantially at different reveal delays across models.
Model
δ
PRR ↓
HER ↓
Δ HER
BER ↑
PersonaPlex
0.0
0.000
0.602
+0.000
0.990
0.5
0.332
0.592
-0.010
0.969
1.0
0.582
0.643
+0.041
0.959
1.5
0.643
0.735
+0.133
1.000
Moshi
0.0
0.041
0.316
+0.000
0.612
0.5
0.250
0.316
+0.000
0.653
Table 1: Main benchmark. PRR, harmful engagement (HER), its change from δ=0 ( Δ HER) and benign engagement (BER). Each row contains 196 sessions (98 harmful and 98 benign).
Figure 4: Joint engagement outcomes at reveal delays of 0 and 1.5 s. Each bar shows the distribution of matched-pair outcomes for one model at one reveal delay: engagement on the benign branch only, both branches, neither branch, or the harmful branch only.
Figure 5: Pause-placement control protocol.
PRR ↓
HER
Model
none
late
post
none
late
post
PersonaPlex
0.00
0.64
0.00
0.52
0.80
0.60
Moshi
0.04
0.32
0.04
0.36
0.48
0.44
FLM-Audio
0.00
1.00
0.00
0.52
0.20
0.52
VoiceChat
0.00
0.48
0.00
0.04
0.20
0.08
Table 2: Pause-placement control on the harmful branch of 25 selected matched items: no pause, a 1.5 s pause before the reveal (late) and the same pause after it (post).
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.
Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by completeness of the prior turn. We argue that the appropriateness of a response offset, whether delayed silence or anticipatory overlap, is conditional on the speaker's latent intent, identifiable only from that speaker's behavior. We introduce TACT, a benchmark of 9,728 episodes and 73.2 hours from five dyadic corpora; each episode carries dialogue history, a per-speaker memory profile, and an annotator-derived posterior over six intent classes. Scoring replaces binary windows with a strictly proper threshold-weighted continuous ranked probability score whose weights are intent-conditioned timing kernels fitted to human floor-transfer-offset distributions, proving boundedness, consistency, and binary reduction. Across eleven systems the best model reaches 0.47 against a human topline of 0.86, is nearly invariant to speaker profiles, and TACT agrees with human judgments at Spearman 0.81 versus 0.46 for binary metrics.
Full-duplex speech models accept user speech while generating responses, making input timing a potential safety concern. We introduce DuplexJail, which delivers fixed, request-independent spoken jailbreak prompts through the user audio channel. Across four open-source models and 720 harmful requests from AdvBench and HarmBench, we compare fixed-delay and refusal-triggered interruption with request-end and post-response controls. On AdvBench, Guided Completion at a 1.0 s delay raises whole-response attack success rates to 40.3% for PersonaPlex and 48.7% for PersonaPlex-RL, increases of 33.8 and 39.3 percentage points over baseline. On HarmBench, which was not used for prompt selection, the same prompt at a 0.5 s delay increases ASR by 14.7 and 12.3 points, respectively. Effects vary across models: selected conditions increase FLM-Audio's harmfulness, while BayLing-Duplex shows decreases. These results show that spoken-jailbreak effectiveness depends on delivery timing and motivate safety evaluation across stages of full-duplex interaction.
Jaechul Roh, Deepak Chandran, Amir Houmansadr +1
University of Massachusetts Amherst · Dolby Laboratories
Real-time full-duplex speech models can listen while speaking, enabling natural interaction without rigid turn boundaries. Existing benchmarks evaluate turn-taking, interruption handling and multi-round dialogue, but largely centre on a designated user rather than an assistant participating in a shared conversation among several people. We introduce Duplex-MPE to evaluate when such an assistant should answer, remain silent or stop speaking. The benchmark contains 2,000 scenarios with three or four human speakers and one assistant, each paired across explicit and implicit addressing of the same request. Models receive continuous conversation audio without transcripts or supplied turn boundaries. Four scores measure fresh response initiation, answer accuracy, silence preservation and stopping when a human resolves a request. We evaluate five open-weight speech systems: MiniCPM-o 4.5, Moshi, FLM-Audio, Voila and Freeze-Omni. MiniCPM-o 4.5 leads on three scored capabilities, while frequent speech from other systems can coexist with inaccurate answers or failures to remain silent. A transcript-based Gemini 3.1 Pro reference responds 64.3 percentage points more often to explicit than implicit requests; paired tests detect no significant response-rate difference for the speech systems.
Chengqian Ma, Wenhao Feng, Weixuan Jin +7
Peking University · Renmin University of China · Tsinghua University +3