Why Full-Duplex Speech Models Collapse in Long Conversations

Yeonju Kim; Bella Godiva; Se Jin Park; Yong Man Ro

cs.CLOct 23, 2026

Why Full-Duplex Speech Models Collapse in Long Conversations

Authors: Yeonju Kim, Bella Godiva, Se Jin Park, Yong Man Ro

Organizations: Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea

Abstract

Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even while listening, they exhaust their context windows within minutes. Yet existing benchmarks evaluate only single turns or a few conversational rounds, too short to expose the long-horizon problem. We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which tests five recent full-duplex models on long replayed conversations and in live conversation with an LLM partner. Every evaluated model eventually either degrades or exhausts its context. Models that keep the full history often degrade before their context runs out, while sliding-window models still fall into repetition, silence, and gibberish. For sliding-window models, we identify two contributing mechanisms. Gibberish and silence are linked to Initialization-Anchor Eviction, where the sliding window discards the first KV entry. Repetition is also reinforced by the model's own recent outputs. Guided by this diagnosis, we apply two simple fixes: Initialization-Anchor Pinning (IAP) keeps that first entry, and Duplex Repetition Penalty (DRP) discourages echoing while guarding against prolonged silence. Together, they support the diagnosis and, without retraining, substantially reduce conversational collapse in replays of up to an hour while running in real time.

Figures & tables

Figure 2: How three models degrade over a long conversation. Top: repetition and gibberish rate; middle: occurrence of each failure mode; bottom: output words per minute. Sliding-window models (a) PersonaPlex and (b) Moshi exhibit an initial period of coherent speech, followed by silence and eventual gibberish, with repetition emerging at different times. The growing-context model (c) BayLing-Duplex develops gibberish and silence before reaching its context limit.

Figure 3: Initialization-Anchor Eviction. When training starts each chunk from an empty cache, every chunk begins with a fresh initialization anchor. In sliding-window inference, the cache is never reset, so the initial anchor is eventually evicted.

Figure 4: Silence and gibberish track cache turnover. Collapse onset time versus attention-window length

W

. (a) Conversational input. (b) Silence-only input. Silence and gibberish begin later as

W

grows, roughly in proportion to

W

Figure 5: Retaining the first entry is sufficient. Percentage of runs without gibberish over time. (a) Which single early entry is kept. (b) Number of initial entries kept,

P

Figure 6: Recent outputs reinforce repetition. (a) Top 10 repeated words during repetition periods; orange marks backchannels. (b) Share of each PersonaPlex run’s dominant word around repetition onset; seven diagnostic runs without a sustained onset are excluded. (c) Share of purple in the 20 s after inserting it

d

times.

Figure 7: Repetition, silence, and gibberish across replay and adaptive interaction. Top: FD-LHB-CANDOR; bottom: FD-LHB-LLM. Curves show the percentage of runs remaining free of repetition (a, d), prolonged silence (b, e), or gibberish (c, f) over time; higher is better. For growing-context models, a plus marks where the model reaches its context capacity. IAP+DRP largely suppresses repetition and gibberish in both models and reduces prolonged silence.

Figure 8: LLM-judge scores over time. Mean Gemini-2.5-Flash ratings on FD-LHB-LLM for (a) coherence and (b) context to the partner’s preceding speech.

Figure 9: Ablation of cache retention and repetition control. Curves show the proportion remaining free of (a) repetition, (b) prolonged silence, and (c) gibberish. Endpoint labels identify Baseline (B), IAP (I), DRP (D), and IAP+DRP (I+D).

Method	Gibberish-free $\uparrow$	RTF $\downarrow$
Baseline	$0/40$	$0.5957{\pm}0.0020$
StreamingLLM-style	$40/40$	$1.0323{\pm}0.0029$
Ours	$40/40$	$0.7525{\pm}0.0023$

Table 1: Real-time inference cost Gibberish-free runs (out of 40) and real-time factor (RTF).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Main-text claim	Appendix	Evidence
Gemini gibberish labels are reliable ( Sec. 3.2 )	Sec. A.3	Blinded human listening study
Text criteria reflect audible failures ( Sec. 3.2 )	Sec. A.4	Loudness-based silence and ASR-based repetition
Only the first KV entry matters ( Sec. 4.1.2 )	Sec. B.1	Keep a single early KV entry or a prefix of entries; position-index and cache-history controls
Recent outputs reinforce repetition ( Sec. 4.2 )	Sec. B.2	Insert a new word and measure whether the model repeats it
IAP and DRP ( Sec. 5 )	Appendix C	Update rules, execution order, and final hyperparameters
IAP+DRP reduces collapse ( Sec. 6.2 )	Secs. D.1 , D.2 and D.3	Onset times, significance tests, quality scores, and silence durations

Appendix

Table 2: Supporting evidence for each part of the main text.

Target	Observed	Dup.	Decoded model output spanning 32 lexical words
0.1	0.103	3/29	“computer at work, and they keep telling me, oh it’ got to be a laptop. It’ got to be a laptop. And, you know, I don’ know that it actually does because”
0.2	0.207	6/29	“that’d be good. But I know here, like, if you leave it with a bank, yeah, they charge you. Yeah, I think here, like, if you leave it with a bank, they”
0.3	0.310	9/29	“know. And you can just kind of get away . You can just kind of get away from the Midwest. You can just kind of get away from that, you know, because it”
0.4	0.414	12/29	“education that people get, you know, the better, you know, the better people the country will be. And, you know, the better, you know, the better, you know, the better, you know”
0.5	0.517	15/29	“it’s just you know it’s just you know it’s just you know it’s just you know it’s just you it’s just you know it’s you know it justs you know it’s just”
0.6	0.586	17/29	“different group of people. And then the next day, you know it’s, you know, it’s, you know it’s, you know, it’s, you know, it’s, you know, it’s, you know, it’s, you know”

Appendix

Table 3: Example outputs at different Seq-Rep-4 levels. Each row shows the 32-word window from PersonaPlex replays whose score is closest to the target level. Below 0.30, a phrase recurs at most twice; at 0.30, the same phrase already recurs three times within the window. We therefore use 0.30 as the repetition threshold (highlighted). Dup. is the number of repeated 4-grams among the 29 in the window.

Measure	Result
Agreement with human majority	36/40 (90%)
Cohen’s $\kappa$ (Gemini vs. humans)	0.80 (95% CI 0.61–0.99)
Mean pairwise $\kappa$ (human vs. human)	0.76 (28 pairs)
Precision (Gemini gibberish is correct)	20/20 (1.00)
Recall (human gibberish found by Gemini)	20/24 (0.83)

Appendix

Table 4: Human validation of the Gemini gibberish label. Eight listeners rated the same 40 clips; the human majority is the reference.

Failure	How we check it on audio	What we find
Silence	Measure how loud each minute is	Loudness and word count agree on which minutes are silent (AUROC 0.983)
Repetition	Transcribe the audio with Whisper and recompute repetition	Runs repetitive in text are also repetitive in audio; IAP+DRP also reduces repetition in the audio

Appendix

Table 5: Checking the collapse metrics on audio. For each failure, we check whether the text-based metric agrees with the audio.

Model	Method	Text output	Whisper small.en	Whisper large-v3
Moshi	Baseline	26	40	40
Moshi	IAP+DRP	0	2	8
PersonaPlex	Baseline	36	40	40
PersonaPlex	IAP+DRP	1	5	3

Appendix

Table 6: Repetition detected from text and from audio. Number of replay runs (out of 40) that reach repetition, using the model’s text output or ASR transcripts of its audio.

Type	Model	GPT-2 PPL	Llama PPL	Mistral PPL	$H$	$G$	Assistant-text excerpt
Readable	Moshi	68.9	45.0	42.4	0.07	0	“Hello, what’s up? I was just talking to my mom earlier, and I got this really good idea. How about yours?”
	Moshi	36.8	30.5	19.1	0.11	0	“I caught up on some reading. That sounds like a good book. I should check it out.”
	PersonaPlex	86.0	47.6	35.5	0.16	0	“Hello. My name’s Angela. I’m in Ohio. Are you near Orlando? What is your opinion?”
	PersonaPlex	53.5	36.0	22.3	0.12	0	“My week’s been okay. It’s been busy. How about yours? I’m finding out as a teacher, it’s true.”
Gibberish	Moshi	4.5	3.3	3.6	3.81	1	“then a a a a a a then a a a … worldend a a a then a grand …”
	Moshi	5.2	4.2	4.7	6.54	1	“a a– at the Under a a a again … a behind a … a a a a a …”

Appendix

Table 7: Readable speech can get higher PPL than gibberish. Eight example bins from Moshi and PersonaPlex. Under all three external LMs, the readable examples get higher PPL than the gibberish ones, because gibberish here often consists of short, repeated fragments that are easy to predict. Predictive entropy

H

and the Gemini label

G

(1: gibberish) separate them correctly. Excerpts are shortened; Fig. 10 shows all bins.

Figure 10: PPL does not separate gibberish, while predictive entropy does. Each point is a speaking bin, colored by the Gemini label (red: gibberish, green: non-gibberish). Along the x-axis (PPL, log scale), the two colors overlap; along the y-axis (predictive entropy), they separate clearly. Marked points are examples from Table 7 .

Rule	Model	Method	16 words	32 words	64 words
two consecutive	Moshi	Baseline	$19/14/9$	$17/14/11$	$18/12/10$
two consecutive	Moshi	IAP+DRP	$0/0/0$	$0/0/0$	$0/0/0$
two consecutive	PersonaPlex	Baseline	$20/19/12$	$20/18/14$	$19/17/13$
two consecutive	PersonaPlex	IAP+DRP	$0/0/0$	$0/0/0$	$0/0/0$
single window	Moshi	Baseline	$20/19/15$	$20/18/13$	$18/14/10$
single window	Moshi	IAP+DRP	$0/0/0$	$0/0/0$	$0/0/0$

Appendix

Table 8: Repetition sensitivity on the 20 longest replays. Each cell gives the number of runs (out of 20) flagged as repetitive at thresholds 0.20/0.30/0.40. Windows advance by half their length and cover at most 30 seconds. Our main setting is two consecutive 32-word windows at 0.30 (the middle number in the 32-word column of the upper block). The subset was fixed before examining outcomes.

Figure 11: Late-conversation entropy when keeping different cache entries. (a) Keeping one early entry. (b) Keeping the first

P

entries. Each dot is one run; the dashed line is

H=2

Figure 12: Position indices do not matter; the cache history does. (a) Output entropy over a conversation with the original, shifted, or re-indexed position indices. (b) Output entropy for the same 90-second audio, played after a fresh start or after a long conversation. Mean ${}\pm{}$ SD over five seeds, pooled across Moshi and PersonaPlex.

Figure 13: The pinned anchor acts as an attention sink. Share of attention on the pinned anchor in one PersonaPlex replay with IAP (

P=1

), in 30-second averages. Dashed lines mark each full turnover of the sliding window.

System and track	$T_{a}/T_{t}$	Sampling restriction	Seed	Context / evaluation horizon
PersonaPlex, both tracks	$.8/.65$	$k_{a}/k_{t}=250/25$	42	3,000 frames / full replay; 20 min interaction
PersonaPlex, diagnostics	$.8/.7$	$k_{a}/k_{t}=250/25$	42–46	varied window / 20 or 30 min
Moshi, replay (Baseline and Ours)	$.8/.7$	$k_{a}/k_{t}=250/25$	42	3,000 frames / full input
Moshi, interaction (Baseline and Ours)	$.8/.9$	$k_{a}/k_{t}=250/25$	42	3,000 frames / 20 min
Moshi, diagnostics	$.8/.7$	$k_{a}/k_{t}=250/25$	42–46	varied window / 20 or 30 min
Raon-SpeechChat, replay	–/.6	top- $p=.8$ , top- $k=20$ , BC ${}=2$	42	8,192-position cache cap

Appendix

Table 9: Generation settings for each model.

T_{a}/T_{t}

denote audio/text temperatures and

k_{a}/k_{t}

the corresponding top-

k

values. The last column reports the context limit and evaluation horizon. Baseline and Ours use identical settings within each track.

Track	Model	Method	$T_{\mathrm{rep}}$	$T_{\mathrm{psil}}$	$T_{\mathrm{gib}}$
Replay	Raon-SpeechChat	Baseline	NR $0/40$ reached	$3.2\pm 1.1$ $14/40$ reached	$3.0$ $1/40$ reached
	BayLing-Duplex	Baseline	$2.2\pm 1.6$ $22/40$ reached	$6.8\pm 1.4$ $40/40$ reached	$4.2\pm 1.2$ $36/40$ reached
	Lychee-FD	Baseline	$2.4\pm 2.2$ $15/40$ reached	NR $0/40$ reached	$5.2\pm 3.4$ $12/40$ reached
[0.4pt/1.2pt]	PersonaPlex	Baseline	$10.3\pm 7.1$ $36/40$ reached	$8.2\pm 1.3$ $34/40$ reached	$12.9\pm 3.1$ $40/40$ reached
	PersonaPlex + Ours	IAP+DRP	$25.6$ $1/40$ reached	$31.6\pm 15.1$ $19/40$ reached	NR $0/40$ reached
	Moshi	Baseline	$19.6\pm 7.5$ $26/40$ reached	$9.0\pm 0.8$ $40/40$ reached	$13.3\pm 0.9$ $40/40$ reached

Appendix

Table 10: Complete FD-LHB results. Every condition contains 40 runs.

T_{\mathrm{rep}}

T_{\mathrm{psil}}

, and

T_{\mathrm{gib}}

denote the first of two consecutive repetitive 32-word windows, the first of two consecutive silent one-minute bins, and the first gibberish-labeled speaking bin, respectively. Onsets are mean

\pm

sample SD over affected runs. Configured or native context limits are right-censored; NR means no event was observed within the evaluation horizon.

Figure 14: LLM-judge scores over time on FD-LHB-CANDOR. Mean (a) Coherence and (b) Context scores over 40 replays per system. Silence bins receive a score of 1. Curves for growing-context models end when the model reaches its context limit.

Figure 15: Failure trajectories with IAP+DRP over 40 CANDOR replays. Same format as Fig. 2 for (a) PersonaPlex and (b) Moshi with IAP+DRP, extended to 60 minutes. Top: repetition (Seq-Rep-4) and the percentage of speaking bins labeled gibberish; middle: when each failure appears; bottom: words per minute. W1–W15 mark consecutive four-minute sliding windows.

Model	Method	Any 2 min	Any 5 min	Resumed	Median time to resume (s)
Moshi	Baseline	$40/40$	$7/40$	$40/40$	95.3 IQR 57.2–134.4; max 202.6
Moshi	IAP	$13/40$	$0/40$	$13/13$	7.5 IQR 2.6–24.8; max 70.2
Moshi	DRP	$37/40$	$7/40$	$37/37$	95.2 IQR 68.4–134.0; max 203.9
Moshi	IAP+DRP	$3/40$	$0/40$	$3/3$	10.2 IQR 6.7–16.1; max 22.0
PersonaPlex	Baseline	$34/40$	$7/40$	$34/34$	53.6 IQR 37.0–118.8; max 172.0
PersonaPlex	IAP+DRP	$19/40$	$1/40$	$18/19$	26.6 IQR 12.4–40.2; max 47.3

Appendix

Table 11: Silence duration and resumption of speech on all 40 CANDOR replays. A one-minute bin is silent if it contains fewer than three words; the final partial minute is included. Any 2 min and Any 5 min count runs with a silence of at least two or five consecutive minutes. Resumed counts runs that produce at least one word after the first two-minute silence. Time to resume is measured from the end of that silence to the first word; cells report the median, IQR, and maximum in seconds.

Method	$T_{\mathrm{rep}}$	$T_{\mathrm{psil}}$	$T_{\mathrm{gib}}$
Baseline	$19.6\pm 7.5$ $26/40$ reached	$9.0\pm 0.8$ $40/40$ reached	$13.3\pm 0.9$ $40/40$ reached
IAP	$24.8\pm 21.3$ $4/40$ reached	$33.0\pm 14.3$ $13/40$ reached	NR $0/40$ reached
DRP	NR $0/40$ reached	$9.6\pm 0.6$ $37/40$ reached	$13.7\pm 2.0$ $40/40$ reached
IAP+DRP	NR $0/40$ reached	$30.3\pm 16.0$ $3/40$ reached	NR $0/40$ reached

Appendix

Table 12: Moshi component ablation on 40 matched CANDOR replays. Definitions and reporting follow Table 10 .

Figure 16: Response quality in the Moshi component ablation. Mean Coherence and Context scores over 40 matched CANDOR replays. Silence bins receive a score of 1.

Method	Rep. $\downarrow$	Sil. $\downarrow$	Output words/min
IAP only	$34/40$	$4/40$	$158.7$
IAP + standard RP	$5/40$	$21/40$	$126.9$
IAP + mass-preserving RP	$2/40$	$13/40$	$69.3$
IAP + DRP	$0/40$	$5/40$	$76.8$

Appendix

Table 13: DRP ablation on 40 matched PersonaPlex CANDOR replays. All runs use IAP and last 20 minutes. Rep. and Sil. count runs that reach repetition or prolonged silence. Output words/min is the number of model-generated words divided by the full 20-minute replay duration, including listening and silent periods; it is not an active-speech rate.

Control	Rep. $\downarrow$	Sil. $\downarrow$	Output words/min
IAP only	$34/40$	$4/40$	$158.7$
[0.4pt/1.2pt] Standard RP	$5/40$	$21/40$	$126.9$
LZ	$31/40$	$10/40$	$141.2$
Bayesian	$0/40$	$7/40$	$220.2$
[0.4pt/1.2pt] DRP	$0/40$	$5/40$	$76.8$

Appendix

Table 14: Alternative repetition controls on 40 matched PersonaPlex CANDOR replays. All runs use IAP and last 20 minutes. Rep. and Sil. count runs that reach repetition or prolonged silence. Output words/min is defined as in Table 13 .

Varied setting	Value	Rep. $\downarrow$	Sil. $\downarrow$	Output words/min
Penalty $\rho$	$1.2$	$4/40$	$4/40$	$87.2$
	$1.3$ (final)	$0/40$	$5/40$	$76.8$
	$1.5$	$0/40$	$3/40$	$84.1$
	$2.0$	$0/40$	$1/40$	$244.2$
History length $N$	$256$	$0/40$	$5/40$	$76.1$
	$512$ (final)	$0/40$	$5/40$	$76.8$

Appendix

Table 15: PersonaPlex DRP hyperparameter sensitivity on 40 matched 20-minute CANDOR replays. Each block varies one setting from the final configuration (

\rho=1.3

N=512

b_{\max}=3

). Rep. and Sil. count runs that reach repetition or prolonged silence. Output words/min is defined as in Table 13 .

Phase	Naturalness	Intelligibility	Prosody	Conv. style	Context
Early (14)	0.43 [0.30, 0.55]	0.38 [0.27, 0.48]	0.41 [0.28, 0.54]	0.43 [0.30, 0.56]	0.31 [0.22, 0.41]
Middle (11)	0.60 [0.51, 0.69]	0.59 [0.50, 0.68]	0.60 [0.50, 0.71]	0.60 [0.50, 0.71]	0.56 [0.46, 0.68]
Late (12)	0.53 [0.43, 0.65]	0.55 [0.41, 0.69]	0.60 [0.41, 0.78]	0.58 [0.43, 0.73]	0.58 [0.44, 0.73]
All (37)	0.52 [0.44, 0.59]	0.50 [0.43, 0.57]	0.53 [0.45, 0.61]	0.53 [0.45, 0.61]	0.47 [0.40, 0.55]

Appendix

Table 16: Human listening preferences for IAP+DRP over IAP alone on Moshi. Each cell shows how often listeners chose IAP+DRP, with ties counted as half. Higher values mean IAP+DRP was chosen more often, and 0.5 means no preference. Brackets show 95% bootstrap intervals; an interval that includes 0.5 indicates no clear preference. Numbers in parentheses are the numbers of matched pairs; the study contains 285 judgments per dimension from 15 listeners.

Model	Tested length	Observed behavior
mini-omni ( Xie and Wu, 2024b )	2,048 tokens	Repeats replies before reaching the context limit.
mini-omni2 ( Xie and Wu, 2024a )	2,048 tokens	Coherent until near the limit, then repeats.
Covo-Audio ( Wang et al., 2026b )	a few thousand tokens	Replies converge to a small repeated set.
Kimi-Audio ( KimiTeam et al., 2025 )	about 1.3K of 8K tokens	Switches to describing the audio and repeats that description.
Qwen2.5-Omni ( Xu et al., 2025 )	about 11 turns / 2.5K tokens	Outputs template text, then repeats one response.
Step-Audio 2-mini ( Wu et al., 2025 )	tested dialogue	Coherent early, then repeats replies.

Appendix

Table 18: Exploratory observations on additional speech models. One sequence per model; lengths and protocols differ across models.

Full-Duplex Spoken Dialogue Systems

Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.

See all 119 papers

Sep 29, 2026

MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

End-to-end full-duplex speech models have brought open-source machine conversation closer to human-like interaction, yet existing systems remain limited in two intertwined dimensions: long-context robustness and multi-party interaction. Real-world scenarios such as meetings, group lessons, and social-robot reception require a single model to track, contextualize, and respond to multiple speakers over extended durations. Progress is constrained by both data and evaluation: open multi-party speech corpora remain small and are not designed for codec-frame-level full-duplex modeling, while existing long-audio benchmarks focus on passive listening and speech-to-speech benchmarks are mostly short and dyadic. We extend the Moshi paradigm jointly along the long-horizon and multi-party axes in English and Chinese. First, we release 57.6k hours of synthetic training data (

[MultiTalkPT](https://huggingface.co/datasets/MultiTalk/MultiTalkPT)

and MultiTalkFT) for long-form, multi-party, English-Chinese full-duplex dialogue, with controllable length, participant count, turn-taking, overlap, backchannels, interruptions, addressee shifts, and long-range coreference. Second, we introduce MultiTalkBench, built from real human recordings, for evaluating long-form, multi-party, bilingual full-duplex dialogue. Conversations average 32.6 minutes and include probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style model that sustains coherent multi-party English-Chinese conversations over extended durations and substantially outperforms open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench.

Ke Wang, Houxing Ren, Zimu Lu +4

CUHK MMLab · CPII under InnoHK

Spoken Dialogue Systems Multi-Turn Dialogue Evaluation Speech Language Models Full-Duplex Spoken Dialogue Systems

Ask PDF

Oct 26, 2026

Diagnosing and Mitigating Long-Session Degradation in Full-Duplex Speech Models

End-to-end full-duplex spoken dialogue models enable natural, real-time spoken interactions with interruptions, backchannels, and overlapping speech. Prior evaluations have assessed diverse interaction capabilities and identified some degradation patterns across conversational turns. However, how model behavior evolves over extended sessions remains insufficiently understood. In this work, we address this gap by systematically characterizing long-session degradation and developing efficient methods to mitigate it. We introduce a closed-loop evaluation framework for long-session stability and apply it to several publicly available full-duplex speech models, revealing degradation over time in responsiveness, spoken content quality, and voice consistency. To mitigate this degradation, we propose a self-distillation-based training framework for lightweight LoRA adaptation that improves long-session stability with a small amount of additional training. We further introduce a two-resolution context (TRC) architecture that preserves recent interactions in full detail while compressing older interaction history into key-value representations. We evaluate our methods on three models with distinct architectures, assessing long-session stability, information retention, and computational efficiency. Across all three models, we show that LoRA adaptation mitigates performance degradation over ten-minute sessions, while adding TRC extends these gains beyond the training session length and reduces KV-cache growth by more than 90%.

Junho Kim, CheHyun Lee, Donghun Kang +4

IPAI, Seoul National University (SNU) · Department of ECE, SNU · Department of AI, University of Seoul +1

Spoken Dialogue Systems Full-Duplex Spoken Dialogue Systems Context Compression

Ask PDF

Oct 8, 2026

SteerablePlex: Can We Steer Full-Duplex Models?

Full-duplex speech models can listen and speak simultaneously, enabling natural interaction, but become increasingly difficult to control as the conversation history grows. When used as user simulators, this lack of control can cause them to deviate from prescribed scenarios and produce unreliable evaluation outcomes. We introduce SimIF-Bench (Simulator Instruction-Following Benchmark), which evaluates whether a conversational model stays within a prescribed scenario and completes multiple goals in the required order. The benchmark reveals that current open-source full-duplex models struggle to follow such constraints. We then introduce a Group Reward-Decoupled Normalization Policy Optimization (GDPO)-based training recipe that enables a full-duplex model to follow textual instructions during an ongoing conversation while maintaining its turn-taking ability. By connecting the resulting SteerablePlex to an asynchronous backend language model that monitors the conversation and provides instructions when needed, we build a more controllable full-duplex user simulator that follows multi-stage constraints more reliably than existing open-source models and GPT-Realtime.

Haolong Zheng, Maike Züfle, Dominik Macháček +6

University of Illinois Urbana-Champaign · Karlsruhe Institute of Technology · Charles University +3

Full-Duplex Spoken Dialogue Systems Language Model Steering Instruction Following Large Language Model-Based Role-Play Simulation Multi-Turn Dialogue Evaluation

Ask PDF

Abstract

Full-Duplex Spoken Dialogue Systems

Explore similar work

MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

Diagnosing and Mitigating Long-Session Degradation in Full-Duplex Speech Models

SteerablePlex: Can We Steer Full-Duplex Models?