What Did I Just Say? Self-Listening for Full-Duplex Speech Models

Xuanning Zhou; Junyi Ao; Xiaotong Liu; Tom Ko; Benyou Wang; Haizhou Li

cs.SDSep 4, 2026

What Did I Just Say? Self-Listening for Full-Duplex Speech Models

Authors: Xuanning Zhou, Junyi Ao, Xiaotong Liu, Tom Ko, Benyou Wang, Haizhou Li

Organizations: Shenzhen Loop Area Institute, China · The Chinese University of Hong Kong, Shenzhen, China

Abstract

Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.

Figures & tables

Figure 1: Self-Listening is inspired by the human self-monitoring mechanism during speech. Humans hear their own speech through feedback such as bone conduction (left); likewise, self-listening feeds model speech already played to the user back to the LLM (right), grounding the model in what the user has heard.

Figure 2: Anchoring gap in a controlled counting probe. Each system is interrupted while counting from 1 to 30 and is asked to report the last spoken number. The horizontal axis shows the last number actually spoken ( $a$ ), while the vertical axis shows the number subsequently reported by the model ( $r$ ). The dashed diagonal ( $r=a$ ) represents perfect anchoring. Vertical segments visualize the anchoring gap, $\Delta_{\mathrm{anchor}}=r-a$ ; crosses denote invalid reports.

Figure 3: Overview of the proposed Self-Listening full-duplex model. User speech, model speech, and model text are organized into a three-channel interleaved sequence, while the played waveform is fed back through the self-listening channel to enable anchoring and full-duplex interaction.

Type	Model	ACC (%) $\uparrow$	STOP (s) $\downarrow$	RESP (s) $\downarrow$
Closed-source	Gemini-3.1-Flash-Live-Preview	30.3	0.679	1.772
	GPT-Realtime-2.1	43.8	0.296	1.548
	Doubao-Realtime	2.3	0.588	1.872
Open-source	Moshi	2.2	1.078	5.674
	Freeze-Omni	0.0	0.588	1.176
	2-ch FD model	7.8	0.425	0.567

Table 1: AnchorSpeech-test evaluation of open-source and closed-source FD (full-duplex) models.

		Interruption			Backchannel
Group	Model	Respond $\uparrow$	STOP $\downarrow$	RESP $\downarrow$	Resume $\uparrow$	STOP $\uparrow$	RESP $\downarrow$
Closed-source	Gemini-3.1-Flash-Live-Preview	0.91	0.66	1.54	0.83	0.61	1.58
	GPT-Realtime-2.1	0.88	0.43	1.54	0.93	0.46	1.39
	Doubao-Realtime	0.85	1.23	1.88	0.97	0.73	1.97
Open-source	Moshi	0.59	1.33	1.87	0.11	1.21	2.33
	Freeze-Omni	0.83	1.53	1.35	0.77	0.66	1.56

Table 2: Performance on the Interruption and Backchannel tasks of Full-Duplex-Bench v1.5.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

Why Full-Duplex Speech Models Collapse in Long Conversations
Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue Systems Long-Context Modeling
Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea
Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
Jun 9, 2026Cheng-Kuang Chang, Kai-Wei Chang, Alexander H. Liu +1Language Model Steering Spoken Dialogue Systems
MIT CSAIL
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Oct 23, 2026J/K move · Enter open · S save

Why Full-Duplex Speech Models Collapse in Long Conversations

Yeonju Kim, Bella Godiva, Se Jin Park, Yong Man Ro

Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea

Full-Duplex Spoken Dialogue Systems Long-Context Modeling KV-Cache Management Multi-Turn Dialogue Evaluation Repetition in Language Generation

Open paper AskPDF arXiv

Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even while listening, they exhaust their context windows within minutes. Yet existing benchmarks evaluate only single turns or a few conversational rounds, too short to expose the long-horizon problem. We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which tests five recent full-duplex models on long replayed conversations and in live conversation with an LLM partner. Every evaluated model eventually either degrades or exhausts its context. Models that keep the full history often degrade before their context runs out, while sliding-window models still fall into repetition, silence, and gibberish. For sliding-window models, we identify two contributing mechanisms. Gibberish and silence are linked to Initialization-Anchor Eviction, where the sliding window discards the first KV entry. Repetition is also reinforced by the model's own recent outputs. Guided by this diagnosis, we apply two simple fixes: Initialization-Anchor Pinning (IAP) keeps that first entry, and Duplex Repetition Penalty (DRP) discourages echoing while guarding against prolonged silence. Together, they support the diagnosis and, without retraining, substantially reduce conversational collapse in replays of up to an hour while running in real time.

Abstract

Speech Language Models

Explore similar work

Why Full-Duplex Speech Models Collapse in Long Conversations

Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM