NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

Jagadeesh Balam; Travis Bartley; Edresson Casanova; Sanjay Chauhan; Chen Chen; Zhehuai Chen; Zijia Chen; Francesco Ciannella

cs.CLSep 18, 2026

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

Authors: Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan, Chen Chen, Zhehuai Chen, Zijia Chen, Francesco Ciannella, +47 more

Organizations: Wookie

Abstract

We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieves the lowest pause-handling takeover rates among evaluated open-weight systems, 100% takeover following user interruptions, and a 4.33/5 post-interruption response-quality score. On Full-Duplex-Bench 1.5, it resumes its response after user backchannels in 93% of cases. NemotronLabs VoiceChat obtains a 55.1 normalized average on VoiceBench and, on Full-Duplex-Bench 3.0 (FDB 3.0), achieves 82.5% tool-selection F1, while argument accuracy and end-to-end tool execution remain areas for improvement. These results demonstrate that full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior.

Figures & tables

Figure 1: NemotronLabs VoiceChat Architecture Overview.

Figure 2: Tool-calling channel. For space efficiency, we omit the tool and agent-text channels on the input side of the figure.

	Pause handling		Smooth turn-taking		User interruption
Model	Synthetic TOR ( $\downarrow$ )	CANDOR TOR ( $\downarrow$ )	TOR ( $\uparrow$ )	Latency (s, $\downarrow$ )	TOR ( $\uparrow$ )	Response quality (GPT-4o, $\uparrow$ )	Latency (s, $\downarrow$ )
Open-weight systems
Moshi	98.5	98.0	94.1	0.265	100.0	0.77	0.257
Freeze-Omni	64.2	48.1	33.6	0.953	86.7	3.62	1.409
PersonaPlex	35.8	43.1	90.8	0.170	95.0	4.29	0.240
MoshiRAG †	32.0	56.0	83.0	0.180	85.0	3.75	1.020

Table 1: Full-Duplex-Bench 1.0 turn-management results. Behavioral rates are reported in percent, latency in seconds, and GPT-4o response-quality scores on a 0–5 scale. TOR denotes Takeover Rate.

Model	Respond ( $\downarrow$ )	Resume ( $\uparrow$ )	Uncertain ( $\downarrow$ )	Unknown ( $\downarrow$ )
Open-weight systems
Moshi	2.0	6.0	0.0	92.0
Freeze-Omni	7.0	80.0	2.0	11.0
MoshiRAG †	5.0	61.0	0.0	34.0
VC-Model	1.0	93.0	2.0	4.0
Closed-API systems

Table 2: FDB 1.5 behavioral response distribution for the user-backchannel condition. Values are percentages. A listener backchannel does not claim the floor, so Resume is the desired behavior.

Model	Arch.	OBQA	MMSU	CE	SD-QA	AE	BBH	WV	IFEval	AdvBench	Avg.
Moshi	FD	25.9	24.0	1.6	15.6	2.0	47.4	1.3	10.1	44.2	29.5
PersonaPlex ‡	FD	24.4	24.9	2.3	18.8	2.7	49.4	2.0	12.0	8.1	30.6
Freeze-Omni	FD	31.0	28.1	3.5	53.5	4.0	50.7	3.2	23.4	97.3	55.2
DuplexCascade	Cascade-FD	56.0	52.9	3.6	45.6	4.4	59.8	3.6	43.4	99.0	65.4
MiniCPM-o 4.5 †	Omni-FD	87.7	66.7	3.6	68.4	4.2	55.0	3.6	80.6	98.9	76.1
VC-Model	FD	61.3	46.1	3.0	32.6	3.4	51.7	2.8	19.3	100.0	55.1

Table 3: VoiceBench intelligence results for open-weight systems with full-duplex conversational capability. CommonEval (CE), AlpacaEval-Full (AE), and WildVoice (WV) are reported on a 1–5 scale; all other task scores and the normalized average are on a 0–100 scale. Higher is better. FD denotes a full-duplex speech-to-speech system, Cascade-FD denotes a full-duplex cascaded ASR–LLM–TTS system, and Omni-FD denotes an omni-modal model supporting full-duplex interaction.

Model	Tool Sel. F1 ( $\uparrow$ )	Arg. Acc. ( $\uparrow$ )	Pass@1 ( $\uparrow$ )
Ours	82.5	42.2	33.0
Gemini Live 2.5	78.6	59.3	49.0
Gemini Live 3.1	81.7	58.8	54.0

Table 4: FDB 3.0 tool-calling results (%). Baseline results are from [ 29 ] .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Data bucket	Scale	CPT	SFT
Retention
Speech-text pretraining	$\approx$ 530k	0.95	0.40
Text-only knowledge	Extract-knowledge shards (text)	–	0.10
Spoken MCQ	$\approx$ 24k h	–	0.03
Single-turn QA	$\approx$ 12k h	0.05	0.02
Conversational

Appendix

Table 5: Data mixture across training stages. Entries are raw sampler weights for weighted randomized round-robin fusion; a dash denotes a bucket unused in that stage. Scale is the approximate size of the shards each bucket indexes, where conv. denotes conversations; the pretraining bucket draws from a different collection in each stage. SFT weights are normalized by the sampler.

System	Speaker	Turn	WER (%) $\downarrow$	SECS $\uparrow$	SQuIM-MOS $\uparrow$
Ground truth	Unseen	–	1.40	0.830	4.457
Chatterbox-TTS	Unseen	–	$1.24\pm 0.02$	$\mathbf{0.887\pm 0.001}$	$4.270\pm 0.003$
Audio Flamingo 3-Chat	Unseen	–	$4.51\pm 0.41$	$0.761\pm 0.002$	$3.600\pm 0.009$
Qwen3-TTS-12Hz-1.7B-Base	Unseen	–	$\mathbf{1.01\pm 0.05}$	$0.827\pm 0.001$	$\mathbf{4.450\pm 0.006}$
VoiceChat-TTS	Unseen	1	$2.00\pm 0.10$	$0.757\pm 0.004$	$4.380\pm 0.004$
VoiceChat-TTS	Unseen	4	$2.20\pm 0.20$	$0.685\pm 0.005$	$4.376\pm 0.003$

Appendix

Table 6: TTS quality and multi-turn stability on LibriTTS. Baselines are evaluated on unseen speakers. VoiceChat-TTS is shown at turns 1 and 4 for unseen and seen speakers. Lower WER and higher SECS/SQuIM-MOS are better.

	Chunk Size (ms)
Dataset	80	160
AMI	15.32	13.54
Earnings22	14.82	14.04
Gigaspeech	12.25	11.65
LS Clean	3.58	3.19
LS Other	8.15	7.19

Appendix

Table 7: ASR Benchmarking results on Hugging Face OpenASR Leaderboard showing WER (%) for all datasets [ 38 ] .

Explore similar work

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengFull-Duplex Speech Models Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
A frontend-backend architecture for tool calls in full-duplex speech models
Sep 16, 2026Ke Hu, Slyne Deng, Chen Chen +10Full-Duplex Speech Models Call
NVIDIA, USA
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sep 8, 2026Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu +7Full-Duplex Speech Models Speech Synthesis
MediaTek Research · Internship at MediaTek Research · National Taiwan University +2

Jun 12, 2026, cs.CLJ/K move · Enter open · S save

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Qingkai Fang, Shoutao Guo, Yang Feng

Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Full-Duplex Speech Models Speech Language Models Autoregressive Language Models Chatbots

Open paper AskPDF arXiv

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Abstract

Explore similar work

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

A frontend-backend architecture for tool calls in full-duplex speech models

TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction