VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

Edresson Casanova; Jaehyeon Kim; Mariana Graterol Fuenmayor; Shehzeen Hussain; Viacheslav Klimkov; Valentin Mendelev; Mikyas Desta; Paarth Neekhara

eess.ASAug 13, 2026

VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

Authors: Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, +12 more

Organizations: NVIDIA Corporation

Abstract

Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.

Figures & tables

Figure 1: VoiceChat-TTS architecture overview.

Figure 2: Character-Aware Subword Encoder architecture overview.

			Unseen Speakers				Seen Speakers
Model	Type	# Turns	CER(%) $\downarrow$	WER(%) $\downarrow$	SECS $\uparrow$	Squim-MOS $\uparrow$	CER(%) $\downarrow$	WER(%) $\downarrow$	SECS $\uparrow$	Squim-MOS $\uparrow$
Ground Truth	-	-	$0.48$	$1.40$	$0.830$	$4.457$	-	-	-	-
Chatterbox-TTS [ 16 ]	Offline	-	$0.45\pm 0.02$	$1.24\pm 0.02$	$\mathbf{0.887\pm 0.001}$	$4.27\pm 0.003$	-	-	-	-
Audio Flamingo 3-Chat [ 5 ]	Streaming	-	$2.85\pm 0.33$	$4.51\pm 0.41$	$0.761\pm 0.002$	$3.60\pm 0.009$	-	-	-	-
Qwen3-TTS-12Hz-1.7B [ 6 ]	Streaming	-	$\mathbf{0.34\pm 0.02}$	$\mathbf{1.01\pm 0.05}$	$0.827\pm 0.001$	$\mathbf{4.45\pm 0.006}$	-	-	-	-
VoiceChat-TTS	Streaming	1	$1.00\pm 0.10$	$2.00\pm 0.10$	$0.757\pm 0.004$	$4.380\pm 0.004$	$1.20\pm 0.10$	$2.40\pm 0.10$	$0.785\pm 0.001$	$4.365\pm 0.001$

Table 1: Comparison between TTS models. VoiceChat-TTS is evaluated with different numbers of consecutive turns.

Model	CER(%) $\downarrow$	WER(%) $\downarrow$	Squim-MOS $\uparrow$
PersonaPlex [ 18 ]	$4.06$	$5.00$	$4.094$
PersonaPlex + VoiceChat-TTS	$\mathbf{2.05\pm 0.78}$	$\mathbf{2.42\pm 0.80}$	$\mathbf{4.292\pm 0.007}$

Table 2: Comparison between PersonaPlex and VoiceChat-TTS on the Smooth Turn Taking subset of Full-Duplex-Bench, using the same PersonaPlex-predicted textual content.

Force Silence	IOR@320ms (%) $\uparrow$	Mean stop latency (ms) $\downarrow$	Leakage@1s (ms) $\downarrow$	AI-CER (%) $\downarrow$
False	$96.8$	$228.3$	$169.1$	$0.255$
True	$\mathbf{100.0}$	$\mathbf{89.9}$	$\mathbf{55.8}$	$\mathbf{0.095}$

Table 3: Effect of deterministic silence forcing on the FDB-timed controlled-interruption benchmark. Timing and leakage metrics are computed using the FDB v1.5 Silero VAD pipeline. AI-CER is computed only on the subsequent assistant turn and excludes the interruption-overlap interval. Values are means over ten stochastic runs; 95% confidence intervals are omitted for compactness.

Model	Conc.	Params	Acoustic ITL	Codec	Total
			(ms) $\downarrow$	(ms) $\downarrow$	(ms) $\downarrow$
Qwen3-TTS-12Hz [ 6 ]	1	1.7B	$15.44$	$4.90$	$20.34$
VoiceChat-TTS	1	977M	$\mathbf{7.16}$	$\mathbf{2.46}$	$\mathbf{9.62}$
Qwen3-TTS-12Hz [ 6 ]	4	1.7B	$17.09$	$4.99$	$22.08$
VoiceChat-TTS	4	977M	$\mathbf{12.35}$	$\mathbf{2.89}$	$\mathbf{15.24}$

Table 4: Streaming latency comparison on an RTX A6000 GPU. ITL denotes the time to produce one frame of acoustic tokens at 12.5 Hz, corresponding to 80 ms of audio. Codec denotes the measured time to decode one acoustic-token frame into waveform audio.

Explore similar work

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Sep 18, 2026Jagadeesh Balam, Travis Bartley, Edresson Casanova +52Tool-Augmented Language Model Agents Speech Language Models
Wookie
Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead
Jun 20, 2026Muyang Du, Jason Roche, Junjie LaiTTS Synthesis Streaming TTS Synthesis
1NVIDIA, China · 2NVIDIA, USA
FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
Jun 8, 2026Hanke Xie, Xiaming Ren, Dake Guo +10Flow Matching Multi-Token Prediction
Huawei Technologies Co., Ltd

Sep 18, 2026J/K move · Enter open · S save

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan+51

Wookie

Tool-Augmented Language Model Agents Speech Language Models Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieves the lowest pause-handling takeover rates among evaluated open-weight systems, 100% takeover following user interruptions, and a 4.33/5 post-interruption response-quality score. On Full-Duplex-Bench 1.5, it resumes its response after user backchannels in 93% of cases. NemotronLabs VoiceChat obtains a 55.1 normalized average on VoiceBench and, on Full-Duplex-Bench 3.0 (FDB 3.0), achieves 82.5% tool-selection F1, while argument accuracy and end-to-end tool execution remain areas for improvement. These results demonstrate that full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior.

Abstract

Explore similar work

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead

FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation