Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial

Jielin Qiu; Zixiang Chen; Liangwei Yang; Ming Zhu; Zhiwei Liu; Juntao Tan; Wenting Zhao; Rithesh Murthy

cs.SDMar 5, 2026

Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial

Authors: Jielin Qiu, Zixiang Chen, Liangwei Yang, Ming Zhu, Zhiwei Liu, Juntao Tan, Wenting Zhao, Rithesh Murthy,

Organizations: Salesforce AI Research

Abstract

We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$ 702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$ 146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component.

Figures & tables

Model	Level	Size	FC	FD	Streaming	License
Moshi [ 4 ]	1	7B	✗	✓	✓	CC-BY
Qwen3-Omni [ 18 ]	2	30B	✓	✗	✓ †	Apache 2.0
Qwen2.5-Omni [ 26 ]	2	7B	✗	✗	Partial	Apache 2.0
GLM-4-Voice [ 27 ]	2	9B	✗	✗	✓	Apache 2.0
Kimi-Audio [ 14 ]	2	7B	✗	✗	✓	–
Step-Audio [ 23 ]	2	130B	✗	✗	✓	Apache 2.0

Table 1: Representative speech-to-speech models with open weights. “FC” = function calling support. “FD” = full-duplex. “Streaming” = supports incremental audio output. † Streaming audio via DashScope Realtime API; local vLLM supports text streaming only (Thinker).

Mode	Inference Time	Audio Duration	RTF
Text only	0.71s	–	–
Text + Audio	26.5s	13.8s	1.93 $\times$
Text + Audio (per sentence)	13.2s	6.8s	1.94 $\times$

Table 2: Qwen2.5-Omni-7B inference latency on a single NVIDIA H200 GPU.

Configuration	TTFT	TTFA	tok/s	FC	Audio Out
vLLM Thinker (1 $\times$ H200)	48ms	516ms ∗	168	✓	✗
Transformers full (1 $\times$ H200)	–	145,694ms	4.6	✓	✓ (2.91 $\times$ RTF)
DashScope Realtime API	449ms	702ms	–	Partial	✓ (streaming)

Table 3: Qwen3-Omni inference latency across deployment configurations. TTFT = time-to-first-text, TTFA = time-to-first-audio, FC = function calling.

With cloud OpenAI API (GPT-4.1-mini)
Component	P50	Mean	Min	Max
Deepgram STT	402	407	184	601
LLM TTFT	457	622	278	784
ElevenLabs TTS TTFB	221	226	216	253
Est. E2E TTFA	958	1054	715	1606
With self-hosted vLLM (Qwen2.5-7B-Instruct on H200)

Table 4: Latency benchmarks (milliseconds). TTFA estimated as STT + LLM TTFT + TTS TTFB (sequential upper bound).

Approach	TTFA	FC	Audio Out	Self-hosted
Qwen2.5-Omni (sentence streaming)	$\sim$ 13,200ms	✗	✓	✓
Qwen3-Omni (transformers, full)	$\sim$ 145,694ms	✓	✓	✓
Qwen3-Omni (vLLM Thinker + TTS)	$\sim$ 736ms ∗	✓	✓	✓
Qwen3-Omni (DashScope Realtime)	$\sim$ 702ms	Partial	✓	✗
Cascaded pipeline (this work)	$\sim$ 755ms	✓	✓	✓

Table 5: Time-to-first-audio comparison across all approaches. “Self-hosted” = runs entirely on local GPUs without external API dependencies.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Ch	Topic	Key Files	Concept
1	Streaming STT	deepgram_streaming.py	WebSocket, partial/final transcripts
2	Streaming LLM	vllm_function_calling.py	vLLM, SSE streaming, tool use
3	Streaming TTS	elevenlabs_streaming.py	Streaming synthesis, TTFB
4	Pipeline	pipeline.py , sentence_buffer.py	Sentence aggregation, overlap
5	WebSocket Server	server.py	Binary audio protocol
6	VAD	vad_basics.py	Silero VAD, state machine

Appendix

Table 6: Complete tutorial chapter listing.

Conversational Agents

Conversational agents are AI systems that interact with people through multi-turn dialogue to answer questions, provide support, or complete tasks.

See all 287 papers

VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Aug 13, 2026Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor +17TTS Synthesis Streaming TTS Synthesis
NVIDIA Corporation
StepAudio 3 Realtime Technical Report
Sep 12, 2026Bin Lin, Bo Zhao, Boyang Zhang +87Audio-Language Models Full-Duplex Spoken Dialogue Systems
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
May 14, 2026Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo +3Audio-Language Model Evaluation Voice Agent Evaluation
Dialpad Inc.

Aug 13, 2026J/K move · Enter open · S save

VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain+16

NVIDIA Corporation

TTS Synthesis Streaming TTS Synthesis Spoken Dialogue Systems Controllable Speech Generation

Open paper AskPDF arXiv

Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.

Abstract

Conversational Agents

Explore similar work

VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

StepAudio 3 Realtime Technical Report

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents