Speech Language Models (SpeechLMs) are increasingly deployed for streaming speech generation, where audio playback must start quickly and never stall. Serving them requires running two very different model components on the same system. An autoregressive LLM backbone emits audio tokens, and a codec converts chunks of those tokens into waveforms. Existing systems run the two as separately scheduled stages, and therefore either serialize them, let them interfere, or strand capacity by placing them on separate devices. We observe that overlapping the backbone and the codec on separate parts of the GPU raises throughput. Although this increases codec latency, most chunks can tolerate it, as long as it neither lies on the critical path of time-to-first-audio (TTFA) nor causes a playback stall. Based on this insight, we present VoxServe, a SpeechLM serving system that allocates GPU resources according to the urgency of chunks. VoxServe executes work on three asynchronous lanes. The LM lane runs the backbone on a large partition of the GPU's streaming multiprocessors, the background codec lane runs on a small partition beside it, and the urgent lane uses the whole GPU. The scheduler queues each request to the LM lane and routes every chunk the backbone completes by its urgency. A request's first chunk and chunks close to their playback deadline go to the urgent lane, and the rest are batched on the background lane. On Qwen3-TTS, VoxServe sustains 1.5x the request rate of the best existing system at a 200ms TTFA target without playback stalls. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.
Figures & tables
Figure 1: Typical workflow of SpeechLM inference. An LLM backbone autoregressively generates discrete audio tokens from text (and optionally audio) inputs, and an audio codec reconstructs continuous waveforms from those tokens.
Figure 2: Three ways to place the LLM backbone (LM) and the codec on one GPU: (a) time sharing, (b) colocation on concurrent streams, and (c) an SM partition. In (b) and (c), urgent chunks wait behind codec batches.
Figure 3: Microbenchmarks of Qwen3-TTS on one H100. (a) Speed of each component alone on an SM partition, relative to all 132 SMs. (b) Throughput of the backbone (batch 32) and the codec (batches of 16), both running back-to-back, under time sharing or on each SM partition (labels: LM/codec SMs). Below the dotted line, the codec falls behind the backbone. (c) Codec latency on each partition, and at 132 SMs alone on the whole GPU (dotted lines).
Figure 4: Overview of VoxServe . In each iteration, the per-GPU scheduler forms an LM batch and routes each request’s next chunk by urgency: first chunks and chunks whose request has little slack go to the urgent lane on all SMs, and other chunks to the background codec batch.
Figure 5: Serving performance for Qwen3-TTS on one H100 with LibriTTS workload. (Left) SLO attainment, the share of requests whose TTFA is at most 200 ms and whose playback never stalls, with a dotted line at 90%. (Middle) p99 TTFA, with the axis cut at 1 s and a dotted line at the 200 ms target. (Right) Goodput, the requests per second that meet the SLO, with the offered rate dotted. Lines pool three arrival seeds (median for p99), and bands show their range.
Figure 6: Serving performance for CosyVoice 3 (0.5B) and Higgs Audio v3 (4B) on one H100 with LibriTTS workload. (Left) SLO attainment. (Right) p99 TTFA, with the axis cut at 1 s. Lines pool three arrival seeds.
Figure 7: SLO capacity of Qwen3-TTS on two and four H100s with LibriTTS workload. SGLang-Omni runs with replication or disaggregation of LM and codec stages onto separate GPUs (LM/codec GPUs). Solid bars give the highest rate at which 90% of requests meet the SLO, and hatched bars 99%.
Figure 8: Serving performance for Qwen3-TTS on one H100 with Seed-TTS workload, one arrival seed per system.
Figure 9: Serving performance for Qwen3-TTS on one B200 with LibriTTS workload. (Left) SLO attainment. (Right) p99 TTFA, with the axis cut at 1 s. Lines pool three arrival seeds.
Figure 10: Ablation on Qwen3-TTS, LibriTTS. SLO capacity (TTFA ≤ 200 ms, no stall) at 90% (solid) and 99% (hatched) attainment as VoxServe ’s mechanisms are added one at a time to a synchronous loop that time-shares the GPU (three arrival seeds pooled per rate, rates in steps of 2 req/s).
Figure 11: Qwen3-TTS on one H100 with LibriTTS workload around and past capacity, with prefill-first admission, deadline-aware admission, and the synchronous design. (Left) Completed requests per second of wall time, including the time to finish requests still running when arrivals stop, with the offered rate dotted. (Right) SLO attainment.
Figure 12: Sensitivity of VoxServe to the SM split on Qwen3-TTS, LibriTTS, one H100. SLO capacity at 90% (solid) and 99% (dashed) attainment against the codec’s SMs (“all” shares the GPU without a split). The default is 32 SMs.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Backbone
Codec
Encoder
K
Tokens/s
Chunk
Orpheus 3B Canopy Labs (2025)
3B Llama
SNAC (convolutional)
–
7
82
28 (0.3 s)
Zonos-v0.1 Zyphra Team (2025)
1.6B
DAC (convolutional)
–
9
86
50 (0.6 s)
Higgs Audio v3 Boson AI (2026)
4B Qwen3
Higgs codec (DAC decoder)
yes
8
25
45 (1.8 s)
CSM 1B Iribe et al. (2025)
1B + depth transformer
Mimi (Transformer)
–
32
12.5
10 (0.8 s)
Qwen3-TTS Hu et al. (2026)
1.7B Qwen3 + depth transformer
Qwen3 codec (Transformer + conv.)
–
16
12.5
10 (0.8 s)
Voxtral-TTS Liu et al. (2026)
4B Mistral
Voxtral codec
–
37
12.5
25 (2.0 s)
Appendix
Table 1: SpeechLMs supported by VoxServe . K : codes per audio token (codebooks). Chunk: tokens per codec invocation (in parentheses: audio duration).
Recent progress in speech dialogue systems requires Text-to-Speech (TTS) models to be faster and more responsive. Modern speech dialogue systems impose two primary requirements on TTS models: low latency and support for streaming inputs and outputs. However, most existing single-codebook LLM-based TTS methods rely on multi-stage pipelines that lack native streaming capabilities. These systems typically suffer from high end-to-end latency due to slow autoregressive prediction and multi-step flow matching. To address these limitations, we propose FlashTTS, an open-source and low-latency streaming TTS framework. FlashTTS introduces a lagged multi-track architecture that natively processes streaming text and speech inputs, thereby eliminating the need for sentence-level buffering. To accelerate acoustic generation, we integrate parallel Multi-Token Prediction (MTP) with an X-pred mean flow matching decoder. This configuration achieves high-fidelity token-to-mel generation in exactly two function evaluations (2-NFE). By jointly optimizing input processing and decoding efficiency, FlashTTS offers a practical foundation for real-time speech dialogue systems. Experiments show that FlashTTS substantially reduces First-Packet Latency to 325ms compared to robust streaming baselines, all while preserving strong zero-shot voice cloning and cross-lingual intelligibility. Speech samples are available. The model code and checkpoints will be released as open source.
Text-to-speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality usually comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU. Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches about 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. In a blinded study, 40 listeners preferred TontaubeV1's prosody over Fish Audio S2 Pro in 70.25% of 400 comparisons (95% CI: 64.2-75.7%). On an LLM-as-a-judge benchmark, it matches ElevenLabs Flash v2.5 and outperforms the April 2026 Gradium API and Cartesia Sonic 3 on prosody. The weights are released on Hugging Face under the Tontaube Community Model License 1.0.
While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications. Direct adaptation to streaming scenarios often leads to catastrophic inference performance degradation due to the severe mismatch between training and streaming inference. To bridge this gap, we present the first autoregressive (AR) models tailored for streaming TSE. Our approach introduces a Chunk-wise Interleaved Splicing Paradigm that ensures highly efficient and stable streaming inference. To ensure the coherence between the extracted speech segments, we design a historical context refinement mechanism that mitigates boundary discontinuities by leveraging historical information. Experiments on Libri2Mix show that while AR generative baseline exhibits performance degradation at low latencies, our approach maintains 100% stability and superior intelligibility. Furthermore, our streaming results are comparable to or even surpass offline baselines. Additionally, our model achieves a Real-Time-Factor (RTF) of 0.248 on consumer-level GPUs. This work provides empirical evidence that AR generative backbones are viable for latency-sensitive applications through the Chunk-wise Interleaved Splicing Paradigm.
Shuhai Peng, Hui Lu, Jinjiang Liu +8
Shenzhen International Graduate School, Tsinghua University · The Chinese University of Hong Kong · SenseTime Research