Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing

Pakorn Nathong; Kunat Pipatanakul

cs.DCSep 4, 2026

Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing

Authors: Pakorn Nathong, Kunat Pipatanakul

Organizations: Paxa Labs

Abstract

Instance-billed serverless platforms charge for CPU and memory over the lifetime of a warm instance, making idle inference state a direct serving cost. We present billing-aware neural text-to-speech (TTS) serving on serverless CPUs, optimizing CPU-seconds and GB-seconds rather than throughput or latency alone. Conventional runtimes are poorly suited to this setting: per-request parallelism causes CPU contention under concurrency, while warm instances retain gigabytes of billable inference and page-cache state. We address these costs with request-sized concurrent inference, which bounds per-request CPU parallelism, and a reclaimable instance lifecycle, which releases inference state and page-cache memory after idle periods while retaining the server process and compile cache. On Kokoro-82M, our system achieves 2.71 audio-seconds per CPU-second versus 0.89 with ONNX Runtime defaults and reduces cost per audio-hour from $0.0631 with PyTorch to$ 0.0153, a 4.1x reduction. Idle billed memory falls from 8.7 GB to 1.33 GB, while restoration reaches first audio in 2.2 s versus 7.7 s for a PyTorch cold start. Under bursty traffic, lifecycle reclamation is essential for translating inference efficiency into lower serverless cost.

Figures & tables

Figure 1. Billed cost for one day of bursty traffic: two hours of synthesized audio delivered in eight equal bursts. Request sizing improves busy efficiency but leaves a large warm-idle footprint; the reclaimable lifecycle removes the idle-memory component that otherwise dominates the bill.

Panel of Figure 2 — Figure 2. Serving design. Top: concurrent requests use bounded per-request parallelism from a fixed inference pool. Bottom: after an idle period, the server releases inference state and restores it from the compile cache on the next request.

	audio-s/CPU-s $\,\uparrow$	first audio p50/p90 $\,\downarrow$	warm-idle MB $\,\downarrow$	cold start s $\,\downarrow$	restore s $\,\downarrow$	$/audio-h$ ,\downarrow$
PyTorch eager	0.66	2.46/8.0	8659	7.7	–	0.0631
ONNX Runtime default	0.89	0.94/4.9	3969	5.8	–	0.0468
ONNX Runtime sized	1.36	2.36/8.4	3958	5.7	–	0.0306
OpenVINO default	1.35	1.26/ 2.6	5250	6.9	–	0.0309
OpenVINO sized	2.71	0.69 /4.2	12097	7.4	–	0.0154
This work	2.71	0.94/4.2	1331	6.2	2.2	0.0153

Table 1. End-to-end serving of Kokoro-82M on the primary 32-vCPU Emerald-Rapids host. Efficiency and first-audio latency are measured at four concurrent clients; idle memory is measured 600 s after the last response unless noted; cold start is a fresh process start to first audio; and cost uses

(\alpha,\beta)

. The sized OpenVINO idle value is measured after 30 s, before any release. Restore is the time to first audio for the first request after a release. Best measured values are in bold .

Configuration	eff. $\,\uparrow$	first p50 s $\,\downarrow$	idle MB $\,\downarrow$	cold s $\,\downarrow$	restore s $\,\downarrow$	$/aud-h$ \downarrow$
OpenVINO defaults	1.35	1.26	5250	6.9	–	0.0309
+ sized threads, 4 streams	2.31	0.56	8684	6.7	–	0.0180
+ compile cache	2.33	0.64	7490	5.4	–	0.0179
+ 8 streams	2.71	0.69	12097	7.1	–	0.0154
+ idle release (this work)	2.71	0.94	1331	6.2	2.2	0.0153

Table 2. Mechanism ablation on the primary host. Efficiency and first audio are measured at four clients; idle memory is measured after 600 s, except for the eight-stream configuration measured after 30 s before release. Cold start is a process start to first audio, from a cached compile where a cache exists. Restore is the first-audio time of the first request after a release, reported as the median of 10 release-and-restore cycles.

Figure 3. Sustained throughput with increasing concurrency. Sized configurations scale to eight clients and then plateau; additional requests primarily queue.

Figure 4. Resident memory after the last request. Framework baselines retain their warm-idle footprint, while the reclaimable lifecycle reaches its released-state floor after the quiet period.

Figure 5. pocket-tts under the same lifecycle analysis. Left: billed memory remains resident after traffic stops. Right: serving exposes warm and process-cold states, with no intermediate release state.

Cost-Aware Inference

Cost-aware inference allocates models, computation, and serving resources to meet quality goals at lower latency, energy use, and monetary cost.

See all 281 papers

Rewriting TTS Inference Economics: Lightning V2 on Tenstorrent Achieves 4x Lower Cost Than NVIDIA L40S
Mar 24, 2026Ranjith M. S., Akshat Mandloi, Sudarshan KamathTTS Synthesis AI Accelerator Inference
Smallest AI
ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform
Jul 29, 2026Xiaoxiao Jiang, Suyi Li, Sheng Yao +5Efficient Neural Network Inference T2I Generation
Hong Kong University of Science and Technology · Alibaba Group
TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment
Sep 25, 2026Junxi Liu, Xiquan Li, Wenhao Guan +5Text-to-Audio Generation
X-LANCE Lab, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China +1

Mar 24, 2026J/K move · Enter open · S save

Rewriting TTS Inference Economics: Lightning V2 on Tenstorrent Achieves 4x Lower Cost Than NVIDIA L40S

Ranjith M. S., Akshat Mandloi, Sudarshan Kamath

Smallest AI

TTS Synthesis AI Accelerator Inference Cost-Aware Inference Mixed-Precision Quantization

Open paper AskPDF arXiv

Text-to-Speech (TTS) models are significantly more numerically fragile than Large Language Models (LLMs) due to their continuous waveform generation and perceptual sensitivity to small numerical perturbations. While aggressive precision reduction techniques such as BlockFloat8 (BFP8) and low-fidelity (LoFi) compute have been widely adopted in language models, applying similar strategies to TTS systems often results in audible artifacts, phase instability, and spectral distortion. In this work, we present Lightning V2, a production-grade TTS model co-optimized for Tenstorrent hardware. Through precision-aware architectural design and hardware-software co-optimization, we achieve over 95% LoFi computational fidelity and more than 80% BlockFloat8 deployment without measurable degradation in audio quality. Leveraging Tenstorrent's Network-on-Chip (NoC), distributed SRAM, and deterministic execution model, we reduce memory movement and redundant weight fetches, enabling efficient low-precision inference. Compared to an NVIDIA L40S baseline, Lightning V2 achieves approximately 4x lower on-prem accelerator cost at equivalent throughput, while maintaining production audio fidelity. Our results demonstrate that precision co-design, combined with hardware-aware optimization, can fundamentally reshape the economics of real-time speech inference.

Abstract

Cost-Aware Inference

Explore similar work

Rewriting TTS Inference Economics: Lightning V2 on Tenstorrent Achieves 4x Lower Cost Than NVIDIA L40S

ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform

TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment