TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment

Junxi Liu; Xiquan Li; Wenhao Guan; Yifan Duan; Zhikang Niu; Yanru Huo; Ziyang Ma; Xie Chen

cs.SDSep 25, 2026

TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment

Authors: Junxi Liu, Xiquan Li, Wenhao Guan, Yifan Duan, Zhikang Niu, Yanru Huo, Ziyang Ma, Xie Chen

Organizations: X-LANCE Lab, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · Nanyang Technological University

Abstract

Text-to-audio (TTA) generation has advanced rapidly in generation quality and instruction following. However, representative systems often require around a billion parameters, limiting deployment on resource-constrained devices. This paper introduces TinyAudio, a compact flow-matching-based TTA model for low-resource deployment. At its core, TinyAudio uses TA-DiT, a 35M single-stream flow-matching Transformer. TinyAudio also includes TA-CLAP, a 32M audio-aligned text encoder, and TA-VAE, whose 20M decoder reconstructs 44.1 kHz audio from compressed latents. TinyAudio has only 87M parameters in total, over 90% fewer than representative billion-parameter pipelines, and uses 0.48 GB peak GPU memory. TinyAudio achieves competitive generation quality on AudioCaps and TTA-Bench. We further introduce TinyAudio-MF, a MeanFlow-accelerated model that enables real-time generation with a four-core CPU quota. Our results demonstrate a practical quality-footprint trade-off for low-resource TTA deployment.

Figures & tables

Figure 1: AudioCaps quality versus generator size. Bubble labels indicate generator parameters, and lower FD is better. TinyAudio operates in a substantially smaller parameter regime while retaining competitive distributional quality.

Figure 2: Single-stream TA-DiT. Projected text tokens and noised audio latents are concatenated for joint self-attention. The right panel shows layer-shared conditional modulation: shared MLPs combine text, timestep, and block embeddings to produce shift–scale and gate parameters. Only audio tokens predict latent velocity.

Model	Footprint / Efficiency					AudioCaps				TTA-Bench
Model	Gen.	Deploy.	Mem.	NFE	RTF	FAD $\downarrow$	FD $\downarrow$	KL $\downarrow$	IS $\uparrow$	CE $\uparrow$	CU $\uparrow$	PC $\uparrow$	PQ $\uparrow$	CLAP $\uparrow$
AudioLDM-L-Full	739M	953M	3.89 GB	200 $\times$ 2	1.190	4.32	29.50	1.68	8.17	3.275	5.137	3.219	5.854	0.441
AudioLDM2-Large	718M	1467M	5.93 GB	200 $\times$ 2	4.173	3.45	25.53	1.65	7.98	3.441	5.440	2.957	5.984	0.415
Tango-Full	866M	1295M	5.28 GB	200 $\times$ 2	1.880	2.68	15.64	1.24	8.78	3.264	5.151	3.360	5.954	0.440
Tango-2-Full	866M	1295M	5.28 GB	200 $\times$ 2	1.879	2.87	15.93	1.20	10.05	3.468	5.196	3.800	5.895	0.467
MeanAudio-L-Full	480M	1032M	4.79 GB	25	0.057	2.59	14.06	1.21	12.46	3.304	5.151	2.995	5.743	0.469

Table 1: Generation quality, deployment footprint, and inference efficiency. Gen. and Deploy. denote generator and total inference-time parameter counts. Mem. (peak GPU memory during inference) and RTF (end-to-end real-time factor) are measured for all systems on a single GPU in FP32 with batch size 1. NFE counts function evaluations;

{\times}2

indicates classifier-free guidance. Bold and underline indicate best and second-best results. TTA-Bench results use the Accuracy subset, with baseline values following the Resonate evaluation. [ 16 ]

Model	OVL $\uparrow$	REL $\uparrow$
TinyAudio	3.560 $\pm$ 0.985	3.520 $\pm$ 0.940
TangoFlux	3.505 $\pm$ 0.989	3.600 $\pm$ 0.992
EzAudio	3.155 $\pm$ 1.169	3.190 $\pm$ 1.169
AudioLDM	3.015 $\pm$ 1.004	2.730 $\pm$ 0.962

Table 2: Subjective evaluation of overall quality (OVL) and text relevance (REL). Results are reported as mean

\pm

standard deviation. Best and second-best scores are shown in bold and underlined.

Architecture
Setting	FAD $\downarrow$	KL $\downarrow$	IS $\uparrow$
Base	2.494	1.399	10.695
Per-block AdaLN	2.658	1.370	10.455
Dual-stream MMDiT	3.168	1.349	10.093
Text Conditioning
w/o CLAP adaptation	4.051	1.593	8.466

Table 3: Component ablations on the AudioCaps test set. All variants are trained for 200K steps on the combined AudioSet and AudioCaps corpus; Base denotes the corresponding complete TA-DiT configuration.

Model	Params	T $\rightarrow$ A			A $\rightarrow$ T
		R@1	R@5	R@10	R@1	R@5	R@10
LAION-CLAP	200M	35.1	71.9	83.7	44.2	80.8	90.3
TA-CLAP	67M	36.1	70.2	83.0	48.9	80.9	90.6

Table 5: Audio–text retrieval performance of TA-CLAP on AudioCaps test split.

Text-to-Audio Generation

Text-to-Audio Generation creates speech, music, and sound effects from natural-language prompts, aiming to match their content, timing, and acoustic qualities.

See all 44 papers

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation
Sep 23, 2026Runwu Shi, Kai Li, Yujin Wang +8Text-to-Audio Generation Audio Representation Learning
Institute of Science Tokyo · Tsinghua University · Wuhan University +2
FdAudio: MeanFlow-Anchored Fréchet-Distance Post-Training for One-Step Text-to-Audio Generation
Jul 11, 2026Kuan-Po Huang, Bo-Ru Lu, Ho-Lam Chung +2Text-to-Audio Generation Flow Matching
National Taiwan University · Amazon
SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation
Jun 30, 2026Binh Mai, Tran Quoc Bao Le, Hung Dinh +1Text-to-Audio Generation Diffusion Model Distillation
Posts and Telecommunications Institute of Technology, Hanoi, Vietnam

Sep 23, 2026J/K move · Enter open · S save

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

Runwu Shi, Kai Li, Yujin Wang, Dong Yang+7

Institute of Science Tokyo · Tsinghua University · Wuhan University+2

Text-to-Audio Generation Audio Representation Learning Audio Tokenization Flow-Based Generative Modeling

Open paper AskPDF arXiv

Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce Unite-Audio, to our knowledge, is the first to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve text-conditioned generation. Experiments show competitive TTA performance with a compact latent flow model, while ablation studies confirm the benefit of jointly learning the audio representation and generative model. Audio samples are available at https://runwushi.github.io/Unite-Audio.

Abstract

Text-to-Audio Generation

Explore similar work

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

FdAudio: MeanFlow-Anchored Fréchet-Distance Post-Training for One-Step Text-to-Audio Generation

SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation