Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

Potsawee Manakul; Woody Haosheng Gan; Martijn Bartelds; Guangzhi Sun; William Held; Diyi Yang

cs.SDFeb 18, 2026

Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

Authors: Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds, Guangzhi Sun, William Held, Diyi Yang

Organizations: Stanford University · SCB 10X · University of Southern California · University of Cambridge ·

Abstract

Current audio language models are predominantly text-first, either extending pre-trained text LLM backbones or relying on semantic-only audio tokens, limiting general audio modeling. This paper presents a systematic empirical study of native audio foundation models that apply next-token prediction to audio at scale, jointly modeling semantic content, acoustic details, and text to support both general audio generation and cross-modal capabilities. We provide comprehensive empirical insights for building such models: (1) We systematically investigate design choices -- data sources, text mixture ratios, and token composition -- establishing a validated training recipe. (2) We conduct the first scaling law study for discrete audio models via IsoFLOP analysis on 64 models spanning $3{\times}10^{18}$ to $3{\times}10^{20}$ FLOPs, finding that optimal data grows 1.6 $\times$ faster than optimal model size. (3) We apply these lessons to train SODA (Scaling Open Discrete Audio), a suite of models from 135M to 4B parameters on 500B tokens, comparing against our scaling predictions and existing models. SODA serves as a flexible backbone for diverse audio/text tasks -- we demonstrate this by fine-tuning for voice-preserving speech-to-speech translation, using the same unified architecture.

Figures & tables

Figure 1: Three token types examined in § 4.3 : (a) Semantic-only, (b) Semantic+Acoustic, and (c) Utterance-level interleaved Semantic+Acoustic+Text, where (c.1) shows text-first format, and (c.2) shows audio-first format. Superscript

i

denotes utterance index, where utterance(

i

) and utterance(

i

+1) are adjacent audio segments from the same document. See Appendix A.2 for detailed data formatting.

Figure 2: Impact of adding Nemotron on NLL for audio and text validation data. Full results on other metrics are shown in Figure 5 .

Figure 3: Validation NLL (audio+text) versus downstream task performance. Circular points: 64 IsoFLOP models (§ 5 ); star-shaped points: final SODA runs (§ 6 ). Regression lines are fitted on 64 IsoFLOP models only. Full results with other metrics are shown in Appendix E .

	Semantic		Acou.	Text		Cross-Modal
	sBLI $\uparrow$	sWUG $\uparrow$	Salm $\uparrow$	tBLI $\uparrow$	tWUG $\uparrow$	ASR ${}_{\scriptsize\text{WER}}$ $\downarrow$	TTS ${}_{\scriptsize\text{WER}}$ $\downarrow$
S	58.6	72.1	67.3	$\times$	$\times$	$\times$	$\times$
S+A	50.9	59.0	70.1	$\times$	$\times$	$\times$	$\times$
S+A+T	50.4	58.1	70.4	67.8	71.6	18.3	27.1

Table 1: Token ablation results. S =Semantic, A =Acoustic, T =Text.

\times

indicates the model lacks the capability to perform the task.

Panel of Figure 4 — Figure 4: IsoFLOP analysis for discrete audio modeling. (a) and (b) show the loss landscape across model sizes and token counts for each compute budget. (c) shows the fitted scaling laws with extrapolation, revealing that optimal data $D^{*}$ scales faster than optimal model size.

	Speech (Semantic)		Speech (Acoustic)	Text (Knowledge)			Cross-Modal
Model	sBLIMP $\uparrow$	sWUGGY $\uparrow$	Salmon $\uparrow$	tBLIMP $\uparrow$	tWUGGY $\uparrow$	HellaS $\uparrow$	ASR $\downarrow$	TTS ${}_{\text{WER}}$ $\downarrow$	TTS ${}_{\text{SIM}}$ $\uparrow$
TWIST-7B	59.0	73.9	61.6	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
SpiritLM-base-7B	58.3	69.0	57.2	73.3	80.3	–	$\sim$ 22 †	$\sim$ 40 †	$\sim$ 0.05 ‡
SpiritLM-Expr-7B	54.2	65.0	67.1	73.6	75.8	–	$\sim$ 38 †	$\sim$ 50 †	$\sim$ 0.10 ‡
Llama-Mimi-1.3B	54.3	68.7	73.6	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
Llama-Mimi-8B	55.1	68.8	73.2	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$

Table 2: Results comparing SODA against existing Speech Language Models. SODA is the only model with capabilities across all skills, serving as a unified backbone.

\times

indicates lacks the capability; – indicates results not reported due to HellaSwag contamination known in Llama2 pre-training which SpiritLM is based on ( Touvron et al., 2023 ) . † SpiritLM uses 10-shot evaluation on LibriSpeech-clean for ASR/TTS whereas ours is 0-shot with TTS on seed-tts-eval; ‡ TTS-SIM reproduced—semantic-only tokens cannot preserve voice.

Init.	Pre-training	es	fr	de	other	SIM $\uparrow$
		X $\rightarrow$ En (BLEU $\uparrow$ )
Scratch	None	3.5	4.5	3.5	2.7	0.349
Qwen3	Text	3.0	4.5	3.0	2.1	0.375
SODA	Text + Audio	13.5	14.7	9.4	4.3	0.466
SODA-P	Text + Audio	20.9	21.1	14.6	4.5	0.469

Table 3: Speech-to-speech (X

\rightarrow

En) translation on CVSS-T. SODA-Prelim, despite weaker overall recipe and English performance, was pre-trained on multilingual Yodas (en, es, fr, de, th, ar, hi, zh).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Split	Type	Hours	$\sim$ Tokens	Source
Yodas ( Li et al., 2023 )	English	Speech	164K	$\sim$ 131B †	YouTube (100+ langs)
Emilia ( He et al., 2024 )	English+Yodas-English	Speech	139K	$\sim$ 110B †	YouTube (6 langs)
MLS ( Pratap et al., 2020 )	English	Speech	44.5K	$\sim$ 35B †	LibriVox audiobooks
Nemotron-CC ( Su et al., 2025 )	HQ-Actual	Text	–	$\sim$ 220B*	Web (CC filtered)

Appendix

Table 4: Training data statistics. Hours and tokens are for English subsets used in our experiments. Token counts assume 100 tokens/sec for audio (8 Mimi codebooks at 12.5 Hz) and standard BPE tokenization for text. *Estimated from disk size.

\dagger

Token counts include both audio-first and text-first formats (i.e., unrepeated token counts would be half of this value).

SODA-prelim
	Semantic		Acoustic	Text		ASR	TTS
Data	sWUGGY $\uparrow$	sBLIMP $\uparrow$	Salmon $\uparrow$	tWUGGY $\uparrow$	tBLIMP $\uparrow$	WER $\downarrow$	WER $\downarrow$	SIM $\uparrow$
Yodas (multilingual)	57.8	50.9	69.4	71.3	69.0	22.0	9.2	0.516
Annealing from SODA-prelim (at final 20% cooldown steps)
MLS (English)	56.8	50.6	70.3	69.1	72.2	92.6	35.7	0.366
Emilia (English)	58.5	51.0	69.8	58.5	70.5	7.8	6.1	0.557

Appendix

Table 5: Speech data selection results. Top: SODA-prelim baseline (full training). Middle: annealing experiments comparing different data sources. Bottom: small-scale validation runs (150M parameters, 10B tokens). MLS performs poorly on cross-modal tasks despite being a curated audiobook corpus, while Yodas and Emilia show complementary strengths. ASR = LibriSpeech-clean-test, TTS=seed-tts-eval.

Figure 5: Full results of text-injection ratio sweep (0%–90% Nemotron text data). Colors indicate the trend as text percentage increases: Red (Cross-Modal skills: ASR, TTS) shows degradation in performance as text ratio increases beyond 5%. Orange (Semantic and Acoustic understanding: sWUGGY, sBLIMP, Salmon) indicates little variation across text ratios, suggesting these capabilities are robust to text inclusion. Green (Text Knowledge tasks: tWUGGY, tBLIMP, HellaSwag) shows monotonic improvement as text percentage increases, with notable gains from 0% to 2.5%.

NLL	ASR-0	ASR-2	TTS-W	TTS-S	Sal.	sWUG	sBLI	tWUG	tBLI	Hella	Avg-Sp	Avg-Tx
NLL_1	0.959	0.967	0.955	0.989	0.841	0.941	0.614	0.879	0.824	0.887	0.895	0.864
NLL_2	0.956	0.962	0.953	0.986	0.842	0.936	0.621	0.874	0.821	0.887	0.894	0.861
NLL_3	0.954	0.959	0.952	0.985	0.842	0.929	0.620	0.866	0.819	0.884	0.892	0.856
NLL_4	0.959	0.965	0.952	0.987	0.845	0.937	0.623	0.871	0.819	0.886	0.896	0.859
NLL_5	0.962	0.961	0.960	0.986	0.820	0.944	0.602	0.899	0.827	0.904	0.891	0.877
NLL_6	0.903	0.908	0.933	0.949	0.802	0.892	0.552	0.891	0.873	0.917	0.848	0.894

Appendix

Table 6: Spearman correlation (absolute values) between NLL metrics and downstream tasks across 64 IsoFLOP models. Avg_Speech averages 7 speech metrics; Avg_Text averages 3 text metrics. Option 1 achieves the best balance between speech and text correlations.

Figure 6: Validation NLL (audio-first) versus downstream task performance (Part 1). Each subplot shows one metric across 64 models from the IsoFLOP study, with Spearman

\rho

reported. Cross-modal metrics (ASR, TTS) show strong correlation with NLL.

Figure 7: Validation NLL (audio-first) versus downstream task performance (Part 2). Speech semantic and acoustic metrics show continued improvement without saturation. Text knowledge tasks (tWUGGY, HellaSwag) show emergence patterns with accelerated improvement at lower NLL values.

Model Size	$D^{*}$	Over-training Factor
135M	0.53B	940 $\times$
600M	5.55B	90 $\times$ ( $\sim$ Llama3’s)
1.7B	28.6B	18 $\times$ ( $\sim$ Llama2’s)
4B	110B	4.5 $\times$

Appendix

Table 7: Training regime for SODA models.

D^{*}

is the compute-optimal token count predicted by our scaling laws (§ 5.2 ). Over-training factor indicates how many times more tokens we train on compared to optimal.

Figure 8: Validation loss vs. compute for SODA models. Grey squares show compute-optimal models from § 5 . We fit a power-law regression

L^{*}=a\cdot C^{b}

on these seven points and extrapolate to higher compute (grey line). Colored circles show final SODA runs.

Figure 9: Loss vs. compute at different overtraining ratios

K

using the fitted parametric equation. Grey points show IsoFLOP sweep data. Colored curves show predicted loss at

K=1

(optimal),

K=5

K=20

, and

K=100

. Colored circles show actual SODA models (135M, 600M, 1.7B, 4B), which achieve lower losses than predicted. The discrepancy shows that SODA models outperform the parametric prediction, suggesting favorable scaling dynamics or lower irreducible loss at larger scales.

Panel of Figure 10 — Figure 10: Training loss curve (extracted from W&B) comparison between Cold-Start and Warm-Start at 600M and 1.7B scales. Warm-Start exhibits instability with many small spikes throughout training for both model sizes. The 600M Warm-Start run shows a particularly large spike around 135K steps. We note that with our limited compute budget, we are able to train only one run for each model configuration. However, we note that better regularization or different hyperparameters could stabilize the training process of Warm-Start.

Figure 11: Warm-Start vs. Cold-Start training comparison at 600M and 1.7B scales. Top row : Training dynamics (NLL) and cross-modal skills (ASR, TTS). Cold-Start achieves lower ASR-WER throughout training, while TTS converges similarly. The NLL plot highlights a loss spike at 135K steps for the 600M Warm-Start run. Bottom row : Understanding tasks. Salmon (acoustic) and sWUGGY (semantic) show similar trajectories regardless of initialization, suggesting these are learned fresh. tWUGGY (text knowledge) shows Warm-Start maintaining a persistent advantage throughout training, with Cold-Start never catching up.

	Speech (Semantic)		Speech (Acoustic)	Text (Knowledge)			Cross-Modal
Model	sBLIMP $\uparrow$	sWUGGY $\uparrow$	Salmon $\uparrow$	tBLIMP $\uparrow$	tWUGGY $\uparrow$	HellaS $\uparrow$	ASR $\downarrow$	TTS ${}_{\text{WER}}$ $\downarrow$	TTS ${}_{\text{SIM}}$ $\uparrow$
SODA-600M (cold-start)	51.2	58.9	70.1	70.7	73.1	35.8	10.2	7.6	0.555
SODA-600M (warm-start)	51.1	59.1	70.7	70.8	77.0	36.3	14.6	6.6	0.559
SODA-1.7B (cold-start)	51.4	60.6	70.6	70.3	74.7	44.5	7.0	6.9	0.560
SODA-1.7B (warm-start)	51.8	60.3	70.3	71.0	79.2	47.1	17.3	6.8	0.557

Appendix

Table 8: Warm-Start vs. Cold-Start training comparison at 600M and 1.7B scales. Final evaluation results are shown.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 287 papers

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
Apr 13, 2026Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar +15Audio Understanding Audio-Language Models
NVIDIA, USA · University of Maryland, USA
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Jun 1, 2026Hui Li, Yangfan Gao, Junlin Shang +4Audio Representation Learning Neural Audio Codecs
Fudan University
S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
Apr 27, 2026Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau +2Model Compression Audio Representation Learning
LTCI, T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France

Apr 13, 2026J/K move · Enter open · S save

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze+14

NVIDIA, USA · University of Maryland, USA

Audio Understanding Audio-Language Models Audio Reasoning Temporal Audio Grounding

Open paper AskPDF arXiv

We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reasoning over speech, environmental sounds and music. Compared to Audio Flamingo 3, AF-Next introduces: (i) a stronger foundational audio-language model that significantly improves accuracy across diverse audio understanding tasks; (ii) scalable strategies for constructing large-scale audio understanding and reasoning data beyond existing academic benchmarks; (iii) support for long and complex audio inputs up to 30 minutes; and (iv) Temporal Audio Chain-of-Thought, a new reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio, enabling fine-grained temporal alignment and improved interpretability. To enable these capabilities, we first conduct a systematic analysis of Audio Flamingo 3 to identify key gaps in audio understanding and reasoning. We then curate and scale new large-scale datasets totaling over 1 million hours to address these limitations and expand the existing AudioSkills-XL, LongAudio-XL, AF-Think and AF-Chat datasets. AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training and post-training stages. Extensive experiments across 20 audio understanding and reasoning benchmarks, including challenging long-audio tasks, show that AF-Next outperforms similarly sized open models by large margins and remains highly competitive with and sometimes surpasses, much larger open-weight and closed models. Beyond benchmark performance, AF-Next exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. In addition to all data, code and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think and AF-Next-Captioner.

Abstract

Audio-Language Models

Explore similar work

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models