Quantifying the Generation Modality Gap in Speech-Text Language Models

Ju-Chieh Chou; Jiawei Zhou; Karen Livescu

cs.CLSep 13, 2026

Quantifying the Generation Modality Gap in Speech-Text Language Models

Authors: Ju-Chieh Chou, Jiawei Zhou, Karen Livescu

Organizations: TTI-Chicago · Stony Brook University

Abstract

Pure speech language models often lag behind text and speech-text language models in generating coherent content, but this gap is difficult to quantify because speech and text systems are typically evaluated with different metrics and trained on different data. We study the speech-text modality gap in a family of spoken language models, based on flow matching for continuous acoustic feature generation. We construct a unified generation-based evaluation suite that compares speech-only, text-only, and speech-text language models trained on matched data distributions and evaluated in matched generation settings. We evaluate generated continuations along multiple dimensions: semantic coherence, measured by transcribing generated speech and scoring it with a reference language model; local phonetic structure, measured by phone n-gram distributional statistics; speaker consistency and acoustic quality; and emotion-based distributional metrics. Across datasets, we find that joint speech-text modeling substantially improves semantic coherence. However, the improvement is not uniform across metrics: phone-level metrics change only modestly, speaker similarity and predicted quality are lower for speech-text continuations, while emotion-based distributional metrics improve. Compared with larger-scale speech-only models, our speech-text model closes much of the scaling gap in transcript-based semantic coherence, suggesting that text provides an efficient semantic training signal for spoken language modeling.

Figures & tables

Fig. 1 : Speech-text LM architecture. The transformer processes text and speech streams and produces a context vector

h_{m}

at the

m^{\textrm{th}}

time step, which is used by the text and speech prediction heads to generate future text and speech output. The speech-only and text-only models use the same backbone with only the corresponding stream and output head. The speech head includes both a discrete token head (eq. 2 ) and a conditional flow matching (CFM) head (eq. 4 ).

Fig. 2 : GenPPL across training sets and prompts. Each panel corresponds to a training set. X-axis labels indicate the prompt source;

\emptyset

is the unprompted setting. We compare text-only generation, the internal text stream of the speech-text model (speech-text inner), ASR transcripts of speech-text continuations (speech-text ASR), and ASR transcripts of speech-only continuations.

Fig. 3 : Left: GenPPL by training dataset, aggregated over all prompt sources and the unprompted setting. Error bars are 95% confidence intervals estimated by bootstrap resampling. Middle: GenPPL, aggregated over the training sets, vs. Llama-Mimi (1.3B and 8B). The x-axis labels are the #params excluding embedding layers (as in [ 39 ] ) and the training set size. Right: Training dynamics of genPPL when training on Emilia 10k.

Fig. 4 : 5-gram pJSD results. For unprompted generation (

\emptyset

), we compare to the prompt set that matches the training set. The legend follows Fig. 3

Fig. 5 : Speaker similarity from the speaker verification model. The legend follows Fig. 3 .

Fig. 6 : Predicted MOS from the UTMOS model [ 38 ] . The legend follows Fig. 3 .

Fig. 7 : FSD using utterance-level embeddings from emotion2vec-base [ 26 ] . The legend follows Fig. 3 .

Fig. 8 : KL divergence between emotion2vec-large emotion-label posteriors for full reference and generated utterances. The legend follows Fig. 3 .

Model	Training data	Variant	Params.	Prompts	genPPL $\downarrow$	pJSD $\downarrow$	SpkSim $\uparrow$	UTMOS $\uparrow$
Flow-based	C4	Speech-only	456M	C4	228.8	0.47	0.70	3.41
				Emilia	265.1	0.52	0.36	3.28
				LibriSpeech	291.6	0.56	0.42	3.35
		Speech-text	456M	C4	49.9	0.47	0.62	3.09
				Emilia	51.8	0.51	0.26	2.90
				LibriSpeech	63.4	0.53	0.32	3.02

TABLE I : Detailed metrics for each model across training datasets and prompt sets. Parameter counts exclude the input embedding layer. Non-empty prompts for each datasets: C4 (4707 / 7500), Emilia (4733 / 7500) and LibriSpeech (4710 / 7500).

Model	Training data	Variant	Params.	genPPL $\downarrow$	pJSD $\downarrow$	SpkSim $\uparrow$	UTMOS $\uparrow$
Base	Emilia10k	Speech-only	456M	182.1	0.51	0.47	3.19
Base	Emilia10k	Speech-text	456M	52.7	0.51	0.41	3.07
Base + data scale	Emilia45k	Speech-only	456M	149.3	0.51	0.50	3.25
Base + data scale	Emilia45k	Speech-text	456M	51.6	0.50	0.45	3.22
Base + model scale	Emilia10k	Speech-only	1.16B	170.2	0.51	0.48	3.19
Base + model scale	Emilia10k	Speech-text	1.16B	54.5	0.51	0.41	3.05

TABLE II : Scaling results on Emilia, including data scaling, model scaling, and a matched Llama-Mimi reproduction to isolate architectural differences. Metrics are computed on the common subset of 4,644/7,500 non-empty continuations. Parameter counts exclude the input embedding layer.

Prompt set	Dataset	Mode	OLMo 7B	Ministral 8B	Gemma 7B	Qwen2.5 7B	Llama 3.1 8B
C4	C4	Speech–text ASR	48.3	40.5	42.9	43.1	45.0
C4	C4	Speech–text inner	54.0	42.4	46.2	47.2	48.6
C4	C4	Text-only	31.4	25.2	26.3	26.8	27.3
C4	C4	Speech-only	230.4	202.7	216.8	221.7	223.9
C4	Emilia10k	Speech–text ASR	78.4	77.7	82.5	85.0	90.7
C4	Emilia10k	Speech–text inner	64.4	62.3	65.4	68.4	73.8

TABLE III : GenPPL computed using five different reference LMs, across all model and evaluation settings.

Oracle LM	OLMo 7B	Ministral 8B	Gemma 7B	Qwen2.5 7B	Llama 3.1 8B
OLMo 7B	1.00	0.99	0.99	0.99	0.99
Ministral 8B	0.99	1.00	0.99	1.00	1.00
Gemma 7B	0.99	0.99	1.00	0.99	0.99
Qwen2.5 7B	0.99	1.00	0.99	1.00	1.00
Llama 3.1 8B	0.99	1.00	0.99	1.00	1.00

TABLE IV : Pairwise Spearman rank correlations between reference LMs over the same 36 genPPL values.

Explore similar work

Why We Need Speech to Evaluate Speech Translation
May 27, 2026Maike Züfle, Danni Liu, Vilém Zouhar +1Speech Translation Speaker
Karlsruhe Institute of Technology · ETH Zurich
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
May 7, 2026Wenqian Cui, Xiao-Hui Li, Daxin Tan +2Speech Language Models Modality Gap
The Chinese University of Hong Kong · Huawei Technologies
MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching
Aug 12, 2026Xingwei Sun, Heinrich Dinkel, Gang Li +7Text-To-Audio Seed-Tts-Eval Benchmark
MiLM Plus, Xiaomi Inc., Beijing, China · X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China

May 27, 2026, cs.CLJ/K move · Enter open · S save

Why We Need Speech to Evaluate Speech Translation

Maike Züfle, Danni Liu, Vilém Zouhar, Jan Niehues

Karlsruhe Institute of Technology · ETH Zurich

Speech Translation Speaker

Open paper AskPDF arXiv

Speech translation models are increasingly capable of preserving speech-specific information (e.g., speaker gender, prosody, and emphasis), yet evaluation metrics remain blind to such phenomena. We meta-evaluate both text- and speech-based quality estimation metrics on two contrastive datasets targeting gender agreement and prosody, and find that both fall short, even when given direct access to the speech signal. We then train SpeechCOMET, a family of quality estimation models with speech encoders, and evaluate a state-of-the-art SpeechLLM as a judge. Both match or exceed text-based COMET on standard quality estimation, but neither consistently assesses speech-specific phenomena. We identify three causes: (1) speech-specific features are not reliably preserved in current encoders, (2) models tend to ignore the speech source signal, and (3) quality estimation training data contains too few relevant examples. We release all models and code, and argue that progress requires dedicated speech-specific training data and models that genuinely condition on speech.

Abstract

Explore similar work

Why We Need Speech to Evaluate Speech Translation

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching