Quantifying the Generation Modality Gap in Speech-Text Language Models

Ju-Chieh Chou; Jiawei Zhou; Karen Livescu

cs.CLSep 13, 2026

Quantifying the Generation Modality Gap in Speech-Text Language Models

Authors: Ju-Chieh Chou, Jiawei Zhou, Karen Livescu

Organizations: TTI-Chicago · Stony Brook University

Abstract

Pure speech language models often lag behind text and speech-text language models in generating coherent content, but this gap is difficult to quantify because speech and text systems are typically evaluated with different metrics and trained on different data. We study the speech-text modality gap in a family of spoken language models, based on flow matching for continuous acoustic feature generation. We construct a unified generation-based evaluation suite that compares speech-only, text-only, and speech-text language models trained on matched data distributions and evaluated in matched generation settings. We evaluate generated continuations along multiple dimensions: semantic coherence, measured by transcribing generated speech and scoring it with a reference language model; local phonetic structure, measured by phone n-gram distributional statistics; speaker consistency and acoustic quality; and emotion-based distributional metrics. Across datasets, we find that joint speech-text modeling substantially improves semantic coherence. However, the improvement is not uniform across metrics: phone-level metrics change only modestly, speaker similarity and predicted quality are lower for speech-text continuations, while emotion-based distributional metrics improve. Compared with larger-scale speech-only models, our speech-text model closes much of the scaling gap in transcript-based semantic coherence, suggesting that text provides an efficient semantic training signal for spoken language modeling.

Figures & tables

Fig. 1 : Speech-text LM architecture. The transformer processes text and speech streams and produces a context vector

h_{m}

at the

m^{\textrm{th}}

time step, which is used by the text and speech prediction heads to generate future text and speech output. The speech-only and text-only models use the same backbone with only the corresponding stream and output head. The speech head includes both a discrete token head (eq. 2 ) and a conditional flow matching (CFM) head (eq. 4 ).

Fig. 2 : GenPPL across training sets and prompts. Each panel corresponds to a training set. X-axis labels indicate the prompt source;

\emptyset

is the unprompted setting. We compare text-only generation, the internal text stream of the speech-text model (speech-text inner), ASR transcripts of speech-text continuations (speech-text ASR), and ASR transcripts of speech-only continuations.

Fig. 3 : Left: GenPPL by training dataset, aggregated over all prompt sources and the unprompted setting. Error bars are 95% confidence intervals estimated by bootstrap resampling. Middle: GenPPL, aggregated over the training sets, vs. Llama-Mimi (1.3B and 8B). The x-axis labels are the #params excluding embedding layers (as in [ 39 ] ) and the training set size. Right: Training dynamics of genPPL when training on Emilia 10k.

Fig. 4 : 5-gram pJSD results. For unprompted generation (

\emptyset

), we compare to the prompt set that matches the training set. The legend follows Fig. 3

Fig. 5 : Speaker similarity from the speaker verification model. The legend follows Fig. 3 .

Fig. 6 : Predicted MOS from the UTMOS model [ 38 ] . The legend follows Fig. 3 .

Fig. 7 : FSD using utterance-level embeddings from emotion2vec-base [ 26 ] . The legend follows Fig. 3 .

Fig. 8 : KL divergence between emotion2vec-large emotion-label posteriors for full reference and generated utterances. The legend follows Fig. 3 .

Model	Training data	Variant	Params.	Prompts	genPPL $\downarrow$	pJSD $\downarrow$	SpkSim $\uparrow$	UTMOS $\uparrow$
Flow-based	C4	Speech-only	456M	C4	228.8	0.47	0.70	3.41
				Emilia	265.1	0.52	0.36	3.28
				LibriSpeech	291.6	0.56	0.42	3.35
		Speech-text	456M	C4	49.9	0.47	0.62	3.09
				Emilia	51.8	0.51	0.26	2.90
				LibriSpeech	63.4	0.53	0.32	3.02

TABLE I : Detailed metrics for each model across training datasets and prompt sets. Parameter counts exclude the input embedding layer. Non-empty prompts for each datasets: C4 (4707 / 7500), Emilia (4733 / 7500) and LibriSpeech (4710 / 7500).

Model	Training data	Variant	Params.	genPPL $\downarrow$	pJSD $\downarrow$	SpkSim $\uparrow$	UTMOS $\uparrow$
Base	Emilia10k	Speech-only	456M	182.1	0.51	0.47	3.19
Base	Emilia10k	Speech-text	456M	52.7	0.51	0.41	3.07
Base + data scale	Emilia45k	Speech-only	456M	149.3	0.51	0.50	3.25
Base + data scale	Emilia45k	Speech-text	456M	51.6	0.50	0.45	3.22
Base + model scale	Emilia10k	Speech-only	1.16B	170.2	0.51	0.48	3.19
Base + model scale	Emilia10k	Speech-text	1.16B	54.5	0.51	0.41	3.05

TABLE II : Scaling results on Emilia, including data scaling, model scaling, and a matched Llama-Mimi reproduction to isolate architectural differences. Metrics are computed on the common subset of 4,644/7,500 non-empty continuations. Parameter counts exclude the input embedding layer.

Prompt set	Dataset	Mode	OLMo 7B	Ministral 8B	Gemma 7B	Qwen2.5 7B	Llama 3.1 8B
C4	C4	Speech–text ASR	48.3	40.5	42.9	43.1	45.0
C4	C4	Speech–text inner	54.0	42.4	46.2	47.2	48.6
C4	C4	Text-only	31.4	25.2	26.3	26.8	27.3
C4	C4	Speech-only	230.4	202.7	216.8	221.7	223.9
C4	Emilia10k	Speech–text ASR	78.4	77.7	82.5	85.0	90.7
C4	Emilia10k	Speech–text inner	64.4	62.3	65.4	68.4	73.8

TABLE III : GenPPL computed using five different reference LMs, across all model and evaluation settings.

Oracle LM	OLMo 7B	Ministral 8B	Gemma 7B	Qwen2.5 7B	Llama 3.1 8B
OLMo 7B	1.00	0.99	0.99	0.99	0.99
Ministral 8B	0.99	1.00	0.99	1.00	1.00
Gemma 7B	0.99	0.99	1.00	0.99	0.99
Qwen2.5 7B	0.99	1.00	0.99	1.00	1.00
Llama 3.1 8B	0.99	1.00	0.99	1.00	1.00

TABLE IV : Pairwise Spearman rank correlations between reference LMs over the same 36 genPPL values.

Explore similar work

May 27, 2026cs.CL

Why We Need Speech to Evaluate Speech Translation

Speech translation models are increasingly capable of preserving speech-specific information (e.g., speaker gender, prosody, and emphasis), yet evaluation metrics remain blind to such phenomena. We meta-evaluate both text- and speech-based quality estimation metrics on two contrastive datasets targeting gender agreement and prosody, and find that both fall short, even when given direct access to the speech signal. We then train SpeechCOMET, a family of quality estimation models with speech encoders, and evaluate a state-of-the-art SpeechLLM as a judge. Both match or exceed text-based COMET on standard quality estimation, but neither consistently assesses speech-specific phenomena. We identify three causes: (1) speech-specific features are not reliably preserved in current encoders, (2) models tend to ignore the speech source signal, and (3) quality estimation training data contains too few relevant examples. We release all models and code, and argue that progress requires dedicated speech-specific training data and models that genuinely condition on speech.

Maike Züfle, Danni Liu, Vilém Zouhar +1

Karlsruhe Institute of Technology · ETH Zurich

Speech Translation Speaker

Ask PDF

May 7, 2026cs.CL

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.

Wenqian Cui, Xiao-Hui Li, Daxin Tan +2

The Chinese University of Hong Kong · Huawei Technologies

Speech Language Models Modality Gap

Ask PDF

Aug 12, 2026eess.AS

MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.

Xingwei Sun, Heinrich Dinkel, Gang Li +7

MiLM Plus, Xiaomi Inc., Beijing, China · X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China

Text-To-Audio Seed-Tts-Eval Benchmark Character Error Rate Scene Understanding

Ask PDF