Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

Ming-Hao Hsu; Xiaohai Tian; Jun Zhang; Lu Lu; Yuxuan Wang; Zhizheng Wu

cs.CLMar 2, 2026

Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

Authors: Ming-Hao Hsu, Xiaohai Tian, Jun Zhang, Lu Lu, Yuxuan Wang, Zhizheng Wu

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen · ByteDance

Abstract

End-to-end speech large language models (LLMs) often answer the same question less accurately from speech than from text, even when the spoken content matches the text prompt. Existing analyses study this modality gap through the geometry of representations and measure how closely speech representations align with text. Geometric alignment gives the gap a measurable form, but it does not show whether the model uses aligned representations in the same way. This work connects representation geometry with answer formation and asks whether geometric alignment implies functional equivalence. We compare cross-layer Centered Kernel Alignment (CKA) with layer-wise answer margins for four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH. In Qwen2.5-Omni and MiniCPM-o, speech and text representations align in middle layers while their late-layer answer margins still differ, so mid-layer alignment does not imply functional equivalence. Qwen2-Audio instead shows weak alignment, which indicates a different failure mode. An attention intervention at fixed input and token count identifies a causal effect of late attention allocation on answer-code separation, a quantity distinct from answer correctness. Compressing the speech signal or its token sequence repairs a subset of the errors that occur only with speech.

Figures & tables

Figure 1: Geometric alignment without functional equivalence. The same question enters a speech LLM as text and as speech. Speech–text CKA is high in middle layers, yet the answer margins of the two inputs separate in late layers. Curves use Qwen2.5-Omni on BBH, with CKA from Section A.2 and margins averaged over the 233 items answered correctly only from text.

Figure 2: The speech–text modality gap. In the benchmark overview, speech-to-text (S2T) accuracy is below matched text-to-text (T2T) accuracy for all four speech LLMs on both SpeechMMLU and VoiceBench BBH.

Figure 3: Speech and text align in a broad middle-layer band. Each heatmap shows cross-layer CKA between S2T layers on the y-axis and T2T layers on the x-axis after dynamic time warping (DTW) token alignment, with brighter colors denoting higher CKA. The red path marks the best-matching text layer for each speech layer, and the dashed diagonal marks equal layer indices.

Setting	SpMMLU	$\Delta$	BBH	$\Delta$
Gold T2T	73.49	+3.25	72.7	+12.8
Direct S2T	70.24	base	59.9	base
ASR-to-LLM	59.23	−11.0	67.4	+7.5

Table 1: Behavioral gap for Qwen2.5-Omni. Subtable (a) gives benchmark-overview accuracy with deltas in points relative to direct S2T, where SpMMLU denotes SpeechMMLU. Subtable (b) gives the BBH categories Hyperbaton, Navigate, Sports, and Web-of-lies in the 1,024-token diagnostic, where the gap is T2T minus S2T.

Figure 4: Answer margins separate in late layers. Curves show the mean correct-minus-competing candidate probability for Qwen2.5-Omni on 1,000 BBH items, grouped by free-generation correctness, and shading marks the late window of layers 23–28. In this window, the text margin is larger for text-only-correct items and the speech margin is larger for speech-only-correct items.

Model	Benchmark	$\Delta A$	$\Delta W_{.95}$
Qwen2.5-Omni	SpeechMMLU	.0952 [.0867, .1034]	7.56 [6.88, 8.22]
Qwen2.5-Omni	BBH	.0981 [.0925, .1038]	5.64 [5.07, 6.21]
MiniCPM-o	SpeechMMLU	.1343 [.1250, .1438]	5.17 [4.52, 5.80]
MiniCPM-o	BBH	.0916 [.0840, .0993]	5.47 [4.98, 5.95]

Table 2: Controls for the two central findings. Subtable (a) gives the middle-layer CKA lift

A

and band width

W_{.95}

without pooling, each minus its position-permutation baseline, with 95% bootstrap intervals ( Section A.2 ). Subtable (b) gives the mean late T2T-minus-S2T margin of Qwen2.5-Omni on BBH for the two discordant groups in three margin coordinates, with the crossover contrast and its 95% interval.

Figure 5: Speech attention is more dispersed than text attention. Each column shows a selected decision-position attention distribution for Qwen2.5-Omni on SpeechMMLU, labeled by subject, item, layer (L), and head (H). The speech row uses a ten-times smaller vertical scale, and dashed lines mark the sorted-token fraction that covers 90% of the mass.

Figure 6: Sharpening late speech attention increases answer-code separation. Points show the mean change in

|z_{X}-z_{Y}|

for Qwen2.5-Omni on 1,000 BBH items with 95% bootstrap intervals, under two answer-to-code mappings and two rule orders. The input and token count are fixed, and the gap measures separation irrespective of which code is correct.

Model	SpeechMMLU gap	BBH gap	Mid-band	Mode
Qwen2.5-Omni	+3.25	+12.8	clear	Decision Instability
MiniCPM-o	+7.02	+7.7	clear	Decision Instability
LLaMA-Omni	$\approx$ +58.8	$\approx$ +12.1	broad	Intermediate
Qwen2-Audio	+43.07	+2.2	absent	Representation Failure

Table 3: Two failure modes across models. Gaps are T2T-minus-S2T accuracy points in the benchmark overview, approximate for LLaMA-Omni. Mid-band describes the middle-layer CKA band.

		Overview protocol		Shared-instruction protocol
Intervention	Target	Gap repaired	$\Delta$ Acc.	Gap repaired	$\Delta$ Acc.
Tempo 1.2×	audio duration	61/178 (34.3%)	+2.2	113/233 (48.5%)	+2.2
Silence trim	pauses	68/178 (38.2%)	+2.0	104/233 (44.6%)	+1.3
Token merging	speech tokens	26/178 (14.6%)	+0.5	99/233 (42.5%)	−1.5

Table 4: Compression repairs a subset of speech-only errors on BBH. Gap repaired counts corrected answers among the text-correct, speech-wrong items of Qwen2.5-Omni under each protocol, and

\Delta

Acc. is the aggregate accuracy change in points.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Cross-layer best-match sensitivity to DTW settings. Qwen2.5-Omni, 100 items per benchmark. The top row reports the

R^{2}

of a linear fit to matched text-layer indices across speech layers, and the bottom row the fraction of adjacent speech-layer pairs mapped to the same text layer. Open rings mark the width-64 band and coincide with the unbanded values. Pooling-free, null-adjusted results appear in Section A.2 .

Model	Benchmark	$\Delta A$	$\Delta W_{.95}$	Cross-item $(\Delta A,\Delta W_{.95})$
Qwen2.5-Omni	SpeechMMLU	.0952 [.0867, .1034]	7.56 [6.88, 8.22]	(.0299, − 1.37)
Qwen2.5-Omni	BBH	.0981 [.0925, .1038]	5.64 [5.07, 6.21]	(.0081, −0.72)
MiniCPM-o	SpeechMMLU	.1343 [.1250, .1438]	5.17 [4.52, 5.80]	(.0521, −0.46)
MiniCPM-o	BBH	.0916 [.0840, .0993]	5.47 [4.98, 5.95]	(−.0031, −3.30)

Appendix

Table 5: Native-length CKA controls. The first two endpoints subtract the within-item mean position-permutation endpoint and report item means with 95% bootstrap intervals. The last column gives the alignment-free matched-minus-wrong-item differences in

A

and

W_{.95}

, respectively, without an inferential interval.

Figure 8: Best-match paths and phase windows from the CKA heatmaps. Black lines connect each speech layer to its best-matching text layer, point colors give the peak CKA in that speech-layer row, and the dashed diagonal marks equal layer indices. Background shading marks the three phase windows, whose speech-layer ranges appear above each panel and are listed in Section A.2 . The late plateau describes repeated best-matching text layers, with functional behavior tested separately by answer readouts.

Figure 9: Signed-logit counterpart of the probability replication. The items, readout and groups match Figure 4 . Curves show mean correct-minus-competing logits, with the fixed late window shaded.

Model	Benchmark	S2T acc.	T2T acc.	Unparsed S2T / T2T	Group sizes
Qwen2.5-Omni	SpeechMMLU	66.7	72.3	0 / 4	183 / 34 / 17 / 66
Qwen2.5-Omni	BBH	62.0	69.7	–	147 / 62 / 39 / 52
MiniCPM-o	SpeechMMLU	50.0	57.3	64 / 73	101 / 71 / 49 / 79
MiniCPM-o	BBH	56.7	50.0	51 / 98	96 / 54 / 74 / 76
Qwen2-Audio	SpeechMMLU	33.3	55.3	10 / 8	70 / 96 / 30 / 104
Qwen2-Audio	BBH	49.0	42.3	26 / 81	78 / 49 / 69 / 104

Appendix

Table 6: Free-generation correctness groups of the 300 items per benchmark. Accuracy counts unparsed answers as wrong. Group sizes are both correct, text only correct, speech only correct, and both wrong. Qwen2.5-Omni on BBH uses the groups of Section D.3 , whose unparsed counts are reported there.

Figure 10: Qwen2.5-Omni logit margin by correctness group. Rows are SpeechMMLU and BBH, and columns are the free-generation groups with their sizes. Curves show the mean candidate-readout margin of the group. The BBH groups are those of the 1,000-item analysis in Figure 4 restricted to 300 items. In the text-only-correct group the late-layer margin (mean of the last six coordinates) is +0.63 for text and −1.40 for speech on SpeechMMLU, and +1.58 and −0.46 on BBH.

Figure 11: Layer-wise projected-logit entropy of Qwen2.5-Omni. Curves show the mean over 300 items per benchmark. Speech and text have similar entropy trajectories, with mean absolute differences of 0.67 nats on SpeechMMLU and 0.76 nats on BBH.

Figure 12: Layer-wise subject probe of Qwen2.5-Omni. Logistic-regression (linear) and one-hidden-layer MLP probes on the mean hidden state of the evidence tokens predict the SpeechMMLU subject of the 300 items (34 subjects, chance 2.9%), with 5-fold cross-validated accuracy. Linear speech accuracy rises from 0.13 at the embedding output to 0.71 at layer 10 and text accuracy from 0.48 to 0.70, and both stay between 0.6 and 0.7 afterwards; fold standard deviations reach 0.10. These curves describe probe performance rather than generated-answer correctness.

Figure 13: Cosine similarity between speech and text layer-wise updates in Qwen2.5-Omni. An update is the difference between consecutive hidden states after the fixed base-layer-0 token alignment, averaged over 100 items per benchmark. For every speech layer the largest cosine lies at the text layer with the same index. The mean diagonal cosine is 0.33 on SpeechMMLU and 0.30 on BBH, against 0.03 off the diagonal.

Layers	Items	Median cov. ratio	Median $\Delta$ Cov .90	Q90 $\Delta$ Cov .90	Pairs $\Delta$ Cov $>$ 0	Q90 $\Delta$ Ent.
7–21	100	2.20×	+16.0	+83.5	78.4%	+0.037
23–28	100	3.74×	+48.0	+102.5	86.1%	+0.073

Appendix

Table 7: All-head attention coverage audit for Qwen2.5-Omni on SpeechMMLU. We summarize all head–layer pairs within each item, then report the across-item medians of the within-item medians and 90th percentiles (Q90). The positive-pair fraction is averaged over items.

\Delta

denotes S2T minus T2T, and Ent. is attention entropy divided by the log of valid prompt length. Attention is normalized over all valid prompt keys at the final prompt query. These windows differ from the CKA phase windows. Speech usually needs more tokens to cover 90% of the attention mass, while normalized entropy differences vary across heads.

Metric	T2T	S2T	$\Delta$
Entropy norm	0.361	0.657	+0.296
$p_{\max}$	0.638	0.105	−0.533
Top-10 mass	0.933	0.483	−0.450
Cov .90 count	8	101	+93

Appendix

Table 8: Selected-head attention summary for Qwen2.5-Omni on SpeechMMLU.

\Delta

is S2T minus T2T, computed from the displayed values. Entropy norm is normalized attention entropy,

p_{\max}

is the largest attention coefficient, top-10 mass sums the ten largest coefficients, and Cov .90 counts the tokens needed to cover 90% of attention.

Condition	Order	Identity mapping	Swapped mapping
S2T late sharpening	01	+.0291 [.0237, .0346]	+.0076 [.0026, .0127]
	10	+.0332 [.0274, .0389]	+.0129 [.0076, .0184]
S2T middle sharpening	01	−.0039 [−.0091, .0012]	+.0056 [.0019, .0093]
	10	+.0006 [−.0048, .0059]	−.0029 [−.0069, .0013]
S2T late permutation	01	−.0314 [−.0410, −.0219]	−.0004 [−.0084, .0077]
	10	−.0344 [−.0436, −.0253]	+.0054 [−.0032, .0138]

Appendix

Table 9: Change in absolute answer-code separation. Each entry is the mean treatment-minus-baseline change with its 95% interval on the same 1,000 items. Positive values indicate greater separation regardless of which semantic answer is favored. Order 01 states the class-0 rule first, and order 10 states the class-1 rule first.

Setting	T2T	S2T	Use in the paper
Benchmark overview	72.7	59.9	Aggregate model comparison and overview repair changes
1,024-token diagnostic	71.9	59.7	Category breakdown, duration groups, 178-item gap subset
Calibration	—	62.3	Input- and last-layer moment matching
Word repetition	62.4	—	$r$ = 1 reference for the text stress test
Shared instructions, 256 tokens	74.1	61.5	233-item gap subset, paired readouts, complete behavioral transitions

Appendix

Table 10: Behavioral settings and their baseline accuracies. All values are percentages for Qwen2.5-Omni on BBH. A dash denotes a route not used for that comparison. Baseline differences identify separate settings, so their accuracy changes and correctness groups are interpreted within each row.

Condition	Acc. (%)	Unparsed	Repairs	Harms	Retained C/W	Gap repairs	$\Delta$ Acc. [95% CI]
Tempo 1.2×	63.7	20	151	129	486/234	113/233	+2.2 [−1.1, 5.4]
Silence trimming	62.8	22	139	126	489/246	104/233	+1.3 [−1.9, 4.5]
Token merging	60.0	21	139	154	461/246	99/233	−1.5 [−4.8, 1.8]
ASR cascade	71.5	14	219	119	496/166	181/233	+10.0 [6.7, 13.4]

Appendix

Table 11: Complete paired transitions relative to direct speech. Repairs are wrong-to-correct transitions, harms are correct-to-wrong transitions, and retained C/W denotes unchanged correct/wrong outcomes across all 1,000 items. Gap repairs use the 233 items correct only under gold text as their denominator. Accuracy changes are percentage points with category-stratified paired-bootstrap 95% intervals.

Dataset / slice	N	WER	Med. WER	Marker	Sub.	Ins.	Del.
SpeechMMLU	200	3.28%	0.00%	98.4%	1.89%	0.66%	0.72%
BBH overall	200	4.65%	3.45%	99.5%	4.09%	0.26%	0.29%
Hyperbaton	45	2.55%	0.00%	100.0%	1.82%	0.60%	0.13%
Navigate	51	3.09%	0.00%	100.0%	2.76%	0.06%	0.28%
Sports	54	3.22%	2.44%	100.0%	2.48%	0.31%	0.43%
Web-of-lies	50	9.66%	9.84%	98.0%	9.24%	0.11%	0.31%

Appendix

Table 12: ASR transcript audit. We transcribe 200 randomly sampled items per benchmark with Whisper-large-v2. Word error rate (WER), substitution (Sub.), insertion (Ins.), and deletion (Del.) rates are averaged over items. Med. WER is the median item WER. Marker recall measures recognition of “the answer is” on BBH and that phrase plus “question” and “option A/B/C/D” on SpeechMMLU.

Group	Count	Duration (s)	Seconds per word
Both correct	541	22.41 [22.04, 22.77]	0.44 [0.43, 0.44]
Only T2T correct	178	23.58 [22.97, 24.18]	0.45 [0.44, 0.46]
Only S2T correct	56	21.23 [20.32, 22.17]	0.45 [0.43, 0.46]
Both wrong	225	21.54 [20.97, 22.11]	0.42 [0.42, 0.43]

Appendix

Table 13: BBH temporal-load metadata by correctness group. Durations use the 1,000-item, 1,024-token diagnostic setting in Table 10 . Confidence intervals are 95% bootstrap intervals over items, pooled across categories.

$r$	SpeechMMLU	BBH
1	73.9	62.4
2	67.4	56.2
4	57.3	55.4
8	51.2	53.5

Appendix

Table 14: Text-redundancy stress test for Qwen2.5-Omni. Each word of the text prompt is repeated

r

times, and entries are T2T accuracy in percent.

Task	N	Qwen2.5-Omni			MiniCPM-o
		S2T	Text	$\Delta$	S2T	Text	$\Delta$
Continuation writing	110	100.0	22.7	+77.3	99.1	22.7	+76.4
Homophone reasoning	101	97.0	26.7	+70.3	95.0	35.6	+59.4
Intent detection	115	88.7	4.3	+84.3	87.0	25.2	+61.7
Polysemy reasoning	98	99.0	43.9	+55.1	96.9	38.8	+58.2
Deixis resolution	103	94.2	24.3	+69.9	83.5	51.5	+32.0

Appendix

Table 15: MMSU results on selected speech-native tasks. Acoustic cues can be answer-relevant on MMSU. Scores use deterministic parsing, counting unparsed answers as wrong. These selected tasks have large speech advantages, with

\Delta

denoting S2T minus text accuracy in percentage points.

Figure 14: MiniCPM-o layer-wise projected-logit entropy. Curves show the mean over 300 items per benchmark. Speech and text both show a late-layer decrease, with larger differences between the routes than in Qwen2.5-Omni.

Figure 15: MiniCPM-o layer-wise logit margin by correctness group. Rows are SpeechMMLU and BBH. In the text-only-correct group the text margin exceeds the speech margin in late layers, and speech margins are smaller than text margins when both routes are correct.

Figure 16: MiniCPM-o pre-LN and post-LN norms. Left is SpeechMMLU and right is BBH. Dashed curves are the block input before the input layer norm and solid curves after it. Post-LN norms are comparatively stable across modalities despite late pre-LN divergence.

Figure 17: LLaMA-Omni cross-layer CKA heatmaps. Left is SpeechMMLU with DTW base layer 10 and right is BBH with base layer 20. Red lines mark the best-matching text layer of each speech layer and dashed lines the diagonal. Similarity is high across most layer pairs and lowest at the first two text layers.

Figure 18: LLaMA-Omni layer-wise projected-logit entropy. Curves show the mean over 300 items per benchmark. Entropy curves are broadly comparable across modalities.

Figure 19: LLaMA-Omni layer-wise logit margin by correctness group. Rows are SpeechMMLU and BBH. The text-only-correct group exhibits negative speech margins and positive text margins in late layers.

Figure 20: LLaMA-Omni pre-LN and post-LN norms. Left is SpeechMMLU and right is BBH. Post-LN norms are comparatively stable across modalities while pre-LN norms grow with depth.

Figure 21: Qwen2-Audio cross-layer CKA heatmaps. Left is SpeechMMLU with DTW base layer 10 and right is BBH with base layer 20. Red lines mark the best-matching text layer of each speech layer and dashed lines the diagonal. Unlike Qwen2.5-Omni and MiniCPM-o, Qwen2-Audio lacks a clear high-similarity middle-layer band.

Figure 22: Qwen2-Audio candidate-only readout. Per-layer probability of the correct answer option among candidates for S2T and T2T at the decision position after an assistant answer prefix, averaged over 300 items per benchmark. Thin lines mark chance, 0.25 on SpeechMMLU and 0.5 on BBH. Speech does not raise the correct option, especially on SpeechMMLU.

Figure 23: Qwen2-Audio layer-wise projected-logit entropy. Curves show the mean over 300 items per benchmark. Speech entropy is not uniformly higher than text entropy across layers.

Figure 24: Qwen2-Audio layer-wise logit margin by correctness group. Rows are SpeechMMLU and BBH. Margins favor the correct option less strongly under speech than under text on SpeechMMLU.

Figure 25: Qwen2-Audio pre-LN and post-LN norms. Left is SpeechMMLU and right is BBH. Post-LN norms remain comparatively stable across modalities.

Dataset	Item	Question summary
SpeechMMLU	anatomy_1	Which craniofacial condition is associated with a “dished face” profile?
SpeechMMLU	anatomy_11	Which signs follow damage to the third cranial nerve after an orbital injury?
BBH	bbh_hyperbaton_105	Choose the correct ordering of adjectives describing a repulsive, normal-size, green, old-fashioned cardboard surfboard.
BBH	bbh_hyperbaton_111	Choose between “little black driving ship” and “driving little black ship.”
BBH	bbh_web_of_lies_74	Jamey tells the truth. Raymond says Jamey lies. Audrie says Raymond lies. Michaela says Audrie lies. Rashida says Michaela lies. Does Rashida tell the truth?

Appendix

Table 16: Qualitative cases (text correct, speech wrong). We summarize selected questions where T2T answered correctly but S2T failed in the earlier comparison that selected them. The accompanying figure plots the per-layer probability of the correct option, normalized over answer candidates.

Figure 26: Candidate-probability traces for four instances selected as text-correct and speech-wrong. Curves show, for each fixed item, the probability of the correct option after normalization over answer candidates in Qwen2.5-Omni. Only hyperbaton_105 has a higher text probability at the last layer.

Datasets and Audio
Parameter	Value or Method
SpeechMMLU	Dataset-provided paired question_text , answer options, question_audio , and speaker identifier from MiMo-Audio-Eval
VoiceBench BBH	Dataset-provided paired prompt , reference answer, and audio clip for hyperbaton , navigate , sports_understanding , and web_of_lies
T2T Prompt	The paired text field corresponding to the audio item. No ASR transcript is used for gold-text T2T
Hidden State Extraction
Extraction Window	Last user segment, i.e., question turn only

Appendix

Table 17: Summary of implementation hyperparameters.

Asset and source	Stated license or condition
Qwen2.5-Omni-7B	Apache-2.0
MiniCPM-o 2.6	Apache-2.0 for code and weights
Qwen2-Audio-7B-Instruct	Apache-2.0
LLaMA-Omni	Code: Apache-2.0. Model: academic, noncommercial use with citation
Whisper-large-v2	MIT in the OpenAI release, with differing checkpoint-card metadata
MiMo-Audio-Eval toolkit	Apache-2.0

Appendix

Table 18: Provider-stated licenses of existing assets. Links point to the official model cards, dataset cards, or repositories. These are source declarations rather than a complete audit of every underlying asset.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
May 7, 2026Wenqian Cui, Xiao-Hui Li, Daxin Tan +2Audio Understanding Speech Prosody
The Chinese University of Hong Kong · Huawei Technologies
Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
Jun 10, 2026Zhen Ye, Xu Tan, Yiming Li +10Spoken Language Understanding Finite Scalar Quantization
Hong Kong University of Science and Technology, Hong Kong SAR · Tencent, China · Independent Researcher +4
Quantifying the Generation Modality Gap in Speech-Text Language Models
Sep 13, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Generation Evaluation Speech Language Models
TTI-Chicago · Stony Brook University

May 7, 2026J/K move · Enter open · S save

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

Wenqian Cui, Xiao-Hui Li, Daxin Tan, Qiyong Zheng+1

The Chinese University of Hong Kong · Huawei Technologies

Audio Understanding Speech Prosody Speech Language Models

Open paper AskPDF arXiv

Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.

Abstract

Speech Language Models

Explore similar work

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

Quantifying the Generation Modality Gap in Speech-Text Language Models