cs.CLAug 24, 2026
SaveDo Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text
Organizations: Maum AI Inc. · KAIST · Atmanity Inc.
Abstract
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
Figures & tables
Figure 1: Token-wise similarity maps between and on LibriTTS 64 test-clean for BLSP-emo 51 , Qwen2-Audio-Instruct 8 , and our models. Existing SLMs show weak diagonal patterns, whereas our model exhibits a clear diagonal trend by addressing structural difference between speech and text. Non-diagonal activations correspond to identical text tokens appearing at different positions.
| Model | CKA |
| BLSP-emo 51 | 0.3570 |
| Qwen2-Audio-Instruct 8 | 0.4113 |
| DiVA 17 | 0.3992 |
| DeSTA2 30 | 0.4302 |
| Ours | 0.6399 |
Table 1: Centered Kernel Alignment (CKA) metric between speech and text in the LLM input space.
Figure 2: Model architecture. Given a speech input, a frozen speech encoder extracts frame-level features, which are mapped into the LLM input space with a windowed Q-former adapter. To explicitly resolve length mismatch, the adapter dynamically allocates the number of query tokens to match the target text length during training, decoupling length alignment problem from semantic alignment. The model is trained with behavior alignment, together with an token-level internal alignment which encourages fine-grained correspondence between speech and text.
| Benchmark | Model | |||
| AIR-Bench | Chat-speech | |||
| SALMONN | 6.16 | |||
| BLSP | 6.17 | |||
| DeSTA2 | 7.16 | |||
| Qwen2-Audio | 7.18 | |||
| Phi-4-Multimodal | 7.47 | |||
Table 2: Performances on Air-bench chat, SpeechR, MMSU, and Speech-IFeval benchmarks. Best results are highlighted in bold , and second-best results are underlined . For SpeechR, FC, LR, and CoH denote final correctness, logical relevance, and coherence, respectively. For Speech-IFEval, CEQ, CW, and CoT indicate close-ended question, creative writing, and chain of thought, respectively. Models marked with * are closed-source models.
| Air-bench | SpeechR | MMSU | Speech-IFeval | Rel. (%) | |
| Ours(w/o ) | 7.59 | 47.33 | 52.45 | -22.94 | -16.20 |
| Ours(w/o ) | 7.24 | 52.00 | 53.14 | -17.68 | -10.66 |
| Ours(MSE) | 7.66 | 48.46 | 53.63 | -24.51 | -15.59 |
| Ours(InfoNCE) | 7.48 | 48.75 | 53.67 | -23.30 | -15.39 |
| Ours(Cformer) | 5.07 | 36.13 | 39.97 | -29.60 | -48.71 |
| Ours(FQA) | 7.78 | 52.41 | 53.25 | -18.26 | -9.07 |
Table 3: Ablation study across four benchmarks. We report performance on each benchmark and relative changes (%) with respect to the full model, where higher values indicate better performance.
Figure 3: Trade-off between representation similarity and task performance across different values of .
Figure 4: Effect of the number of instructions on model performance.
| Model | AIR-Bench Foundation | AIR-Bench | SpeechR | MMSU | CKA | |
| Linguistic | Speech-specific | Chat-speech | ||||
| Ours (text input) | 70.35 | 44.11 | 8.43 | 58.85 | 34.43 | - |
| Ours (w/o ) | 53.09 | 47.90 | 7.59 | 47.33 | 52.45 | 0.6436 |
| Ours (InfoNCE) | 52.94 | 47.89 | 7.48 | 48.75 | 53.67 | 0.7113 |
| Ours | 53.33 | 50.63 | 7.85 | 52.91 | 53.85 | 0.6399 |
Table 4: Analysis of linguistic and speech-specific performance. The text-input variant performs strongly on tasks primarily requiring linguistic content, but underperforms on tasks that rely more heavily on speech-specific information.
Figure 5: Attention analysis on IEMOCAP across real speech, neutralized cloned speech generated by Qwen3-TTS 18 , and text. Brighter colors indicate higher attention weights from the input representations to the corresponding emotion label. Our model shows clear activations around regions associated with emotional expression in real speech.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Score | Criterion |
| 0 | Useless: ignores the instruction or refuses to respond |
| 1 | Poor: barely related, incomplete, or unhelpful |
| 2 | Weak: partially relevant but lacking clarity or depth |
| 3 | Fair: mostly relevant and informative, but limited in quality |
| 4 | Good: relevant, coherent, and reasonably informative |
| 5 | Excellent: directly follows the instruction and provides clear, useful content |
Table 5: LLM-as-a-judge evaluation scale.
| LibriSpeech | LibriTTS | IEMOCAP | |
| L1 distance | 8.198 | 4.073 | 4.071 |
| Pearson Correlation | 0.971 | 0.972 | 0.920 |
Table 6: Performance of our speech rate predictor across datasets. We report L1 distance and Pearson correlation with respect to ground-truth token lengths.
| Air-bench | SpeechR | MMSU | Speech-IFeval | Rel. (%) | CKA | |
| 7.24 | 52.00 | 53.14 | -17.68 | -10.66% | 0.6044 | |
| 7.85 | 52.91 | 53.85 | -12.18 | 0.00% | 0.6399 | |
| 7.78 | 54.90 | 53.77 | -13.81 | -2.31% | 0.6601 | |
| 7.57 | 52.17 | 53.12 | -17.48 | -9.20% | 0.6914 | |
| 7.64 | 53.15 | 53.62 | -17.66 | -8.44% | 0.7266 | |
| 7.62 | 50.77 | 51.81 | -23.62 | -14.90% | 0.744 |
Table 7: Ablation results across different values of . We report performance on multiple benchmarks along with representation similarity (CKA).
| Air-bench | SpeechR | MMSU | Speech-IFeval | Rel. (%) | |
| instructions=1 | 7.59 | 47.33 | 52.45 | -22.94 | -16.20% |
| instructions=2 | 7.64 | 48.04 | 52.22 | -23.92 | -16.27% |
| instructions=3 | 6.96 | 48.46 | 52.10 | -27.71 | -20.34% |
| instructions=4 | 7.53 | 48.66 | 53.77 | -23.08 | -15.09% |
| instructions=5 | 7.74 | 50.59 | 53.85 | -21.51 | -12.35% |
| instructions=6 | 7.85 | 52.91 | 53.85 | -12.18 | 0.00% |
Table 8: Ablation results with varying numbers of instructions. All metrics improve consistently as the number of instructions increases, with the largest gains observed when . Higher is better for all metrics, while Speech-IFeval is better when closer to zero.
| Dataset | Hours | Information |
| GigaSpeech ( 6 ) | 10,044.55 | transcription, data source |
| DailyTalk ( 27 ) | 21.67 | transcription, emotion, action |
| LJSpeech ( 21 ) | 23.92 | transcription |
| SLURP ( 3 ) | 26.27 | transcription, intent, action, scenario |
| VCTK ( 59 ) | 43.89 | transcription, age, gender, accent, region |
| Libriheavy ( 22 ) | 51,024.12 | transcription |
Table 9: Summary of datasets
| Category | Instruction |
| Content Repetition | 1. Repeat the provided speech, ensuring to maintain its original meaning and details. |
| 2. Provide the speech exactly as given—do not alter wording, structure, or omit any content. | |
| 3. Echo the content of the speech, maintaining its exact purpose and details. | |
| Keyword Extraction | 1. Extract the most frequently occurring words or phrases in the speech, excluding common stopwords, to identify main topics. |
| 2. Identify and list the most common words or phrases from the speech, omitting typical stopwords, to highlight central themes. | |
| 3. Extract significant words or phrases that appear often in the speech, exclude basic stopwords, to uncover the main subjects. |
Table 10: Instruction Set