cs.CLNov 13, 2026

What Do Multilingual Self-Supervised Speech Encoders Know About Sentence Meaning?

Authors: Kevin Krahn, Eric Fosler-Lussier

Organizations: Dept. of Computer Science and Engineering, The Ohio State University

Abstract

Self-supervised speech models (S3Ms) have been scaled up to thousands of languages and millions of hours of audio, yet little is known about how these models structure semantic knowledge across languages. In this work, we probe sentence-level semantics in massively multilingual S3Ms across 76 languages, using linear maps from frozen speech representations into a frozen text embedding space, evaluated by cross-modal retrieval. Sentence meaning is linearly accessible but highly uneven across languages and resource tiers. Cross-lingual probes can match language-specific probes on languages never seen in probe training, but per-model monolingual decodability does not predict transfer. Within the OmniASR wav2vec family, scaling from 300M--7B disproportionately benefits high-resource languages, and ASR tuning sharply improves semantic decodability while making representations more language-specific. However, semantic content is entangled with acoustics, inflating apparent performance on single-speaker corpora.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Understanding When and Where Self-Supervised Speech Models Learn Phonemes and Prosodic Cues through Layer-Wise and Time-Resolved Probing

    Nov 8, 2026Masaru Tanibata, Shun Takahashi, Hiroki Ouchi +1Self-Supervised Speech Representation LearningSpeech Prosody

  2. Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders

    Aug 5, 2026Yexing Du, Kaiyuan Liu, Youcheng Pan +4Speech TranslationMixture of Experts