eess.ASOct 7, 2026

Mitigating Accent-Language Confusion in Self-Supervised Speech Representations for Language Identification

Authors: Minu Kim, Jihwan Lee, David R. Mortensen, Shrikanth Narayanan

Organizations: Signal Analysis and Interpretation Lab (SAIL), University of Southern California, USA · Language Technologies Institute, Carnegie Mellon University, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Spoken language identification (LID) aims to recognize the target language regardless of accent. In practice, however, LID models fine-tuned from self-supervised speech representations frequently confuse accents with languages, misclassifying non-native (L2) speech as the speaker's first language (L1). We show that non-native speech representations lie between native target-language and native L1 poles, causing systematic misclassification. To address this, we introduce a geometric projection that estimates an L1-bias direction solely from native speech and removes it before the frozen LID head. Across five MMS-LID models and non-native corpora, this projection substantially improves target language identification for L2-accented speech while preserving predictions for native speech. These results show that accent-induced L1 bias can be corrected directly within the representation space without L2 training data or model adaptation.

Figures & tables

Figure 1: Geometry of L1-biased representations illustrated for native and non-native English speech. Top-left (Poles): Native reference distributions for English and non-English L1s. Top-right (X): Axis u^\hat{u} estimated from native speech (English+non-English). Bottom-left (Y): Decision margin from the frozen LID head. Bottom-right (Combined): Projections ( t=x⋅u^t=x\cdot\hat{u} ) show L2-accented English positioned between native English and L1s, causing L1-related misclassifications.
SvarahL2-ARCTICALLSSTAR
Nepali473Spanish600Cantonese551
Kannada347Hindi450Mandarin458
Malayalam311Mandarin450Turkish382
Urdu292Korean300Spanish377
Odia276Vietnamese300Korean370
Telugu214Arabic150Portuguese179
Table 1: Per-L1 clip counts in the evaluation corpora. Only non-native English clips from speakers whose L1 is represented in Fleurs are retained.
Figure 2: Accent-induced confusion in MMS-LID-4017. (a) Prediction breakdown for non-native English speech, categorized into English, speaker’s own L1 ( Own L1 ), languages related to or in contact with L1 ( Near L1 ), and remaining languages ( True Miss ). (b) Mean Whisper token-normalized negative log-likelihood (NLL) per speaker (ordered by NLL), with horizontal lines indicating corpus averages and dashed lines denoting the Fleurs native English baseline.
Figure 3: English identification performance across models and corpora. Open and filled markers show English identification rates before and after projection, respectively. Blue numbers indicate percentage-point changes for non-native English, while red annotations at the bottom show model-averaged native L1 identification rates before and after projections. The projection substantially improves English identification on Svarah and L2-ARCTIC while preserving predictions for both native groups.
Figure 4: Effect of removing the native English–L1 direction. For MMS-LID-4017, u^\hat{u} is estimated solely from the Fleurs dev set and visualized on test representations from Fleurs and Svarah. Projecting u^\hat{u} out moves non-native English representations toward the English side of the frozen LID decision boundary while preserving representations for both native groups.
Figure 5: Validity of the L1-bias direction. (a) Alignment of u^\hat{u} versus 20 random directions with the native English–L1 axis (MMS-LID-4017). (b) English identification accuracy before projection and after removing u^\hat{u} or random directions.

Explore similar work

CardsList

  1. Contrastive Regularization for Accent-Robust ASR

    May 5, 2026Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham +1Contrastive LearningContrastive Loss

    Air Traffic Management Research Institute, Nanyang Technological University, Singapore · Center of AI Research, VinUniversity, Vietnam

  2. Do Speech Representations Preserve Regional Accent Across Read and Spontaneous Speech?

    Oct 5, 2026Paula A. Perez-Toro, Tomas Arias-Vergara, Annette Schwarz +4

    Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany. · Leibniz Institute for Educational Trajectories (LIfBi), Bamberg, Germany. · Otto-Friedrich-Universität Bamberg, Bamberg, Germany.

  3. Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning

    Sep 24, 2026Yoomee Cho, Jisun LeeCross-Lingual Voice CloningSpeaker Similarity

    ESTsoft, Seoul, Republic of Korea