cs.CLMay 26, 2026

A Study of ASR Adaptation and Representation Dimensionality Reduction in Persian Speech Emotion Recognition Using Whisper

Authors: Ali Shendabadi, Parnia Izadirad, Mostafa Salehi

Organizations: Faculty of Intelligent Systems Engineering College of Interdisciplinary Sciences and Technologies University of Tehran, Tehran, Iran

Abstract

Speech Emotion Recognition (SER) in low-resource languages remains a challenging problem due to limited labeled data. In this work, we study the use of Whisper for Persian SER with a particular focus on representation dimensionality reduction and language-specific model adaptation. We propose a SER framework in which frame-level embeddings extracted from the Whisper encoder are reduced in dimensionality using PCA, eliminating the need for learned projection layers and substantially reducing the number of trainable parameters. The reduced representations are aggregated using an attention-based pooling mechanism and classified with a lightweight prediction head. In addition, we investigate whether fine-tuning Whisper on a Persian automatic speech recognition (ASR) task improves downstream SER performance. Experiments conducted on the ShEMO dataset under a speaker-independent evaluation protocol show that PCA-based dimensionality reduction consistently improves emotion recognition performance while reducing training latency and memory usage. ASR fine-tuning yields only modest gains for SER, suggesting limited transfer from language adaptation to emotion-related representations under the evaluated conditions. These findings provide practical insights into the efficient use of large pretrained speech models for emotion recognition in low-resource languages.

Explore similar work

CardsList
  1. How Language-Independent Are Emotional Attributes? A Study on Training Data Scaling and Cross-Lingual Generalization

    Oct 10, 2026Dániel Halmai, Gábor GosztolyaSpeech Emotion Recognition

  2. HyPASE: Hyperbolic Geometry for Parameter-Efficient Speech Emotion Fine-Tuning Framework for Large Audio-Language Models

    Aug 5, 2026Tian Jin, Ruikang Zhang, Zefeng Zhao +2Audio-Language ModelsMultimodal Emotion Recognition

  3. EmoNet: A Lightweight Attention-Based CNN for Speech Emotion Recognition

    Oct 25, 2026Muhammad Moiz Khalid, Abdur Rehman Tariq, Muhammad Okasha YaminSpeech Emotion Recognition