SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech

Hsing-Hang Chou; Yun-Shao Lin; Ching-Chin Sung; Chi-Chun Lee

eess.SPOct 8, 2026

SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech

Authors: Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung, Chi-Chun Lee

Organizations: Department of Electrical Engineering, National Tsing Hua University, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan

Abstract

Cochlear implants (CIs) restore speech access but weaken cues needed for vocal emotion recognition. Prior CI-oriented enhancement requires parallel normal/strong recordings and intensity labels. We propose SEER, a retrieval-based framework that learns which same-emotion reference helps each source remain recognizable after CI processing. A source-conditioned retriever learns CI-aware utility from sampled emotional voice conversion outcomes, while uncertainty-guided exploration avoids exhaustive pair evaluation; neither parallel recordings nor intensity labels are required. SEER improves Source macro-F1 at N8 by 7.30 points on RAVDESS and 11.66 points on ESD, with significant ESD gains across N4/N8/N16. Sixteen-listener RAVDESS gains are significant across all conditions. Exhaustive analysis finds an aggregate benefit from stronger references but little effect from matching gender or content.

Explore similar work

Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors
Jun 12, 2026Youjun Chen, Xurong Xie, Mengzhe Geng +9Explainable Artificial Intelligence Reinforcement Learning
The Chinese University of Hong Kong, Hong Kong SAR, China · Institute of Software, Chinese Academy of Sciences, China · National Research Council Canada, Canada +1
EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis
Jun 22, 2026Wai Laam Mak, Isibor Kennedy Ihianle, Pedro MachadoSpeech Processing Emotion Recognition in Conversations
School of Science and Technology, Nottingham Trent University, Nottingham, UK
Enriching Speech Emotion Representations with Conversational Context
Sep 22, 2026Arthur Peuvot, Romaric Besançon, Gaël de Chalendar +2Representation Learning Emotion Recognition in Conversations
Université Paris-Saclay, CEA, List, France · LISN, CNRS, Université Paris-Saclay, France

Jun 12, 2026, cs.SDJ/K move · Enter open · S save

Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors

Youjun Chen, Xurong Xie, Mengzhe Geng, Zengrui Jin+8

The Chinese University of Hong Kong, Hong Kong SAR, China · Institute of Software, Chinese Academy of Sciences, China · National Research Council Canada, Canada+1

Explainable Artificial Intelligence Reinforcement Learning Emotion Recognition in Conversations Speech Emotion Recognition Learning with Noisy Labels

Open paper AskPDF arXiv

Explainable and trustworthy speech emotion recognition (SER) remains a challenging task to date, largely due to the scarcity of SER data with reliable speech emotion descriptor (SED) labels, such as prosodic features and speaker traits. This paper presents a confidence score and reinforcement learning (RL) based on-the-fly SED rectification approach for post-training SER systems on automatically annotated SED labels. Experiments on IEMOCAP and MELD suggest that explainable SER systems incorporating the proposed confidence score and RL-based SED rectification approach consistently outperform baselines without data selection or SED rectification. The best performing system, which integrates both components, surpasses the baseline without data selection and SED rectification, achieving SER gains of 2.9% and 3.3% absolute (3.7% and 5.4% relative) on IEMOCAP and MELD benchmarks, respectively.

Abstract

Explore similar work

Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors

EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis

Enriching Speech Emotion Representations with Conversational Context