Multimodal Contrastive Representation Learning Based Voice Conversion for Electrolaryngeal Speech Enhancement

Ding Ma; Jinyi Mi; Jiajun He; Kazuhiro Kobayashi; Naoki Nishio; Tomoki Toda

cs.SDOct 13, 2026

Multimodal Contrastive Representation Learning Based Voice Conversion for Electrolaryngeal Speech Enhancement

Authors: Ding Ma, Jinyi Mi, Jiajun He, Kazuhiro Kobayashi, Naoki Nishio, Tomoki Toda

Organizations: Graduate School of Informatics, Nagoya University, Nagoya, Japan · Alibaba Token Foundry, Alibaba Group, Hangzhou, China · Headquarters, TARVO, Inc., Nagoya, Japan · Department of Otorhinolaryngology, Graduate School of Medicine, Nagoya University, Nagoya, Japan ·

Abstract

Electrolaryngeal (EL) speech suffers from severe acoustic mismatches with normal speech, resulting in degraded naturalness and intelligibility. These mismatches also make sequence-to-sequence voice conversion struggle to learn robust intermediate representations, hindering EL-speech-to-normal-speech conversion (EL2SP) performance. To address this, we propose a novel multimodal contrastive representation learning framework to 1) achieve richer multimodal-representation learning, and 2) facilitate the transferability of multimodal representations to speech-only EL2SP. First, EL speech and text representations are fused to learn multimodal representations for target speech generation, during which cross-modal contrastive learning is employed to encourage a semantically consistent speech--text representation space. Next, the fused representations are treated as a fixed target to guide speech-only representation learning through cross-stage contrastive learning, enabling multimodal knowledge to be transferred to the EL2SP task while preserving architectural and inference simplicity. Experimental results demonstrate that the proposed method consistently outperforms both the speech-only and multimodal-based baselines.

Figures & tables

Fig. 1: Overview of the proposed two-stage multimodal contrastive representation learning VC framework, MCRL-VC.

Systems	MCD ( $\downarrow$ )	CER ( $\downarrow$ )	SECS ( $\uparrow$ )	SSQA ( $\uparrow$ )	NMOS ( $\uparrow$ )
S-VC	7.42	37.0	0.587	3.80	2.64 $\pm$ 0.09
MR-VC	7.35	35.5	0.630	3.83	3.04 $\pm$ 0.10
MCRL-VC (proposed)	7.20	32.7	0.647	3.84	3.15 $\pm$ 0.09
MCRL-VC-Rec (proposed)	7.25	33.9	0.644	3.88	–
EL	13.28	41.8	0.014	2.12	1.08 $\pm$ 0.04
NL (ground truth)	–	15.1	–	4.48	4.62 $\pm$ 0.07

TABLE I: Objective and subjective evaluation results on the comparable systems, original EL, and normal (NL) speech for Patient 1, with 95% confidence intervals shown for NMOS.

Systems	MCD ( $\downarrow$ )	CER ( $\downarrow$ )	SECS ( $\uparrow$ )	SSQA ( $\uparrow$ )	NMOS ( $\uparrow$ )
S-VC	6.24	23.3	0.734	3.91	2.56 $\pm$ 0.09
MR-VC	6.04	22.1	0.738	3.93	2.86 $\pm$ 0.09
MCRL-VC (proposed)	5.87	20.1	0.743	3.95	3.28 $\pm$ 0.10
MCRL-VC-Rec (proposed)	5.96	21.6	0.739	3.99	–
EL	12.40	25.1	0.047	2.76	1.18 $\pm$ 0.05
NL (ground truth)	–	10.9	–	4.22	4.41 $\pm$ 0.09

TABLE II: Objective and subjective evaluation results on the comparable systems, original EL, and normal (NL) speech for Patient 2, with 95% confidence intervals shown for NMOS.

Fig. 2: Heatmaps of cosine similarity matrices between the second-stage speech representations and the corresponding first-stage fused representations within each system: (a) MR-VC, (b) MCRL-VC-Rec, and (c) MCRL-VC, using the test set from the Patient 1 dataset.

Voice Conversion

Voice conversion changes a speaker's vocal identity or delivery while preserving the spoken content.

See all 35 papers

Advancing Electrolaryngeal Speech Enhancement Through Speech-Text Representation Learning
Jun 1, 2026Ding Ma, Jinyi Mi, Fengji Li +5Representation Learning Speech Processing
Graduate School of Informatics, Nagoya University, Nagoya 464-8601, Japan · School of Biological Science and Medical Engineering, Beihang University, Beijing 100191, China · TARVO, Inc., Nagoya 460-0008, Japan +1
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
Sep 16, 2026Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf +2Audio Representation Learning Streaming ASR
Signal Processing and Speech Communication Laboratory, Graz University of Technology · Comprehensive Center for AI in Medicine, Medical University of Vienna · Dept. of Otolaryngology, Head & Neck Surgery, Div. of Phoniatrics-Logopedics, Speech & Hearing Sci. Lab, Med. Univ. of Vienna
Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition
Mar 24, 2026Lucas H. Ueda, João G. T. Lima, Paula D. P. CostaContrastive Learning Multimodal Emotion Recognition
Dept. of Computer Engineering and Automation (DCA), Faculdade de Engenharia El´etrica e de Computac¸˜ao, and are part of the AI Lab., Recod.ai, Institute of Computing, Universidade Estadual de Campinas, UNICAMP

Jun 1, 2026J/K move · Enter open · S save

Advancing Electrolaryngeal Speech Enhancement Through Speech-Text Representation Learning

Ding Ma, Jinyi Mi, Fengji Li, Lester Phillip Violeta+4

Graduate School of Informatics, Nagoya University, Nagoya 464-8601, Japan · School of Biological Science and Medical Engineering, Beihang University, Beijing 100191, China · TARVO, Inc., Nagoya 460-0008, Japan+1

Representation Learning Speech Processing Speech Enhancement Voice Conversion

Open paper AskPDF arXiv

Objective: laryngectomees depend on an electromechanical device to generate electrolaryngeal (EL) speech. Compared with normal speech, EL speech suffers from severe distortion, limited phonetic variation, unnatural prosody, and temporal shifts, degrading naturalness and intelligibility. Although sequence-to-sequence (seq2seq) voice conversion (VC) based EL-speech-to-normal-speech conversion (EL2SP) is promising, substantial mismatches between EL and normal speech inevitably cause cumulative mapping errors that limit performance. To address this, we describe a novel representation learning framework integrating speech and text representations to improve mapping and reconstruction quality within a seq2seq VC model. Methods: our methodology comprises two main stages: 1) representation integration and learning, and 2) reconstruction training. A network capable of incorporating auxiliary text information is first constructed with pretrained modules to learn speech--text-based integrated representations. Then, an autoencoder-style reconstruction strategy finalizes EL2SP model to inherit these representations without increasing model complexity. We introduce three fusion strategies including middle-, input-, and hybrid-level fusion strategies that progressively enhance learning. Moreover, besides standard seq2seq VC objectives, an additional reconstruction loss on the integrated representation is introduced to refine representation transfer. Results: experiments under different EL2SP datasets consistently demonstrate that our methods, combined with data augmentations, outperform baselines relying solely on speech representations. Furthermore, progressive improvements with system design depth validate the effectiveness of our methods. Significance: the proposed methods provide an extensible and practical methodology for EL speech enhancement and assistive communication technologies.

Abstract

Voice Conversion

Explore similar work

Advancing Electrolaryngeal Speech Enhancement Through Speech-Text Representation Learning

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition