Low resource cross-modal alignment using HGNN to enhance speech representation

Yannick Yomie Nzeuhang; Marie Tahon; Paulin Melatagia Yonta

cs.CLSep 19, 2026

Low resource cross-modal alignment using HGNN to enhance speech representation

Authors: Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia Yonta

Organizations: Department of computer sciences, University of Yaounde I, Street, Yaounde, 812, Cameroon. · IRD, UMMISCO, Street, Bondy, F-93143, France. · LIUM, Le Mans Universit´e, Avenue Olivier Messiaen, Le Mans, 72085, France.

Abstract

Speech-text space alignment is a multimodal representation learning method consisting to map different speech and text into a shared representation space, leading to enrichment of the representation of each modality. Proposed architectures, such as SAMU-XLSR, typically follow a student/teacher framework, with the goal of fine-tuning an audio encoder to produce representations that closely match those of the text. In this way a speech representation is semantically enriched. However, such systems generally require large amounts of training data and considerable computational resource, making them difficult to apply to low resources languages under frugal constraints. The present work proposes a data-efficient space alignment method based on Heterogeneous Graph Neural Networks and link prediction. The core idea is to leverage message passing to explicitly transfer information from the text modality to the speech modality, thereby reducing the need for large training datasets and intrinsically enriching the acoustic representations, all in a more interpretable manner. Although thoroughly explored for high-resource languages, word-level tasks in speech remain relevant for certain low-resource languages. Therefore, we conducted experiments on speech-text alignment at the word level using the TIMIT (English) dataset and Yemba (a Cameroonian language). Our approach yields results comparable to those of SAMU-XLSR, a state-of-the-art method, and even surpasses it for the Yemba language in the task of word retrieval, while using far fewer resources, demonstrating its power, frugality, and efficiency.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the heterogeneous graph construction protocol and the GCN training architecture.

	a	d	f	l	n	o	p	r	s	t	6	E	g	I	U
down	1	1	0	0	1	0	0	0	0	0	0	0	0	0	1
go	0	0	0	0	0	1	0	0	0	0	0	0	1	0	1
left	0	0	1	1	0	0	0	0	0	1	0	1	0	0	0
no	0	0	0	0	1	1	0	0	0	0	0	0	0	0	1
right	1	0	0	0	0	0	0	1	0	1	0	0	0	1	0
stop	0	0	0	0	0	0	1	0	1	1	1	0	0	0	0

Table 1: Bag-of-phonemes representations of each word from the “Google Speech Commands”.

Transductive
Setting and pipeline	$\overline{\text{WO@5}}$ (%)	$\overline{\text{PO@5}}$ (%)	Inter-Sim	Intra-Sim
SAMU-XLSR	17.0	29.4	0.77	0.87
XLSR+HGNN+LaBSE	30.6	42.2	0.89	0.94
XLSR+HGNN+Articulatory	23.8	44.8	0.29	0.98
XLSR+HGNN+Bag of phoneme	9.4	27.8	0.23	0.98
Inductive Semantic

Table 2: Comparison between our approach (Learned HGNN variants) and SAMU-XLSR in transductive and inductive settings on TIMIT. Word and phonetic overlaps are reported as

\overline{\text{WO@5}}

and

\overline{\text{PO@5}}

in percentages

Metric	SAMU-XLSR	Our Approach
Train parameters	$300M$	$9M$
Hardware	32 V100-32GB GPUs	32 CPUs
Train Data	6.8 Kh	23 min (TIMIT)
		2h (YD)
Train iterations	400K	1K

Table 3: Comparison of Computational Resources and Training Requirements between SAMU-XLSR and Our Approach

Transductive
Setting and pipeline	$\overline{\text{WO@5}}$ (%)	$\overline{\text{PO@5}}$ (%)	Inter-Sim	Intra-Sim
SAMU-XLSR	13.6	22.0	0.67	0.82
XLSR+HGNN+LaBSE	20.2	33.2	0.26	0.98
XLSR+HGNN+Articulatory	12.6	26	0.16	0.98
XLSR+HGNN+Bag of phonemes	22.2	38.6	0.31	0.98
Inductive

Table 4: Comparison between our approach (Learned HGNN variants) and SAMU-XLSR in transductive and inductive settings on the Yemba dataset. Word and phonetic overlaps are reported as

\overline{\text{WO@5}}

and

\overline{\text{PO@5}}

in percentages

Panel of Figure 2 — Figure 2: Performance comparison between SAMU-XLSR and our approach XLSR + HGNN + LaBSE, across TIMIT and Yemba inductive dataset on word overlap metric at varying top-k values

TIMIT Wav2vec
Setting	LaBSE		Articulatory		Bag of phonemes
	Init	Learned/+LP	Init	Learned/+LP	Init	Learned/+LP
Intra-similarity	0.78	0.95	0.78	0.96	0.78	0.93
Inter-similarity	0.70	0.18	0.70	0.25	0.70	0.23
TIMIT XLSR
Intra-similarity	0.98	0.97	0.98	0.98	0.98	0.96

Table 5: Intra- (

\uparrow

) and inter-similarity (

\downarrow

) under transductive and inductive settings on TIMIT dataset (Wav2Vec, XLSR) and Yemba (XLSR)

Transductive
Setting	LaBSE			Articulatory			Bag of phonemes
	Init	Learned	+LP	Init	Learned	+LP	Init	Learned	+LP
$\overline{\text{WO@5}}$ (%)	6.4	29.2	27.8	3.6	27.4	31.2	5.0	13.0	8.6
$\overline{\text{PO@5}}$ (%)	17.6	39.8	40.0	10.0	47.4	54.2	11.6	37.4	21.2
Inductive (Semantic)
$\overline{\text{WO@5}}$ (%)	5.0	26.6	19.4	9.0	19.0	17.6	5.6	7.2	8.8

Table 6: Average word overlap (

\overline{\text{WO@5}}

\uparrow

) and phonetic overlap (

\overline{\text{PO@5}}

\uparrow

) under transductive and inductive settings on the TIMIT dataset (using Wav2Vec embeddings). Results are reported with

k=5

. Values are percentages, where 100% corresponds to perfect retrieval.

Transductive
Setting	LaBSE			Articulatory			Bag of phonemes
	Init	Learned	+LP	Init	Learned	+LP	Init	Learned	+LP
$\overline{\text{WO@5}}$ (%)	2.8	30.0	29.2	3.6	26.6	26.8	5.8	11.0	8.2
$\overline{\text{PO@5}}$ (%)	7.0	41.2	40.8	10.2	47.2	43.4	15.0	33.8	22.8
Inductive (Semantic)
$\overline{\text{WO@5}}$ (%)	4.6	25.0	20.2	6.2	15.6	14.6	6.8	11.4	10.4

Table 7: Average word overlap (

\overline{\text{WO@5}}

\uparrow

) and phonetic overlap (

\overline{\text{PO@5}}

\uparrow

) under transductive and inductive settings on the TIMIT dataset (using XLSR embeddings). Results are reported with

k=5

. Values are percentages, where 100% corresponds to perfect retrieval.

Transductive
Setting	LaBSE			Articulatory			Bag of phonemes
	Init	Learned	+LP	Init	Learned	+LP	Init	Learned	+LP
$\overline{\text{WO@5}}$ (%)	5.6	20.2	16.2	7.6	12.6	13.2	9	22.2	24.0
$\overline{\text{PO@5}}$ (%)	12.2	33.2	21.0	17.0	26.0	21.6	17.6	38.6	34.6
Inductive
$\overline{\text{WO@5}}$ (%)	2.4	20.0	18.8	3.2	11.8	5.2	4.8	18.8	16.4

Table 8: Average word overlap (

\overline{\text{WO@5}}

\uparrow

) and phonetic overlap (

\overline{\text{PO@5}}

\uparrow

) under transductive and inductive settings on the Yemba dataset (using XLSR embeddings).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Term	English Translation
mbeN	The rain
mbīN	The forest
lekwEt	A mountain
mbuO	The chalk
metúa	A car
kamElE	The vaccination

Appendix

Table 9: Terms and their English Translations

Original	Semantic	Phonetic
much	lots	match
near	close	kneer
often	frequently	orphan
please	kindly	fleece
problem	issue	propel
said	stated	sad

Appendix

Table 10: Correspondences between original words, their semantic and phonetic equivalents

Explore similar work

Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba
Sep 19, 2026Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie TahonAudio Representation Learning Cross-Modal Representation Learning
Department of computer sciences of University of Yaounde I, Yaounde, 812, Cameroun · Department of computer sciences of University of Yaounde I and IRD, UMMISCO, Bondy, France, F-93143 · LIUM, Le Mans Universit´e, Av. Olivier Messiaen, 72085 Le Mans, France
Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
Jun 10, 2026Zhen Ye, Xu Tan, Yiming Li +10Spoken Language Understanding Audio Representation Learning
Hong Kong University of Science and Technology, Hong Kong SAR · Tencent, China · Independent Researcher +4
Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding
Aug 27, 2026Gabriel Pirlogeanu, Dan Oneata, Horia Cucu +1Speech Processing Low-Resource Language Processing
Speech and Dialogue Research Laboratory, POLITEHNICA Bucharest, Romania · Department of Electrical and Electronic Engineering, Stellenbosch University, South Africa

Sep 19, 2026J/K move · Enter open · S save

Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba

Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie Tahon

Department of computer sciences of University of Yaounde I, Yaounde, 812, Cameroun · Department of computer sciences of University of Yaounde I and IRD, UMMISCO, Bondy, France, F-93143 · LIUM, Le Mans Universit´e, Av. Olivier Messiaen, 72085 Le Mans, France

Audio Representation Learning Cross-Modal Representation Learning Low-Resource Language Processing Low-Resource Speech Recognition

Open paper AskPDF arXiv

Acoustic representation learning is crucial for speech processing, yet low-resource languages (LRLs) face severe data scarcity, limiting the effectiveness of traditional and self-supervised methods. As a promising alternative, in this work, we propose to enhance acoustic representation trough a cross-modal transfer knowledge approach, based on heterogeneous graph neural networks (HGNNs), where acoustic and linguistic entities are modeled as distinct node types within a unified graph. Through message-passing mechanisms, linguistic nodes explicitly transfer knowledge to acoustic nodes, enabling structured and interpretable cross-modal information flow. To highlight this knowledge transfer and its benefits, we measured standard clustering metrics as an intrinsic evaluation of acoustic representation, and to emphasize applicability, we performed isolated-word recognition tasks using an English benchmark and a Cameroonian language dataset in low resources settings . Results demonstrate that acoustic representations consistently benefit from linguistic knowledge propagated through the graph. To our knowledge, this is the first demonstration of explicit cross-modal knowledge transfer for acoustic representation learning using HGNNs, highlighting a promising direction for speech representation in low-resource settings.

Abstract

Explore similar work

Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba

Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding