Low resource cross-modal alignment using HGNN to enhance speech representation

Yannick Yomie Nzeuhang; Marie Tahon; Paulin Melatagia Yonta

cs.CLSep 19, 2026

Low resource cross-modal alignment using HGNN to enhance speech representation

Authors: Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia Yonta

Organizations: Department of computer sciences, University of Yaounde I, Street, Yaounde, 812, Cameroon. · IRD, UMMISCO, Street, Bondy, F-93143, France. · LIUM, Le Mans Universit´e, Avenue Olivier Messiaen, Le Mans, 72085, France.

Abstract

Speech-text space alignment is a multimodal representation learning method consisting to map different speech and text into a shared representation space, leading to enrichment of the representation of each modality. Proposed architectures, such as SAMU-XLSR, typically follow a student/teacher framework, with the goal of fine-tuning an audio encoder to produce representations that closely match those of the text. In this way a speech representation is semantically enriched. However, such systems generally require large amounts of training data and considerable computational resource, making them difficult to apply to low resources languages under frugal constraints. The present work proposes a data-efficient space alignment method based on Heterogeneous Graph Neural Networks and link prediction. The core idea is to leverage message passing to explicitly transfer information from the text modality to the speech modality, thereby reducing the need for large training datasets and intrinsically enriching the acoustic representations, all in a more interpretable manner. Although thoroughly explored for high-resource languages, word-level tasks in speech remain relevant for certain low-resource languages. Therefore, we conducted experiments on speech-text alignment at the word level using the TIMIT (English) dataset and Yemba (a Cameroonian language). Our approach yields results comparable to those of SAMU-XLSR, a state-of-the-art method, and even surpasses it for the Yemba language in the task of word retrieval, while using far fewer resources, demonstrating its power, frugality, and efficiency.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the heterogeneous graph construction protocol and the GCN training architecture.

	a	d	f	l	n	o	p	r	s	t	6	E	g	I	U
down	1	1	0	0	1	0	0	0	0	0	0	0	0	0	1
go	0	0	0	0	0	1	0	0	0	0	0	0	1	0	1
left	0	0	1	1	0	0	0	0	0	1	0	1	0	0	0
no	0	0	0	0	1	1	0	0	0	0	0	0	0	0	1
right	1	0	0	0	0	0	0	1	0	1	0	0	0	1	0
stop	0	0	0	0	0	0	1	0	1	1	1	0	0	0	0

Table 1: Bag-of-phonemes representations of each word from the “Google Speech Commands”.

Transductive
Setting and pipeline	$\overline{\text{WO@5}}$ (%)	$\overline{\text{PO@5}}$ (%)	Inter-Sim	Intra-Sim
SAMU-XLSR	17.0	29.4	0.77	0.87
XLSR+HGNN+LaBSE	30.6	42.2	0.89	0.94
XLSR+HGNN+Articulatory	23.8	44.8	0.29	0.98
XLSR+HGNN+Bag of phoneme	9.4	27.8	0.23	0.98
Inductive Semantic

Table 2: Comparison between our approach (Learned HGNN variants) and SAMU-XLSR in transductive and inductive settings on TIMIT. Word and phonetic overlaps are reported as

\overline{\text{WO@5}}

and

\overline{\text{PO@5}}

in percentages

Metric	SAMU-XLSR	Our Approach
Train parameters	$300M$	$9M$
Hardware	32 V100-32GB GPUs	32 CPUs
Train Data	6.8 Kh	23 min (TIMIT)
		2h (YD)
Train iterations	400K	1K

Table 3: Comparison of Computational Resources and Training Requirements between SAMU-XLSR and Our Approach

Transductive
Setting and pipeline	$\overline{\text{WO@5}}$ (%)	$\overline{\text{PO@5}}$ (%)	Inter-Sim	Intra-Sim
SAMU-XLSR	13.6	22.0	0.67	0.82
XLSR+HGNN+LaBSE	20.2	33.2	0.26	0.98
XLSR+HGNN+Articulatory	12.6	26	0.16	0.98
XLSR+HGNN+Bag of phonemes	22.2	38.6	0.31	0.98
Inductive

Table 4: Comparison between our approach (Learned HGNN variants) and SAMU-XLSR in transductive and inductive settings on the Yemba dataset. Word and phonetic overlaps are reported as

\overline{\text{WO@5}}

and

\overline{\text{PO@5}}

in percentages

Panel of Figure 2 — Figure 2: Performance comparison between SAMU-XLSR and our approach XLSR + HGNN + LaBSE, across TIMIT and Yemba inductive dataset on word overlap metric at varying top-k values

TIMIT Wav2vec
Setting	LaBSE		Articulatory		Bag of phonemes
	Init	Learned/+LP	Init	Learned/+LP	Init	Learned/+LP
Intra-similarity	0.78	0.95	0.78	0.96	0.78	0.93
Inter-similarity	0.70	0.18	0.70	0.25	0.70	0.23
TIMIT XLSR
Intra-similarity	0.98	0.97	0.98	0.98	0.98	0.96

Table 5: Intra- (

\uparrow

) and inter-similarity (

\downarrow

) under transductive and inductive settings on TIMIT dataset (Wav2Vec, XLSR) and Yemba (XLSR)

Transductive
Setting	LaBSE			Articulatory			Bag of phonemes
	Init	Learned	+LP	Init	Learned	+LP	Init	Learned	+LP
$\overline{\text{WO@5}}$ (%)	6.4	29.2	27.8	3.6	27.4	31.2	5.0	13.0	8.6
$\overline{\text{PO@5}}$ (%)	17.6	39.8	40.0	10.0	47.4	54.2	11.6	37.4	21.2
Inductive (Semantic)
$\overline{\text{WO@5}}$ (%)	5.0	26.6	19.4	9.0	19.0	17.6	5.6	7.2	8.8

Table 6: Average word overlap (

\overline{\text{WO@5}}

\uparrow

) and phonetic overlap (

\overline{\text{PO@5}}

\uparrow

) under transductive and inductive settings on the TIMIT dataset (using Wav2Vec embeddings). Results are reported with

k=5

. Values are percentages, where 100% corresponds to perfect retrieval.

Transductive
Setting	LaBSE			Articulatory			Bag of phonemes
	Init	Learned	+LP	Init	Learned	+LP	Init	Learned	+LP
$\overline{\text{WO@5}}$ (%)	2.8	30.0	29.2	3.6	26.6	26.8	5.8	11.0	8.2
$\overline{\text{PO@5}}$ (%)	7.0	41.2	40.8	10.2	47.2	43.4	15.0	33.8	22.8
Inductive (Semantic)
$\overline{\text{WO@5}}$ (%)	4.6	25.0	20.2	6.2	15.6	14.6	6.8	11.4	10.4

Table 7: Average word overlap (

\overline{\text{WO@5}}

\uparrow

) and phonetic overlap (

\overline{\text{PO@5}}

\uparrow

) under transductive and inductive settings on the TIMIT dataset (using XLSR embeddings). Results are reported with

k=5

. Values are percentages, where 100% corresponds to perfect retrieval.

Transductive
Setting	LaBSE			Articulatory			Bag of phonemes
	Init	Learned	+LP	Init	Learned	+LP	Init	Learned	+LP
$\overline{\text{WO@5}}$ (%)	5.6	20.2	16.2	7.6	12.6	13.2	9	22.2	24.0
$\overline{\text{PO@5}}$ (%)	12.2	33.2	21.0	17.0	26.0	21.6	17.6	38.6	34.6
Inductive
$\overline{\text{WO@5}}$ (%)	2.4	20.0	18.8	3.2	11.8	5.2	4.8	18.8	16.4

Table 8: Average word overlap (

\overline{\text{WO@5}}

\uparrow

) and phonetic overlap (

\overline{\text{PO@5}}

\uparrow

) under transductive and inductive settings on the Yemba dataset (using XLSR embeddings).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Term	English Translation
mbeN	The rain
mbīN	The forest
lekwEt	A mountain
mbuO	The chalk
metúa	A car
kamElE	The vaccination

Appendix

Table 9: Terms and their English Translations

Original	Semantic	Phonetic
much	lots	match
near	close	kneer
often	frequently	orphan
please	kindly	fleece
problem	issue	propel
said	stated	sad

Appendix

Table 10: Correspondences between original words, their semantic and phonetic equivalents

Explore similar work

Sep 19, 2026

Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba

Acoustic representation learning is crucial for speech processing, yet low-resource languages (LRLs) face severe data scarcity, limiting the effectiveness of traditional and self-supervised methods. As a promising alternative, in this work, we propose to enhance acoustic representation trough a cross-modal transfer knowledge approach, based on heterogeneous graph neural networks (HGNNs), where acoustic and linguistic entities are modeled as distinct node types within a unified graph. Through message-passing mechanisms, linguistic nodes explicitly transfer knowledge to acoustic nodes, enabling structured and interpretable cross-modal information flow. To highlight this knowledge transfer and its benefits, we measured standard clustering metrics as an intrinsic evaluation of acoustic representation, and to emphasize applicability, we performed isolated-word recognition tasks using an English benchmark and a Cameroonian language dataset in low resources settings . Results demonstrate that acoustic representations consistently benefit from linguistic knowledge propagated through the graph. To our knowledge, this is the first demonstration of explicit cross-modal knowledge transfer for acoustic representation learning using HGNNs, highlighting a promising direction for speech representation in low-resource settings.

Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie Tahon

Department of computer sciences of University of Yaounde I, Yaounde, 812, Cameroun · Department of computer sciences of University of Yaounde I and IRD, UMMISCO, Bondy, France, F-93143 · LIUM, Le Mans Universit´e, Av. Olivier Messiaen, 72085 Le Mans, France

Audio Representation Learning Cross-Modal Representation Learning Low-Resource Language Processing Low-Resource Speech Recognition

Ask PDF

Jun 10, 2026

Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

Spoken dialogue models typically start from text LLM backbones, yet reasoning often degrades when conditioning on speech instead of text. We attribute part of this modality gap to a temporal-granularity mismatch: speech tokens are temporally redundant and far longer than text under matched semantics, diluting per-token semantic density and weakening text-native reasoning dynamics. We study speech token design as a representation selection problem and sweep frame rates under a frozen LLM backbone with a fixed information rate. To make low frame rates feasible, we introduce factorized FSQ and a lightweight non-autoregressive audio LM head, scaling capacity to nearly 300,bits/frame without sacrificing efficient prediction. With the bottleneck removed, we sweep frame rates (50

\rightarrow

2.08,Hz) and alignment depth, and observe a consistent best regime for speech QA at 4.17,Hz with intermediate-layer representation alignment.

Zhen Ye, Xu Tan, Yiming Li +10

Hong Kong University of Science and Technology, Hong Kong SAR · Tencent, China · Independent Researcher +4

Spoken Language Understanding Audio Representation Learning Audio Understanding Audio-Language Models Discrete Speech Representation Learning Spoken QA

Ask PDF

Aug 27, 2026

Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding

In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speech data? Given a dataset of images with Hindi spoken captions, we consider how we can map a written English keyword to spoken realisations of that word in Hindi. Previous work trained end-to-end multimodal neural models. Instead, we explore a simpler alignment-based approach built on self-supervised speech representations. Written English tags are automatically obtained from images using off-the-shelf image captioning systems. Hindi utterances associated with the same keyword are then aligned (using self-supervised features), and alignment evidence is aggregated to identify recurring speech segments corresponding to the target word. Experiments evaluating keyword spotting and localization show that our alignment-based approach outperforms a previous attention-based neural model. We also show the benefit of incorporating negative examples during alignment. Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding without transcriptions or explicit model training.

Gabriel Pirlogeanu, Dan Oneata, Horia Cucu +1

Speech and Dialogue Research Laboratory, POLITEHNICA Bucharest, Romania · Department of Electrical and Electronic Engineering, Stellenbosch University, South Africa

Speech Processing Low-Resource Language Processing

Ask PDF