Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining

Xinnian Zhao; Chia-Hua Wu; Pu Wang; Hugo Van Hamme

cs.CLOct 8, 2026

Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining

Authors: Xinnian Zhao, Chia-Hua Wu, Pu Wang, Hugo Van Hamme

Organizations: Department of Electrical Engineering (ESAT), KU Leuven, Leuven, Belgium · Institute of Information Science, Academia Sinica, Taiwan

Abstract

Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge. The bridge is usually treated as plumbing, but it in fact defines the geometry of the speech-to-LLM interface, and the pretraining objective decides whether that interface provides a reusable initialization for downstream tasks. We study a transferable bridge through two complementary properties: global alignment with the text side, and local lexical manifold compatibility, where bridge embeddings remain close to the frozen LLM's input-embedding neighbourhoods. We make this property measurable with a fixed, head-free, timestamp-free diagnostic that applies to any objective, and show that next-word prediction (NWP) and sentence-level contrastive pretraining do not fully capture token-level lexical compatibility. We then introduce Local Prototype Reconstruction (LPR), a lightweight training-only regularizer that requires each aligned bridge token to be reconstructable from a small neighbourhood of frozen LLM token embeddings, with a hard single-prototype anchor as its limiting case. On multilingual ASR and speech translation, LPR improves transfer, with the largest gains on translation and low-resource adaptation. Crucially, our independent diagnostic correlates with downstream gains across objectives, suggesting that lexical manifold compatibility is predictive of reusability for speech-to-LLM bridges.

Figures & tables

Fig. 1: Motivating gap between global alignment and local compatibility. (a) Schematic view: NWP and contrastive objectives can produce bridge embeddings that support generation or utterance-level alignment without necessarily matching the local neighbourhoods of the frozen LLM input-embedding manifold. LPR adds a token-level regularizer toward these neighbourhoods. (b) Fixed head-free probes on the frozen bridge. Contrastive and LPR both reach near-perfect utterance-level retrieval R@1 (

\approx 1.0

), while token-level cosine to nearby LLM embeddings remains low for contrastive (

0.06

) and increases with LPR (

0.49

Fig. 2: Overview of Local Prototype Reconstruction (LPR). During pretraining, transcript tokens are softly aligned to continuous bridge embeddings through a schematic soft-alignment module, local frozen-LLM embedding neighbourhoods are retrieved, and aligned speech representations are reconstructed from these prototypes. The resulting token-level losses refine the sentence-level contrastive bridge. At inference, only the encoder–bridge path is used; transcripts, prototype retrieval, and LPR heads are discarded.


Data	Pretrain Obj.	ASR						ST
		WER (%) $\downarrow$						BLEU $\uparrow$
		De	Fr	Es	It	Pt	Avg.	De $\to$ En	Fr $\to$ En	Es $\to$ En	It $\to$ En	Pt $\to$ En	Avg.
$\sim$ 430h	None	7.97	10.60	5.61	6.94	5.34	7.75	35.24	39.02	41.08	37.33	49.68	39.09
	NWP	7.97	10.58	5.54	6.73	5.11	7.67	35.17	38.97	41.02	37.18	49.25	38.99
	Contra.	7.94	10.52	5.43	6.41	4.98	7.56	37.67	39.93	41.82	38.28	50.90	40.36

TABLE I: Downstream ASR and ST across pretraining objectives (per-language and average). Each row uses one multilingual bridge initialization. LPR performs best in all columns; gains are largest on ST with low-resource adaptation. Best per block in bold.

Fig. 3: Average relative downstream improvement over no pretraining across five source languages. ASR improvement is measured as relative WER reduction; ST improvement is measured as relative BLEU gain. LPR gives the largest average improvement in both tasks, with a larger margin in the low-resource setting.

Pretrain Obj.	Utt. R@1 $\uparrow$	Gold R@1 $\uparrow$	MRR $\uparrow$	MRE $\downarrow$	$H(\alpha)\!\downarrow$	$H(a)\!\downarrow$
NWP	0.631	0.081	0.260	1.887	2.763	3.338
Contra.	0.999	0.376	0.573	1.835	2.754	3.323
LPR (Anch.)	0.999	0.553	0.708	0.718	1.841	1.831
LPR (Recon.)	0.999	0.209	0.430	0.760	2.392	2.219
LPR (full)	0.999	0.443	0.611	0.645	1.912	1.822

TABLE II: Interface-compatibility diagnostics on the multilingual development set, computed with the fixed head-free probe. Utterance retrieval measures global alignment; the remaining metrics measure token-level lexical manifold compatibility.

Fig. 4: Relationship between lexical manifold compatibility and downstream transfer. Each point is one objective-language pair. The x-axis shows MRE from the fixed head-free probe (lower is closer to the local LLM lexical manifold); the y-axis shows gain over no pretraining (ASR: WER reduction; ST: BLEU gain). Dashed lines are OLS fits and

r

denotes Pearson correlation (

n=25

). The association is strongest in the low-resource setting.

Explore similar work

Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
Jun 24, 2026Tomoya Mizumoto, Yusuke FujitaAudio Representation Learning Self-Supervised Speech Representation Learning
SB Intuitions
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
Jun 8, 2026Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3Audio Representation Learning Audio Understanding
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.
DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs
Oct 6, 2026Hemant Yadav, Sunayana Sitaram, Roger Zimmermann +1Automatic Speech Recognition Speech Language Models
IIIT Delhi, India · Microsoft Research, India · National University of Singapore

Jun 24, 2026, eess.ASJ/K move · Enter open · S save

Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?

Tomoya Mizumoto, Yusuke Fujita

SB Intuitions

Audio Representation Learning Self-Supervised Speech Representation Learning Speech Language Models

Open paper AskPDF arXiv

Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM. Unlike encoders based on automatic speech recognition, which often produce representations in separate language-specific spaces, LLMs operate within a unified language-agnostic space. A mechanism is required to align the encoder's language-specific representations with the LLM's shared space. We argue that speech translation provides a principled way to achieve this. Unlike monolingual transcription, translation requires the model to bridge different languages and learn language-agnostic representations. We experimentally evaluate the impact of incorporating translation objectives into speech encoder pre-training. Our results demonstrate that translation-enhanced pre-training improves cross-modal integration and leads to superior performance across downstream Speech LLM tasks.

Abstract

Explore similar work

Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs