Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement

Fei Su; Cancan Li; Juan Liu; Wei Ju; Hongbin Suo; Ming Li

cs.SDMar 4, 2026

Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement

Authors: Fei Su, Cancan Li, Juan Liu, Wei Ju, Hongbin Suo, Ming Li

Organizations: School of Computer Science, Wuhan University, China · School of Artificial Intelligence, Wuhan University, China · AI Center, OPPO, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Digital Innovation Research Center, Duke Kunshan University, China

Abstract

Audio-Visual Speech Recognition (AVSR) integrates acoustic and visual information to enhance robustness in adverse acoustic conditions. Recent advances in Large Language Models (LLMs) have yielded competitive automatic speech recognition performance and shown effectiveness for AVSR. However, prior approaches project audio and visual features independently or apply shallow fusion, limiting cross-modal alignment and complementary exchange while increasing the LLM's computational load. To address this, we propose AVUR-LLM, an LLM-based Audio-Visual Speech Recognition via Sparse Modality Alignment and Visual Unit-Guided Refinement. Experiments on LRS3 demonstrate state-of-the-art results for AVSR. Under additive-noise conditions at 0 dB SNR, it achieves 37% relative improvement over the baseline system.

Figures & tables

Figure 1: Illustration of AVUR-LLM. (a) The Sparse Modality Alignment (SMA) module, audio-conditioned cross-attention sparsely inserted into the upper audio encoder layers; (b) The Adaptive Modulated Fusion (AMF) module, confidence-aware gating that adaptively modulates visual injection in the decoder; (c) The Visual Unit-Guided Refinement (VUR) module, discretizes mid-layer visual features into tokens for LLM rescoring. Training has two stages: Stage 1 performs SMA+AMF and outputs $N$ -best hypotheses; Stage 2 extracts and compresses $X_{v}^{\ell}$ into visual discrete tokens and uses a LoRA-adapted LLM to rescore the hypotheses. Here $\mathbf{X}_{a}$ and $\mathbf{X}_{v}$ denote audio and visual encoder features. $\hat{\mathbf{X}}_{v}$ denotes the SMA-refined visual features, $\mathbf{X}_{v}^{\ell}$ denotes visual features extracted from encoder layer $\ell$ . $\mathbf{X}_{a}^{\mathrm{sg}}$ denotes the audio features serving as keys/values with stop-gradient. $g_{\mathrm{AMF}}$ denotes the AMF gate coefficients.

Method	Audio Encoder	Visual Encoder	Training Data (h)	WER(%) $\downarrow$
Audio-Only
CM-seq2seq [ 31 ]	Conformer	–	433	2.3
Fast Conformer [ 32 ]	Conformer	–	435	1.6
AV-HuBERT [ 7 ]	Transformer	–	433	1.6
Whisper-finetuned [ 12 ]	Transformer	–	433	2.3
Auto-AVSR [ 9 ]	Conformer	–	1902/3448	1.0/1.0

Table 1: WER (%) on LRS3 for AVUR-LLM and previous methods in ASR and AVSR under 30 h, 433 h, and LRS3+VoxCeleb2 (1759 h) settings.

Method	Modality	Clean	SNR Level (dB) , WER (%) $\downarrow$
Method	Modality	Clean	10	5	0	-5	-10
AV-HuBERT [ 7 ]	AV	1.4	2.0	2.6	5.8	16.6	34.9
CMA [ 33 ]		1.5	1.8	2.4	4.4	11.9	25.8
Whisper-Flamingo [ 12 ]		1.1	1.4	2.0	6.3	28.0	42.6
AVGER [ 20 ]		1.1	–	1.9	3.7	12.3	29.1
Llama-AVSR [ 15 ]		0.95	–	2.2	4.2	16.9	–

Table 2: WER (%) on LRS3 (433 h setting) under clean and noisy conditions. We report AVSR results at SNRs {10, 5, 0,

-5

-10

} dB. “–” indicates not reported.

Figure 2: Effect of visual discrete tokens extraction depth and codebook size on WER (%). Solid lines denote clean, dashed lines denote 0 dB SNR. Blue curves denote K-means codebook size $K{=}1000$ , red curves denote $K{=}2000$ .

Method	SMA	AMF	VUR	Clean	Noisy
Whisper-Flamingo [ 12 ]	–	–	–	1.10	6.30
Llama-AVSR [ 15 ]	–	–	–	0.95	4.20
Staged ablation
AVUR-LLM (full)	✓	✓	✓	0.75	1.70
– w/o VUR	✓	✓	✗	0.97	5.50
– w/o AMF (SMA only)	✓	✗	✗	1.30	6.70

Table 3: Ablation of AVUR-LLM on LRS3 (433 h) under clean and noisy (0 dB SNR) conditions.

Explore similar work

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition
Jul 10, 2026Xugang Lu, Peng Shen, Yu Tsao +1Automatic Speech Recognition Audio-Visual Speech Recognition
National Institute of Information and Communications Technology, Kyoto, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan
VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition
Jun 28, 2026Piyush Arora, Navlika Singh, Umberto Cappellazzo +2Automatic Speech Recognition Audio-Visual Speech Recognition
Imperial College London, UK · NatWest AI Research, UK
LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognition
Apr 30, 2026Doyeop Kwak, Jeongsoo Choi, Suyeon Lee +1ASR Evaluation Automatic Speech Recognition
Korea Advanced Institute of Science and Technology, South Korea

Jul 10, 2026J/K move · Enter open · S save

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

Xugang Lu, Peng Shen, Yu Tsao, Hisashi Kawai

National Institute of Information and Communications Technology, Kyoto, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan

Automatic Speech Recognition Audio-Visual Speech Recognition Audio-Visual Alignment Speech Language Models

Open paper AskPDF arXiv

Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).

Abstract

Explore similar work

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition

LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognition