Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition

YoungChae Kim; Da-Hee Yang; Joon-Hyuk Chang

cs.SDAug 26, 2026

Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition

Authors: YoungChae Kim, Da-Hee Yang, Joon-Hyuk Chang

Organizations: Dept. Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · Dept. Electronic and Electrical Engineering, Hanyang University, Seoul, Republic of Korea

Abstract

Large language model (LLM)-based audio-visual speech recognition (AVSR) systems are robust under noise. Contrastive decoding (CD), originally introduced to stabilize LLM generation by contrasting a weaker model against a stronger one at inference time, adjusts predictions without additional training. In this work, we apply CD to AVSR by contrasting audio-only conditioning with full audio-visual conditioning within the same underlying model. However, using a fixed contrastive strength introduces a trade-off across noise levels: stronger intervention helps under severe noise but may over-correct reliable predictions in clean conditions. We propose reliability-aware scaling of CD for AVSR. Instead of using a fixed strength, we adaptively modulate the contrastive influence at each token based on reliability signals derived from attention dynamics and inter-model predictive divergence. Experiments on LRS3 show consistent improvements across clean and low-SNR conditions.

Figures & tables

Figure 1: Overall architecture of the proposed framework. Attention-based reliability signals and a JS divergence filter produce a token-level gate $w_{t}$ , which rescales the contrastive weight $\lambda$ into the effective intervention strength $\lambda_{t}^{(\mathrm{eff})}$ . The right panel shows correction examples under noisy conditions; red indicates errors and blue indicates correct predictions. The Ours example is a sample whose prediction matches the ground truth (GT).

Model(Size)	Test Domain	Method	Clean	0 dB	-5 dB	-10 dB	-15 dB	Avg. Improvement
Llama-AVSR (8B)	LRS3	A+V/A	0.0095/0.01	0.0367/0.0663	0.0945/0.2584	0.2358/0.7573	0.3723/1.0377	-
	LRS3	Ours	0.0082 ( +13.68% )	0.033 ( +10.08% )	0.0866 ( +8.36% )	0.2167 ( +8.10% )	0.3369 ( +9.51% )	+9.95%
	LRS2	A+V/A	0.0536/0.0643	0.0983/0.1527	0.1937/0.411	0.361/0.8575	0.4911/1.0739	-
	LRS2	Ours	0.0465 ( +13.25% )	0.0878 ( +10.68% )	0.1814 ( +6.35% )	0.338 ( +6.37% )	0.4518 ( +8.00% )	+8.93%
Omni-AVSR (1B)	LRS3	A+V/A	0.0142/0.0132	0.0495/0.0936	0.1175/0.3164	0.2509/0.803	0.3305/1.0441	-
	LRS3	Ours	0.0111 ( +21.83% )	0.0468 ( +5.45% )	0.1108 ( +5.70% )	0.2306 ( +8.09% )	0.3053 ( +7.62% )	+9.74%

Table 1: Performance comparison under different SNR conditions on LRS3 (in-domain) and LRS2 (OOD) test sets. Noise is injected using MUSAN at 0, -5, -10, and -15 dB. “A+V/A” denotes the WER obtained under full audio-visual conditioning (Expert) and audio-only conditioning (Amateur), respectively. Relative improvements are computed with respect to the AVSR baseline as

(\text{A+V}-\text{Ours})/\text{A+V}

, and the reported average improvement is calculated across all SNR conditions.

Figure 2: Token-level JS divergence analysis across SNR levels. The Gaussian filter center (

\mu_{\text{sweet}}

) is selected based on validation-set analysis to capture the onset of ASR unreliability. Here, AVSR denotes audio–visual conditioning, whereas ASR denotes audio-only conditioning.

contrastive weight $\lambda$	Clean	0 dB	-5 dB	-10 dB	-15 dB
Baseline (No CD)	0.0095	0.0367	0.0945	0.2358	0.3723
$\lambda=0.1$	0.0081	0.0332	0.0889	0.2187	0.3385
$\lambda=0.2$	0.0090	0.0353	0.0896	0.2160	0.3239
$\lambda=0.3$	0.0101	0.0372	0.0968	0.2143	0.3180
$\lambda=0.4$	0.0114	0.0410	0.1011	0.2030	0.3150
$\lambda=0.5$	0.0130	0.0463	0.1091	0.2237	0.3196

Table 2: Performance of CD under different fixed

\lambda

values without reliability gating. Gray shading marks the best result among static fixed-

\lambda

CD settings for each SNR condition.

Method	$E_{t}$	$H_{t}$	$JS_{t}$	Clean	0 dB	-5 dB	-10 dB	-15 dB
Baseline	-	-	-	0.0095	0.0367	0.0945	0.2358	0.3723
w/ CD + $E_{t}$	✓			0.0086	0.0339	0.0883	0.2189	0.3332
w/ CD + $H_{t}$		✓		0.0095	0.0364	0.0952	0.2147	0.3202
w/ CD + $JS_{t}$			✓	0.0082	0.0346	0.0883	0.218	0.322
Ours	✓	✓	✓	0.0082	0.033	0.0866	0.2167	0.3369

Table 3: Ablation study on Llama-AVSR (8B) under different SNR conditions. For all experiments involving CD, the parameter

\lambda

is fixed at 0.3.

Audio-Visual Speech Recognition

Audio-visual speech recognition combines speech audio and visible mouth movements to transcribe words, especially when sound is noisy or incomplete.

See all 25 papers

Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement
Mar 4, 2026Fei Su, Cancan Li, Juan Liu +3Robust ASR Audio-Visual Speech Recognition
School of Computer Science, Wuhan University, China · School of Artificial Intelligence, Wuhan University, China · AI Center, OPPO, China +2
VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition
Jun 28, 2026Piyush Arora, Navlika Singh, Umberto Cappellazzo +2Automatic Speech Recognition Audio-Visual Speech Recognition
Imperial College London, UK · NatWest AI Research, UK
Adaptive Perturbation Selection for Contrastive Audio Decoding
Jun 30, 2026Aaron Isidore Grace, Zhouyuan Huo, Weiran WangAudio-Language Model Evaluation Audio-Language Model Hallucination Detection
Department of Computer Science University of Iowa Iowa City, IA, USA · Google Mountain View, CA, USA

Mar 4, 2026J/K move · Enter open · S save

Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement

Fei Su, Cancan Li, Juan Liu, Wei Ju+2

School of Computer Science, Wuhan University, China · School of Artificial Intelligence, Wuhan University, China · AI Center, OPPO, China+2

Robust ASR Audio-Visual Speech Recognition Automatic Speech Recognition Audio-Visual Alignment Speech Language Models

Open paper AskPDF arXiv

Audio-Visual Speech Recognition (AVSR) integrates acoustic and visual information to enhance robustness in adverse acoustic conditions. Recent advances in Large Language Models (LLMs) have yielded competitive automatic speech recognition performance and shown effectiveness for AVSR. However, prior approaches project audio and visual features independently or apply shallow fusion, limiting cross-modal alignment and complementary exchange while increasing the LLM's computational load. To address this, we propose AVUR-LLM, an LLM-based Audio-Visual Speech Recognition via Sparse Modality Alignment and Visual Unit-Guided Refinement. Experiments on LRS3 demonstrate state-of-the-art results for AVSR. Under additive-noise conditions at 0 dB SNR, it achieves 37% relative improvement over the baseline system.

Abstract

Audio-Visual Speech Recognition

Explore similar work

Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement

VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition

Adaptive Perturbation Selection for Contrastive Audio Decoding