eess.ASMar 12, 2026

Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition

Authors: Umberto Cappellazzo, Stavros Petridis, Maja Pantic

Organizations: Imperial College London, UK · NatWest AI Research, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.

Figures & tables

Figure 1: WER comparison ( ↓\downarrow ) when ablating audio or video modalities across six state-of-the-art AVSR models in clean conditions. Note that the x-axis is displayed on a logarithmic scale.
Figure 2: Overview of the three proposed SHAP-based analyses in Dr. SHAP-AV. From the Shapley matrix Φ\bm{\Phi} , which captures the contribution of each input feature (rows) to each generated token (columns), we compute: (Left) Global SHAP , which aggregates contributions across all features and tokens to quantify overall modality balance; (Middle) Generative SHAP , which tracks modality contribution dynamics across token generation stages; and (Right) Temporal Alignment SHAP , which examines the correspondence between input feature positions and output token positions.
Figure 3: (Left) : Global audio/video contributions using Permutation SHAP for six AVSR models under varying acoustic conditions on the LRS3 dataset. (Right) : The same analysis using Sampling SHAP .
MethodSNR (dB)
∞\infty1050-5-10
Llama-AVSR [ 25 ]65.464.563.362.252.746.0
Omni-AVSR [ 28 ]59.959.958.257.348.842.7
Auto-AVSR [ 7 ]56.556.456.456.355.955.3
Whisper-Flamingo [ 17 ]94.794.594.293.190.288.3
Table 1: Global audio contributions for four AVSR models under varying acoustic conditions on the LRS2 dataset.
Figure 4: Generative SHAP analysis showing modality contributions as a function of token generation progress (%). Clean and noisy ( −10-10 dB) conditions are compared.
Figure 5: Temporal Alignment SHAP for AV-HuBERT. Top : audio feature heatmaps under clean (left) and noisy (right) conditions. Bottom : grouped video feature analysis (early/middle/late) under clean (left) and noisy (right) conditions.
Figure 6: Global SHAP audio contribution across acoustic noise types. Numbers above bars indicate the achieved WER (%).
Figure 7: Global audio SHAP contributions as a function of input duration (seconds) for Omni-AVSR, Whisper-Flamingo, and AV-HuBERT in clean and noisy conditions. Marginal distributions are shown on top and right axes.
Figure 8: Mean global audio contribution across WER ranges for Omni-AVSR (top) and Whisper-Flamingo (bottom) under 2.52.5 dB and −10-10 dB conditions. Error bars indicate standard deviation; sample counts are annotated above each bar.

Explore similar work

CardsList

  1. M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

    Jun 4, 2026Fei Su, Cancan Li, Ming Li +1Multimodal RobustnessAudio-Visual Representation Learning

    School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China

  2. Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning

    Mar 7, 2026Wenjie Tian, Mingchen Shao, Bingshen Mu +8Audio-Visual Speech RecognitionMultimodal CoT Reasoning

    Northwestern Polytechnical University, Xi’an, China

  3. Assessing True Generalisability of Audio-Visual Speech Recognisers

    Jun 5, 2026Zhaofeng Lin, Stavros Petridis, Maja Pantic +1Automatic Speech RecognitionAudio-Visual Speech Recognition

    Trinity College Dublin, Ireland · Imperial College London, UK