Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification
Authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang, Zitong Yu
Organizations: School of Computing and Information Technology, Great Bay University · Tsinghua Shenzhen International Graduate School, Tsinghua University · College of Computer Science, Nankai University · School of mathematics, Sun Yat-sen University · Dongguan Key Laboratory for Intelligence and Information Technology
Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R2ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R2ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.
Figures & tables
Figure 1: Comparison of traditional methods (top) and R 2 ScP (bottom). Traditional methods typically rely on modality imputation but often yield redundant common knowledge that lacks distinctiveness. In contrast, R 2 ScP preserves valuable modality-specific knowledge from the candidate pool while effectively suppressing the inherent noise knowledge.
Figure 2: Overview of the proposed R 2 ScP framework (when the audio modality is missing). (a) The CMR module retrieves candidate features from a unified semantic space, while the CAP mechanism acts as a semantic filter that refines the coarse retrieved features using the common knowledge between the visual and audio modalities. (b) The overall architecture processes available and purified representations for the answer decoder.
Figure 3: Two-stage training strategy sequentially performs expert pre-training and expert mixing optimization.
Method
Venue
Modalities
Music-AVQA (Audio-Visual)
AVQA
A
V
Q
Exist
Localis
Count
Comp
Temp
Avg.
Avg.
AVQA Specialized Models
PSTP-Net
MM’23
∙
∙
∙
76.18
73.23
71.80
71.79
69.00
72.57
90.20
TSPM
MM’24
∙
∙
∙
82.19
71.85
76.21
65.76
71.17
73.51
90.80
SHMamba
TASLP’25
∙
∙
∙
82.89
67.93
72.65
61.31
68.37
70.64
90.80
QA-TIGER
CVPR’25
∙
∙
∙
83.10
72.50
78.58
63.94
69.59
73.74
–
Table 1: Comparison of different methods on Music-AVQA and AVQA dataset under various missing modality settings ( ∘ indicates a missing modality). A: audio, V: visual, Q: question. Exist, Localis, etc. represent the accuracy in the subtasks of the Music-AVQA dataset.
Modalities
Modules
MUSIC AVQA
AVQA
A
V
CMR
CAP
∙
∘
62.43
57.43
∙
∘
✓
64.11
59.64
∙
∘
✓
67.21
61.78
∙
∘
✓
✓
69.37
63.25
∘
∙
63.54
68.02
Table 2: Ablation on the different components of R 2 ScP.
Figure 4: Impact of purification budget k and number of retrieved samples n .
Method
Avg.
R 2 ScP (ours)
70.72
w/o modality-specific expert pretraining
68.98
w/o expert mixing training
64.21
w/o ranking loss
69.62
Table 3: Effectiveness of the two-stage training strategy. Results are the average performance on Music-AVQA for the visual and audio missing settings.
Figure 5: Generalization analysis on the Music-AVQA dataset across various missing rates
Corpus
Music-AVQA
AVQA
N/A
64.66
64.39
AVQA
66.51
69.19
VGGSound
66.92
69.64
Music-AVQA
70.72
65.72
Table 4: Impact of different retrieval corpora on the performance of Music-AVQA and AVQA datasets. Results are the average performance for the visual and audio missing settings.
Figure 6: Trends in expert loads during experts mixing training on Music-AVQA in the severely incomplete conditions.
Figure 7: t-SNE visualization of our model and other methods on the Music-AVQA dataset.
Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16% on general AVQA benchmarks and 11.17% on a cross-modal hallucination benchmark.
Xuanchen Li, Yuheng Lu, Chenrui Cui +6
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China +1
Audio-Visual Question Answering (AVQA) extends classical VQA by requiring joint reasoning over video and synchronized audio. However, many AVQA systems rely on deeply stacked layers of self- and cross attention across text, video, and audio. Such sequential stacking may incur loss of information such as subtle inter-modal cues over the layers, causing errors to accumulate across sequential attention layers during the fusion. We introduce Q-TriM which performs multi-modal fusion in a shallow and parallel manner instead of a deep and sequential manner. For Q-TriM, we propose a novel framework for attention operation incorporating video and audio conditioned on text. As a result, we obtain not only standard cross attention outputs but also Tri-Modal Attention representations in which Query, Key, and Value come from distinct modalities. These attention representations are combined in parallel at a single stage, thus avoiding the multi-modal fusion with deep stacks in order to mitigate error accumulation and depth-induced issues. Q-TriM achieves state-of-the-art performance on three AVQA benchmarks, including substantial gains on MUSIC-AVQA-R, which demonstrates its robustness and out-of-distribution generalization. Code is available at https://github.com/Sunghun95/Q-TriM
SungHun Kim, SeungJun Baek
Dept. of Computer Science and Engineering Korea University Seoul, South Korea
Multimodal emotion and intent recognition is essential for automated human-computer interaction, It aims to analyze users' speech, text, and visual information to predict their emotions or intent. One of the significant challenges is that missing modalities due to sensor malfunctions or incomplete data. Traditional methods that attempt to reconstruct missing information often suffer from over-coupling and imprecise generation processes, leading to suboptimal outcomes. To address these issues, we introduce an Attention-based Diffusion model for Missing Modalities feature Completion (ADMC). Our framework independently trains feature extraction networks for each modality, preserving their unique characteristics and avoiding over-coupling. The Attention-based Diffusion Network (ADN) generates missing modality features that closely align with authentic multimodal distribution, enhancing performance across all missing-modality scenarios. Moreover, ADN's cross-modal generation offers improved recognition even in full-modality contexts. Our approach achieves state-of-the-art results on the IEMOCAP and MIntRec benchmarks, demonstrating its effectiveness in both missing and complete modality scenarios.
Yuhan Li, Wei Zhang, Juan Chen +3
National University of Defense Technology · The Fifth Electronic Research Institute of MIIT · University of Chinese Academy of Sciences +2