Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

Jiayu Zhang; Shuo Ye; Qilang Ye; Zihan Song; Jiajian Huang; Zitong Yu

cs.CVApr 12, 2026

Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

Authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang, Zitong Yu

Organizations: School of Computing and Information Technology, Great Bay University · Tsinghua Shenzhen International Graduate School, Tsinghua University · College of Computer Science, Nankai University · School of mathematics, Sun Yat-sen University ·

Abstract

Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R $^{2}$ ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R $^{2}$ ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.

Figures & tables

Figure 1: Comparison of traditional methods (top) and R 2 ScP (bottom). Traditional methods typically rely on modality imputation but often yield redundant common knowledge that lacks distinctiveness. In contrast, R 2 ScP preserves valuable modality-specific knowledge from the candidate pool while effectively suppressing the inherent noise knowledge.

Figure 2: Overview of the proposed R 2 ScP framework (when the audio modality is missing). (a) The CMR module retrieves candidate features from a unified semantic space, while the CAP mechanism acts as a semantic filter that refines the coarse retrieved features using the common knowledge between the visual and audio modalities. (b) The overall architecture processes available and purified representations for the answer decoder.

Figure 3: Two-stage training strategy sequentially performs expert pre-training and expert mixing optimization.

Method	Venue	Modalities			Music-AVQA (Audio-Visual)						AVQA
Method	Venue	A	V	Q	Exist	Localis	Count	Comp	Temp	Avg.	Avg.
AVQA Specialized Models
PSTP-Net	MM’23	$\bullet$	$\bullet$	$\bullet$	76.18	73.23	71.80	71.79	69.00	72.57	90.20
TSPM	MM’24	$\bullet$	$\bullet$	$\bullet$	82.19	71.85	76.21	65.76	71.17	73.51	90.80
SHMamba	TASLP’25	$\bullet$	$\bullet$	$\bullet$	82.89	67.93	72.65	61.31	68.37	70.64	90.80
QA-TIGER	CVPR’25	$\bullet$	$\bullet$	$\bullet$	83.10	72.50	78.58	63.94	69.59	73.74	–

Table 1: Comparison of different methods on Music-AVQA and AVQA dataset under various missing modality settings (

\circ

indicates a missing modality). A: audio, V: visual, Q: question. Exist, Localis, etc. represent the accuracy in the subtasks of the Music-AVQA dataset.

Modalities		Modules		MUSIC AVQA	AVQA
A	V	CMR	CAP	MUSIC AVQA	AVQA
$\bullet$	$\circ$			62.43	57.43
$\bullet$	$\circ$		✓	64.11	59.64
$\bullet$	$\circ$	✓		67.21	61.78
$\bullet$	$\circ$	✓	✓	69.37	63.25
$\circ$	$\bullet$			63.54	68.02

Table 2: Ablation on the different components of R 2 ScP.

Figure 4: Impact of purification budget $k$ and number of retrieved samples $n$ .

Method	Avg.
R 2 ScP (ours)	70.72
w/o modality-specific expert pretraining	68.98
w/o expert mixing training	64.21
w/o ranking loss	69.62

Table 3: Effectiveness of the two-stage training strategy. Results are the average performance on Music-AVQA for the visual and audio missing settings.

Figure 5: Generalization analysis on the Music-AVQA dataset across various missing rates

Corpus	Music-AVQA	AVQA
N/A	64.66	64.39
AVQA	66.51	69.19
VGGSound	66.92	69.64
Music-AVQA	70.72	65.72

Table 4: Impact of different retrieval corpora on the performance of Music-AVQA and AVQA datasets. Results are the average performance for the visual and audio missing settings.

Figure 6: Trends in expert loads during experts mixing training on Music-AVQA in the severely incomplete conditions.

Figure 7: t-SNE visualization of our model and other methods on the Music-AVQA dataset.

Audio-Visual QA

Audio-visual question answering (AVQA) combines sound and video to answer questions about events, speech, music, space, and intent.

See all 28 papers

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
May 11, 2026Xuanchen Li, Yuheng Lu, Chenrui Cui +6Cross-Modal Learning Multimodal QA
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China +1
Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering
Jul 4, 2026SungHun Kim, SeungJun BaekMultimodal QA Cross-Modal Attention
Dept. of Computer Science and Engineering Korea University Seoul, South Korea
ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion
Jul 8, 2025Yuhan Li, Wei Zhang, Juan Chen +3Cross-Modal Learning Missing-Modality Learning
National University of Defense Technology · The Fifth Electronic Research Institute of MIIT · University of Chinese Academy of Sciences +2

May 11, 2026J/K move · Enter open · S save

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang+5

Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China+1

Cross-Modal Learning Multimodal QA Multimodal Hallucination Audio-Visual QA Multimodal CoT Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16% on general AVQA benchmarks and 11.17% on a cross-modal hallucination benchmark.

Abstract

Audio-Visual QA

Explore similar work

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering

ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion