Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.
Figures & tables
Figure 1: Illustration of AVQA task and comparison between prior methods and QSTar. (a) Input sample (audio, video, and question) for an AVQA task. (b) Prior works rely on object-level cues, struggling with subtle motions ( e.g., inactive flute player). (c) QSTar enhances spatial–temporal–frequency interaction. Green patches highlight spatial focus. The red box shows diminishing high-frequency bands as the flute stops, while the violin remains active.
Figure 2: Overall framework of the proposed QSTar method. We use pre-trained encoders to extract audio, visual, and linguistic features, Fa , Fv , Fw , respectively. The Query-Guided Multimodal Correlation module (QGMC in yellow area) refines Fa and Fv using query information, resulting in Faq′ and Fvq′ . These features are further enhanced by the Spatial–Temporal–Frequency Interaction module (STFI in purple area), which integrates Spatial–Temporal Interaction (STI) and Temporal–Frequency Interaction (TFI), using additional frequency-aware features from AST ( Gong et al., 2021 ) . The Query Context Reasoning block (QCR in green area) incorporates prompt-based context ( Fprompt ) and sentence-level linguistic features ( Fsentence ) to guide multimodal fusion for answer prediction. For brevity, we remove self-attention units.
Method
Audio QA
Visual QA
Audio-Visual QA
Avg
Count.
Comp.
Avg
Count.
Local.
Avg
Exist.
Count.
Local.
Comp.
Temp.
Avg
FCN ( Fayek & Johnson, 2020 )
70.80
65.66
68.90
64.58
48.08
56.23
82.29
59.92
46.20
62.94
47.45
60.42
60.81
MCAN ( Yu et al., 2019 )
78.07
57.74
70.58
71.76
71.76
71.76
80.77
65.22
54.57
56.77
46.84
61.52
65.83
PSAC ( Li et al., 2019 )
75.02
66.84
72.00
68.00
70.78
69.41
79.76
61.66
55.22
61.13
59.85
63.60
66.62
HME ( Fan et al., 2019 )
73.65
63.74
69.89
67.42
70.20
68.83
80.87
63.64
54.89
63.03
60.58
64.78
66.75
AVSD ( Schwartz et al., 2019 )
72.47
62.46
68.78
66.00
74.53
70.31
80.77
64.03
57.93
62.85
61.07
65.44
67.32
Table 1: Comparison with existing methods on the MUSIC-AVQA ( Li et al., 2022 ) test set, reporting accuracy (%) across different question types. The short names are the abbreviations for question types “Counting", “Comparative", “Location", “Existential", and “Temporal", respectively. For space, FCN-LSTM, VideoLLaMA2, and Qwen2.5-Omni are denoted as FCN, VLLaMa2, and Qwen-Omni, respectively. The best results are bold while the second best are underlined .
Table 4
Method
Trainable Parameters (M)
FLOPs (G)
Avg Accuracy (%)
AVST ( Li et al., 2022 )
18.48
3.19
71.59
LAVISH ( Lin et al., 2023 )
21.09
-
74.46
TSPM ( Li et al., 2024a )
6.22
1.42
76.79
QA-TIGER ( Kim et al., 2025 )
14.51
2.70
77.62
QSTar ( w/o TFI)
11.95
2.15
77.41
QSTar (ours)
13.20
2.43
78.98
Table 4: Efficiency analysis and comparison on MUSIC-AVQA test set.
Figure 3: Qualitative results in the MUSIC-AVQA ( Li et al., 2022 ) dataset. (a) compares answer predictions between QSTar and QA-TIGER ( Kim et al., 2025 ) on two examples. (b) visualizes spatial, temporal, and frequency focuses from QSTar’s STFI module. Green boxes highlight the patch-level visual attention at key timestamps selected via audio-based temporal focus. A 2D spectrogram provides an overview of frequency dynamics for better interpretability.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Prompting
Total Accuracy (%)
PSAC ( Li et al., 2019 )
87.4
HCRN ( Le et al., 2020 )
89.0
PSTP ( Li et al., 2023a )
90.2
TSPM ( Li et al., 2024a )
✓
90.8
QSTar (ours)
90.9
QSTar (ours)
✓
91.2
Appendix
Table 5: Comparison with existing methods on the AVQA ( Yang et al., 2022 ) test set.
Method
Audio QA
Visual QA
Audio-Visual QA
Avg
a
78.03
83.69
74.43
77.52
b
80.20
83.86
75.06
78.30
c
80.07
84.06
76.06
78.89
d (ours)
80.63
84.17
75.98
78.98
Appendix
Table 6: Ablation study on early-stage question-guided feature processing. (a) integrates audio and visual features to form sounding visual features (AVST ( Li et al., 2022 ) ); (b) applies separate cross-attention between linguistic and each modality; (c) uses dual-branch sequential cross-attention (QA-TIGER ( Kim et al., 2025 ) ); (d) our proposed query-guided multimodal correlation (QGMC) module.
Method
Audio QA
Visual QA
Audio-Visual QA
Avg
w/o Fp
79.27
82.74
75.08
77.88
w/o Fast
78.90
83.24
74.53
77.61
STFI (ours)
80.63
84.17
75.98
78.98
Appendix
Table 7: Ablation study on the feature usage in the spatial–temporal–frequency interaction module.
Question Types
Sample Questions
Focuses
Counting
Is there a clarinet sound?
type & loudness
How many musical instruments were heard throughout the video?
type & loudness
How many types of musical instruments were heard throughout the video?
type & loudness
Is there a violin in the entire video?
type
Are there drum and piano instruments in the video?
type
How many types of musical instruments appeared in the entire video?
type
Appendix
Table 8: Overview of the five question types and 33 sample questions from MUSIC-AVQA ( Li et al., 2022 ) , along with their summarized focus areas ( i.e., type, duration, sequence, location, and loudness) for the answering task.
Method
Audio QA
Visual QA
Audio-Visual QA
Avg
w/o Prompts
79.39
83.32
75.47
78.25
w Translation
79.45
83.77
75.59
78.44
w Caption
79.33
84.35
75.06
78.28
w Generative Prompts
79.21
84.56
75.16
78.37
QCR (ours)
80.63
84.17
75.98
78.98
Appendix
Table 9: Ablation study on alternative prompting strategies for query-context construction. Translation refers to question-to-statement conversions used in TSPM ( Li et al., 2024a ) ; Caption denotes video-level descriptions generated by a pretrained captioning model ( Zhou et al., 2024 ) ; Generative Prompts correspond to constrained attribute-oriented texts produced by GPT-4o ( Hurst et al., 2024 ) .
Method
Audio QA
Visual QA
Audio-Visual QA
Avg
w Fast
78.96
83.94
75.12
78.14
w Fa (ours)
80.63
84.17
75.98
78.98
w Fast & Fa
79.83
84.19
75.65
78.65
Appendix
Table 10: Ablation study on the temporal audio pathway.
Figure 4: Prediction results of our method for different question types of Audio QA (counting, comparative) and Visual QA (counting, location) in MUSIC-AVQA ( Li et al., 2022 ) , with video ids: “00007961", “00002646", “00002454", and “00004109", respectively. We provide the top 3 answers predicted by our method in sequence.
Figure 5: Prediction results of our method for different question types of Audio-Visual QA (existential, counting, location, comparative, temporal) in MUSIC-AVQA ( Li et al., 2022 ) , with video ids: “00003803", “00004995", “00008437", “00005464", and “00004026", respectively. We provide the top 3 answers predicted by our method in sequence.
A common approach to adding audio to a vision-language model is to train or adapt a large omni-modal system. We show that a lightweight alternative can be highly effective for music audio-visual question answering (AVQA). Qwen-MusicAVQA-7B connects a frozen Whisper encoder to Qwen2-VL-7B-Instruct through learned linear projections. The same frozen encoder processes both the video's music track and a TTS-spoken question through separate projectors, while the language model fuses visual frames, music, and question audio through pretrained self-attention, with no task-specific fusion network. On MUSIC-AVQA, our system reaches 96.0% +/- 3.9% accuracy across three independent training seeds on the 7,402-question available-video test subset. Our central finding is that downstream accuracy tracks how much fine-grained local temporal information the audio representation preserves. In a matched 32-token comparison, a stride-pooled Whisper frame sequence outperforms a globally pooled PANNs representation expanded to the same budget by 26 percentage points, even though PANNs sees at least as much audio and uses a far larger projector. The effect is not simply sequence versus vector: within Whisper alone, reducing temporal resolution at a fixed token budget costs a comparable amount. Under matched data and inputs, fine-tuned Qwen2.5-Omni-7B reaches 80.9%, against 95.9% for our 30 s variant; because the systems differ in backbone and adaptation, this is a system-level comparison. Accuracy remains high on sampled head and tail splits of the rephrased MUSIC-AVQA-R benchmark (96.5% and 95.6%). Because both encoders stay frozen and the music features are cached, the entire adaptation is cheap to train: the complete two-stage AVQA run takes approximately 5 hours on a single A100 80GB, and every run reported here fits on that one GPU.
Audio-Visual Question Answering (AVQA) extends classical VQA by requiring joint reasoning over video and synchronized audio. However, many AVQA systems rely on deeply stacked layers of self- and cross attention across text, video, and audio. Such sequential stacking may incur loss of information such as subtle inter-modal cues over the layers, causing errors to accumulate across sequential attention layers during the fusion. We introduce Q-TriM which performs multi-modal fusion in a shallow and parallel manner instead of a deep and sequential manner. For Q-TriM, we propose a novel framework for attention operation incorporating video and audio conditioned on text. As a result, we obtain not only standard cross attention outputs but also Tri-Modal Attention representations in which Query, Key, and Value come from distinct modalities. These attention representations are combined in parallel at a single stage, thus avoiding the multi-modal fusion with deep stacks in order to mitigate error accumulation and depth-induced issues. Q-TriM achieves state-of-the-art performance on three AVQA benchmarks, including substantial gains on MUSIC-AVQA-R, which demonstrates its robustness and out-of-distribution generalization. Code is available at https://github.com/Sunghun95/Q-TriM
SungHun Kim, SeungJun Baek
Dept. of Computer Science and Engineering Korea University Seoul, South Korea
Current automated pipelines for audio-visual Question Answering (QA) generally adopt a ``video-caption-QA'' paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) Entity-Anchored Video Scripting transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) Clue-Guided QA Generation prompts models to first mine cross-segment, multimodal clues from the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct the instruction-tuning dataset OmniVideo-100K and a human-verified test set, OmniVideo-Test. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B on OmniVideo-100K yields performance gains of up to 20.59% on OmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks like Daily-Omni and JointAVBench.