Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering

Kun Li; Michael Ying Yang; Sami Sebastian Brandt

cs.CVJan 27, 2026

Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering

Authors: Kun Li, Michael Ying Yang, Sami Sebastian Brandt

Organizations: University of Twente, NL · University of Bath, UK · IT University of Copenhagen, DK

Abstract

Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of AVQA task and comparison between prior methods and QSTar. (a) Input sample (audio, video, and question) for an AVQA task. (b) Prior works rely on object-level cues, struggling with subtle motions ( e.g., inactive flute player). (c) QSTar enhances spatial–temporal–frequency interaction. Green patches highlight spatial focus. The red box shows diminishing high-frequency bands as the flute stops, while the violin remains active.

Figure 2: Overall framework of the proposed QSTar method. We use pre-trained encoders to extract audio, visual, and linguistic features, $F_{a}$ , $F_{v}$ , $F_{w}$ , respectively. The Query-Guided Multimodal Correlation module (QGMC in yellow area) refines $F_{a}$ and $F_{v}$ using query information, resulting in $F_{aq}^{\prime}$ and $F_{vq}^{\prime}$ . These features are further enhanced by the Spatial–Temporal–Frequency Interaction module (STFI in purple area), which integrates Spatial–Temporal Interaction (STI) and Temporal–Frequency Interaction (TFI), using additional frequency-aware features from AST ( Gong et al., 2021 ) . The Query Context Reasoning block (QCR in green area) incorporates prompt-based context ( $F_{\mathrm{prompt}}$ ) and sentence-level linguistic features ( $F_{\mathrm{sentence}}$ ) to guide multimodal fusion for answer prediction. For brevity, we remove self-attention units.

Method	Audio QA			Visual QA			Audio-Visual QA						Avg
	Count.	Comp.	Avg	Count.	Local.	Avg	Exist.	Count.	Local.	Comp.	Temp.	Avg	Avg
FCN ( Fayek & Johnson, 2020 )	70.80	65.66	68.90	64.58	48.08	56.23	82.29	59.92	46.20	62.94	47.45	60.42	60.81
MCAN ( Yu et al., 2019 )	78.07	57.74	70.58	71.76	71.76	71.76	80.77	65.22	54.57	56.77	46.84	61.52	65.83
PSAC ( Li et al., 2019 )	75.02	66.84	72.00	68.00	70.78	69.41	79.76	61.66	55.22	61.13	59.85	63.60	66.62
HME ( Fan et al., 2019 )	73.65	63.74	69.89	67.42	70.20	68.83	80.87	63.64	54.89	63.03	60.58	64.78	66.75
AVSD ( Schwartz et al., 2019 )	72.47	62.46	68.78	66.00	74.53	70.31	80.77	64.03	57.93	62.85	61.07	65.44	67.32

Table 1: Comparison with existing methods on the MUSIC-AVQA ( Li et al., 2022 ) test set, reporting accuracy (%) across different question types. The short names are the abbreviations for question types “Counting", “Comparative", “Location", “Existential", and “Temporal", respectively. For space, FCN-LSTM, VideoLLaMA2, and Qwen2.5-Omni are denoted as FCN, VLLaMa2, and Qwen-Omni, respectively. The best results are bold while the second best are underlined .

Table 4

Method	Trainable Parameters (M)	FLOPs (G)	Avg Accuracy (%)
AVST ( Li et al., 2022 )	18.48	3.19	71.59
LAVISH ( Lin et al., 2023 )	21.09	-	74.46
TSPM ( Li et al., 2024a )	6.22	1.42	76.79
QA-TIGER ( Kim et al., 2025 )	14.51	2.70	77.62
QSTar ( w/o TFI)	11.95	2.15	77.41
QSTar (ours)	13.20	2.43	78.98

Table 4: Efficiency analysis and comparison on MUSIC-AVQA test set.

Panel of Figure 3 — Figure 3: Qualitative results in the MUSIC-AVQA ( Li et al., 2022 ) dataset. (a) compares answer predictions between QSTar and QA-TIGER ( Kim et al., 2025 ) on two examples. (b) visualizes spatial, temporal, and frequency focuses from QSTar’s STFI module. Green boxes highlight the patch-level visual attention at key timestamps selected via audio-based temporal focus. A 2D spectrogram provides an overview of frequency dynamics for better interpretability.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Prompting	Total Accuracy (%)
PSAC ( Li et al., 2019 )		87.4
HCRN ( Le et al., 2020 )		89.0
PSTP ( Li et al., 2023a )		90.2
TSPM ( Li et al., 2024a )	✓	90.8
QSTar (ours)		90.9
QSTar (ours)	✓	91.2

Appendix

Table 5: Comparison with existing methods on the AVQA ( Yang et al., 2022 ) test set.

Method	Audio QA	Visual QA	Audio-Visual QA	Avg
a	78.03	83.69	74.43	77.52
b	80.20	83.86	75.06	78.30
c	80.07	84.06	76.06	78.89
d (ours)	80.63	84.17	75.98	78.98

Appendix

Table 6: Ablation study on early-stage question-guided feature processing. (a) integrates audio and visual features to form sounding visual features (AVST ( Li et al., 2022 ) ); (b) applies separate cross-attention between linguistic and each modality; (c) uses dual-branch sequential cross-attention (QA-TIGER ( Kim et al., 2025 ) ); (d) our proposed query-guided multimodal correlation (QGMC) module.

Method	Audio QA	Visual QA	Audio-Visual QA	Avg
w/o $F_{p}$	79.27	82.74	75.08	77.88
w/o $F_{ast}$	78.90	83.24	74.53	77.61
STFI (ours)	80.63	84.17	75.98	78.98

Appendix

Table 7: Ablation study on the feature usage in the spatial–temporal–frequency interaction module.

Question Types	Sample Questions	Focuses
Counting	Is there a clarinet sound?	type & loudness
	How many musical instruments were heard throughout the video?	type & loudness
	How many types of musical instruments were heard throughout the video?	type & loudness
	Is there a violin in the entire video?	type
	Are there drum and piano instruments in the video?	type
	How many types of musical instruments appeared in the entire video?	type

Appendix

Table 8: Overview of the five question types and 33 sample questions from MUSIC-AVQA ( Li et al., 2022 ) , along with their summarized focus areas ( i.e., type, duration, sequence, location, and loudness) for the answering task.

Method	Audio QA	Visual QA	Audio-Visual QA	Avg
w/o Prompts	79.39	83.32	75.47	78.25
w Translation	79.45	83.77	75.59	78.44
w Caption	79.33	84.35	75.06	78.28
w Generative Prompts	79.21	84.56	75.16	78.37
QCR (ours)	80.63	84.17	75.98	78.98

Appendix

Table 9: Ablation study on alternative prompting strategies for query-context construction. Translation refers to question-to-statement conversions used in TSPM ( Li et al., 2024a ) ; Caption denotes video-level descriptions generated by a pretrained captioning model ( Zhou et al., 2024 ) ; Generative Prompts correspond to constrained attribute-oriented texts produced by GPT-4o ( Hurst et al., 2024 ) .

Method	Audio QA	Visual QA	Audio-Visual QA	Avg
w $F_{ast}$	78.96	83.94	75.12	78.14
w $F_{a}$ (ours)	80.63	84.17	75.98	78.98
w $F_{ast}$ & $F_{a}$	79.83	84.19	75.65	78.65

Appendix

Table 10: Ablation study on the temporal audio pathway.

Panel of Figure 4 — Figure 4: Prediction results of our method for different question types of Audio QA (counting, comparative) and Visual QA (counting, location) in MUSIC-AVQA ( Li et al., 2022 ) , with video ids: “00007961", “00002646", “00002454", and “00004109", respectively. We provide the top 3 answers predicted by our method in sequence.

Panel of Figure 5 — Figure 5: Prediction results of our method for different question types of Audio-Visual QA (existential, counting, location, comparative, temporal) in MUSIC-AVQA ( Li et al., 2022 ) , with video ids: “00003803", “00004995", “00008437", “00005464", and “00004026", respectively. We provide the top 3 answers predicted by our method in sequence.

Multimodal QA

Multimodal question answering combines language with images, video, audio, or documents to answer questions using evidence across modalities.

See all 195 papers

Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA
Aug 11, 2026Maryam DehdashtiAudio-Visual QA
Inference Matter Labs
Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering
Jul 4, 2026SungHun Kim, SeungJun BaekMultimodal QA Cross-Modal Attention
Dept. of Computer Science and Engineering Korea University Seoul, South Korea
OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains
Jun 12, 2026Xinyue Cai, Chaoyou Fu, Yi-Fan Zhang +2Multimodal QA Audio-Visual QA
1Nanjing University · 2CASIA

Aug 11, 2026J/K move · Enter open · S save

Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA

Maryam Dehdashti

Inference Matter Labs

Audio-Visual QA

Open paper AskPDF arXiv

A common approach to adding audio to a vision-language model is to train or adapt a large omni-modal system. We show that a lightweight alternative can be highly effective for music audio-visual question answering (AVQA). Qwen-MusicAVQA-7B connects a frozen Whisper encoder to Qwen2-VL-7B-Instruct through learned linear projections. The same frozen encoder processes both the video's music track and a TTS-spoken question through separate projectors, while the language model fuses visual frames, music, and question audio through pretrained self-attention, with no task-specific fusion network. On MUSIC-AVQA, our system reaches 96.0% +/- 3.9% accuracy across three independent training seeds on the 7,402-question available-video test subset. Our central finding is that downstream accuracy tracks how much fine-grained local temporal information the audio representation preserves. In a matched 32-token comparison, a stride-pooled Whisper frame sequence outperforms a globally pooled PANNs representation expanded to the same budget by 26 percentage points, even though PANNs sees at least as much audio and uses a far larger projector. The effect is not simply sequence versus vector: within Whisper alone, reducing temporal resolution at a fixed token budget costs a comparable amount. Under matched data and inputs, fine-tuned Qwen2.5-Omni-7B reaches 80.9%, against 95.9% for our 30 s variant; because the systems differ in backbone and adaptation, this is a system-level comparison. Accuracy remains high on sampled head and tail splits of the rephrased MUSIC-AVQA-R benchmark (96.5% and 95.6%). Because both encoders stay frozen and the music features are cached, the entire adaptation is cheap to train: the complete two-stage AVQA run takes approximately 5 hours on a single A100 80GB, and every run reported here fits on that one GPU.

Abstract

Multimodal QA

Explore similar work

Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA

Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering

OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains