RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding
Authors: Muyi Sun, Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun
Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China. · Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology, Hong Kong, China. · Institute of Automation, Chinese Academy of Sciences, and also with the University of Chinese Academy of Sciences, Beijing, China
Audio-Visual Learning (AVL) is one fundamental task of multi-modality learning and embodied intelligence, displaying the vital role in scene understanding and interaction. However, previous researchers mostly focus on exploring downstream tasks from a coarse-grained perspective (e.g., audio-visual correspondence, sound source localization, and audio-visual event localization). Considering providing more specific scene perception details, we newly define a fine-grained Audio-Visual Learning task, termed Region-Aware Sound Source Understanding (RA-SSU), which aims to achieve region-aware, frame-level, and high-quality sound source understanding. To support this goal, we innovatively construct two corresponding datasets, i.e. fine-grained Music (f-Music) and fine-grained Lifescene (f-Lifescene), each containing annotated sound source masks and frame-by-frame textual descriptions. The f-Music dataset includes 3,976 samples across 22 scene types related to specific application scenarios, focusing on music scenes with complex instrument mixing. The f-Lifescene dataset contains 6,156 samples across 61 types representing diverse sounding objects in life scenarios. Moreover, we propose SSUFormer, a Sound-Source Understanding TransFormer benchmark that facilitates both the sound source segmentation and sound region description with a multi-modal input and multi-modal output architecture. Specifically, we design two modules for this framework, Mask Collaboration Module (MCM) and Mixture of Hierarchical-prompted Experts (MoHE), to respectively enhance the accuracy and enrich the elaboration of the sound source description. Extensive experiments are conducted on our two datasets to verify the feasibility of the task, evaluate the availability of the datasets, and demonstrate the superiority of the SSUFormer, which achieves SOTA performance on the Sound Source Understanding benchmark.
Figures & tables
Fig. 1: Comparisons of the Region-Aware Sound-Source Understanding task with several previous Audio-Visual Learning tasks. (a).Audio-Visual Correspondence. Holistic correspondence with audio-level and video-level alignment. (b).Sound Source Localization. Coarse-grained (spatial) and region-level sound source localization. (c).Audio-Visual Event Localization. Coarse-grained (temporal) event localization with frame boundaries. (d). Region-Aware Sound-Source Understanding. Fine-grained audio-visual learning with regional sound source localization and sound region description.
Fig. 2: Data samples from the created f-Music and f-Lifescene. The top three are from f-Music and down from f-Lifescene. For each sample, three frames are selected from 10 seconds. Each sample contains a 10s video and its frame-level sound source masks and descriptions. Specifically, the frame-by-frame description of a data portion varies with the change of the sounding object, as shown in the upper-right sample. (Zoom in for better details)
Datasets
Annotations
Tasks
CUAVE [ 29 ]
Vocabulary Lip Edge
Audio-Visual Speech Recognition
LRS2-BBC [ 30 ]
Speech Text Lip Movement
Audio-Visual Speech Recognition
VGG-Sound [ 40 ]
Scene Category
Audio-Visual Correspondence
VoxCeleb1/2 [ 32 ]
Facial Trajectory Speaker ID
Audio-Visual Speaker Recognition
Hollywood [ 41 ]
Action Category Video-level Caption
Audio-Visual Action Recognition
Kinetics [ 42 ]
Action Category Action Localization
Audio-Visual Action Recognition
TABLE I: The statistics of representative audio-visual datasets.
Fig. 3: The framework of the RA-SSU task. Multi-modal inputs are first processed by modality-specific encoders. Then the main networks are designed to align, interact, and integrate audio-visual features. Task Decoders realize the mask and description prediction combined with the initial representations.
Fig. 4: Data annotation process and labeling system for the proposed sound-source understanding task. In this system, the video data is first uploaded into the system. Then the SAM model is used to obtain the initial masks. Based on the initial masks, the TAM is used to gain the frame-level video masks. Finally, the masked region of the frame-level images is fed into the Chat-Univi to get the region-aware descriptions. (Zoom in for better details.)
Fig. 5: SSUFormer: fine-grained Sound-Source Understanding Benchmark. On the left of this architecture, the audio and video are fed into the encoders and mapped to the token representations. Then, the multi-modality features are fused with the attention mechanism in Fig.6. Next, the previous features are integrated into task decoders for mask and description generation. (Zoom in for better details.)
Fig. 6: Illustration of proposed Mixture-of-Hierarchical-prompted Experts (MoHE). The MoHE adaptively integrates the text embeddings and frame-wise visual features via a router to produce temporal consistency captions in the long sequence. (Zoom in for better details.)
Fig. 7: Mask Collaboration Module for task interaction. (a). The plain multi-task output without interaction. (b). Mask collaboration module introduces the interaction between these two tasks. As a fine-grained RA-SSU task, region-aware visual information will give more details for the text description. Thus, combining the multi-modality representations with more regional visual features will give more accurate fine-grained captions.
f-Music
Methods
mIoU
F-score
BLEU
ROUGE-L
METEOR
SSS [ 75 ] AAAI′24
0.5919
0.7597
-
-
-
AVSBench [ 76 ] IJCV′25
0.5663
0.7191
-
-
-
AVIS [ 77 ] CVPR′25
0.5968
0.7227
-
-
-
SSU-S
0.5987
0.7552
-
-
-
AVC [ 10 ] CVPR′19
-
-
0.2546
0.3650
0.2668
TABLE II: Quantitative comparisons with single-task models on f-Music. SSS: Sound Source Segmentation, AVC: Audio-Visual Caption, SSU-D: Sound-Source Understanding with Description Only, SSU-S: Sound-Source Understanding with Segmentation Only.
f-Lifescene
Methods
mIoU
F-score
BLEU
ROUGE-L
METEOR
SSS [ 75 ] AAAI′24
0.6340
0.7064
-
-
-
AVIS [ 77 ] CVPR′25
0.5968
0.5697
-
-
-
AVSBench [ 76 ] IJCV′25
0.5697
0.6029
-
-
-
SSU-S
0.6320
0.7256
-
-
-
AVC [ 10 ] CVPR′19
-
-
0.3590
0.3423
0.3618
TABLE III: Quantitative comparisons with single-task models on f-Lifescene. (Abbreviation similar to TABLE. III).
Methods
BLEU
ROUGE-L
METEOR
NExT-GPT [ 81 ] ICML′24
0.1207
0.1159
0.1569
ModaVerse [ 80 ] CVPR′24
0.1529
0.1476
0.1278
PG-Video-LLaVA [ 78 ] Arxiv
0.2300
0.2649
0.2260
RA-SSU Baseline
0.2661
0.3102
0.2540
SSUFormer
0.4676
0.5168
0.4479
TABLE IV: Comparison with multi-modal Large Models on sound object description task (average of two datasets).
Ablation Dataset: f-Lifescene
Methods
mIoU
F-score
BLEU
ROUGE-L
METEOR
Baseline
0.4892
0.6510
0.1331
0.1285
0.1052
+ Focal
0.5150
0.6862
0.3201
0.2878
0.4531
+ MCM
0.6132
0.6947
0.4429
0.4590
0.4274
+ RCC
0.6141
0.7087
0.4621
0.4728
0.4452
+ MoHE (SSU)
0.6530
0.7368
0.5290
0.5388
0.5050
TABLE V: Quantitative comparisons on ablation analysis.
Fig. 8: Experimental result samples of the SSUFormer on f-Music. The red-highlighted words mean unreasonable output. The green-highlighted words mean satisfactory output. (Zoom in for better details.)
Fig. 9: Experimental result samples of the SSUFormer on f-Lifescene. The red-highlighted words mean unreasonable output. The green words mean accurate perception. (Zoom in for better details.)
Fig. 10: Data samples of the ablation analysis. Due to layout limitations, we present two groups of comparative results in f-Lifescene with more category diversity. MCM and RCC simultaneously improved these two task performances; MoHE further improves the caption results. The red-highlighted words mean unreasonable output. (Zoom in for better details.)
Methods
Computational Efficiency
Params
FPS
FLOPS
Inference Time
AVSBench [ 22 ]
100.14M
251. 45
154.20G
0.039
AVIS [ 77 ]
527.3M
23. 72
2.65T
0.435
PG-Video-LLaVA [ 78 ]
6.76B
475.81
1.32T
0.635
NExT-GPT [ 81 ]
6.74B
455.86
1.62T
0.834
ModaVerse [ 80 ]
7.98B
359.72
1.70T
0.658
TABLE VI: Computational Efficiency of the proposed Method with comparisons. (Params: model parameters. FPS: frame rate inferred per second. FLOPS: floating point operations per second. Inference Time: inference time for a single sample.)
Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.
Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4
POSTECH · KAIST · Hankuk University of Foreign Studies +1
Audio-Visual Segmentation (AVS) aims to produce pixel-level masks of sound producing objects in videos, by jointly learning from audio and visual signals. However, real-world environments are inherently dynamic, causing audio and visual distributions to evolve over time, which challenge existing AVS systems that assume static training settings. To address this gap, we introduce the first exemplar-free continual learning benchmark for Audio-Visual Segmentation, comprising four learning protocols across single-source and multi-source AVS datasets. We further propose a strong baseline, ATLAS, which uses audio-guided pre-fusion conditioning to modulate visual feature channels via projected audio context before cross-modal attention. Finally, we mitigate catastrophic forgetting by introducing Low-Rank Anchoring (LRA), which stabilizes adapted weights based on loss sensitivity. Extensive experiments demonstrate competitive performance across diverse continual scenarios, establishing a foundation for lifelong audio-visual perception. Code is available at∗\footnote{Paper under review} - \hyperlink{https://gitlab.com/viper-purdue/atlas}{https://gitlab.com/viper-purdue/atlas} \keywords{Continual Learning \and Audio-Visual Segmentation \and Multi-Modal Learning}
Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn +1
Audio-Visual Segmentation (AVS) requires predicting pixel-wise masks of sounding objects by grounding acoustic signals in visual space. In multi-source scenes, concurrent sound sources often remain entangled within shared visual representations, leading to merged masks or unstable specialization when using standard alignment-focused transformer architectures. To address this, we present SlotAVS, an object-centric framework driven by competitive bi-modal slot attention. While robust foundation models provide rich semantic priors, SlotAVS shifts the paradigm from global cross-modal fusion to explicit competitive binding. By initializing discrete object slots from temporal audio sequences and enforcing strict slot-wise competition over spatial visual tokens, our method structurally mitigates the blending of distinct acoustic entities. Further, to stabilize instance-level learning, we introduce a Hungarian matching strategy that optimally aligns slots with ground-truth objects using spatial and boundary overlap. Coupled with a numerically stable, confidence-aware mask fusion mechanism, SlotAVS robustly aggregates multi-source predictions. Extensive experiments on the AVSBench dataset demonstrate the efficacy of our approach, achieving 88.6% mIoU on the Single-Source (S4) split and 71.7% mIoU on the Multi-Source (MS3) split. These results show that explicit competitive binding is critical for complex audio-visual scene understanding and effectively leveraging foundation-model priors without relying solely on their capacity.
Arnav Sharma, Anand Mishra, Renu Rameshan
Indian Institute of Technology Jodhpur · Vehant Technologies Pvt Ltd