RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding
Authors: Muyi Sun, Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun
Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China. · Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology, Hong Kong, China. · Institute of Automation, Chinese Academy of Sciences, and also with the University of Chinese Academy of Sciences, Beijing, China
Audio-Visual Learning (AVL) is one fundamental task of multi-modality learning and embodied intelligence, displaying the vital role in scene understanding and interaction. However, previous researchers mostly focus on exploring downstream tasks from a coarse-grained perspective (e.g., audio-visual correspondence, sound source localization, and audio-visual event localization). Considering providing more specific scene perception details, we newly define a fine-grained Audio-Visual Learning task, termed Region-Aware Sound Source Understanding (RA-SSU), which aims to achieve region-aware, frame-level, and high-quality sound source understanding. To support this goal, we innovatively construct two corresponding datasets, i.e. fine-grained Music (f-Music) and fine-grained Lifescene (f-Lifescene), each containing annotated sound source masks and frame-by-frame textual descriptions. The f-Music dataset includes 3,976 samples across 22 scene types related to specific application scenarios, focusing on music scenes with complex instrument mixing. The f-Lifescene dataset contains 6,156 samples across 61 types representing diverse sounding objects in life scenarios. Moreover, we propose SSUFormer, a Sound-Source Understanding TransFormer benchmark that facilitates both the sound source segmentation and sound region description with a multi-modal input and multi-modal output architecture. Specifically, we design two modules for this framework, Mask Collaboration Module (MCM) and Mixture of Hierarchical-prompted Experts (MoHE), to respectively enhance the accuracy and enrich the elaboration of the sound source description. Extensive experiments are conducted on our two datasets to verify the feasibility of the task, evaluate the availability of the datasets, and demonstrate the superiority of the SSUFormer, which achieves SOTA performance on the Sound Source Understanding benchmark.
Figures & tables
Fig. 1: Comparisons of the Region-Aware Sound-Source Understanding task with several previous Audio-Visual Learning tasks. (a).Audio-Visual Correspondence. Holistic correspondence with audio-level and video-level alignment. (b).Sound Source Localization. Coarse-grained (spatial) and region-level sound source localization. (c).Audio-Visual Event Localization. Coarse-grained (temporal) event localization with frame boundaries. (d). Region-Aware Sound-Source Understanding. Fine-grained audio-visual learning with regional sound source localization and sound region description.
Fig. 2: Data samples from the created f-Music and f-Lifescene. The top three are from f-Music and down from f-Lifescene. For each sample, three frames are selected from 10 seconds. Each sample contains a 10s video and its frame-level sound source masks and descriptions. Specifically, the frame-by-frame description of a data portion varies with the change of the sounding object, as shown in the upper-right sample. (Zoom in for better details)
Datasets
Annotations
Tasks
CUAVE [ 29 ]
Vocabulary Lip Edge
Audio-Visual Speech Recognition
LRS2-BBC [ 30 ]
Speech Text Lip Movement
Audio-Visual Speech Recognition
VGG-Sound [ 40 ]
Scene Category
Audio-Visual Correspondence
VoxCeleb1/2 [ 32 ]
Facial Trajectory Speaker ID
Audio-Visual Speaker Recognition
Hollywood [ 41 ]
Action Category Video-level Caption
Audio-Visual Action Recognition
Kinetics [ 42 ]
Action Category Action Localization
Audio-Visual Action Recognition
TABLE I: The statistics of representative audio-visual datasets.
Fig. 3: The framework of the RA-SSU task. Multi-modal inputs are first processed by modality-specific encoders. Then the main networks are designed to align, interact, and integrate audio-visual features. Task Decoders realize the mask and description prediction combined with the initial representations.
Fig. 4: Data annotation process and labeling system for the proposed sound-source understanding task. In this system, the video data is first uploaded into the system. Then the SAM model is used to obtain the initial masks. Based on the initial masks, the TAM is used to gain the frame-level video masks. Finally, the masked region of the frame-level images is fed into the Chat-Univi to get the region-aware descriptions. (Zoom in for better details.)
Fig. 5: SSUFormer: fine-grained Sound-Source Understanding Benchmark. On the left of this architecture, the audio and video are fed into the encoders and mapped to the token representations. Then, the multi-modality features are fused with the attention mechanism in Fig.6. Next, the previous features are integrated into task decoders for mask and description generation. (Zoom in for better details.)
Fig. 6: Illustration of proposed Mixture-of-Hierarchical-prompted Experts (MoHE). The MoHE adaptively integrates the text embeddings and frame-wise visual features via a router to produce temporal consistency captions in the long sequence. (Zoom in for better details.)
Fig. 7: Mask Collaboration Module for task interaction. (a). The plain multi-task output without interaction. (b). Mask collaboration module introduces the interaction between these two tasks. As a fine-grained RA-SSU task, region-aware visual information will give more details for the text description. Thus, combining the multi-modality representations with more regional visual features will give more accurate fine-grained captions.
f-Music
Methods
mIoU
F-score
BLEU
ROUGE-L
METEOR
SSS [ 75 ] AAAI′24
0.5919
0.7597
-
-
-
AVSBench [ 76 ] IJCV′25
0.5663
0.7191
-
-
-
AVIS [ 77 ] CVPR′25
0.5968
0.7227
-
-
-
SSU-S
0.5987
0.7552
-
-
-
AVC [ 10 ] CVPR′19
-
-
0.2546
0.3650
0.2668
TABLE II: Quantitative comparisons with single-task models on f-Music. SSS: Sound Source Segmentation, AVC: Audio-Visual Caption, SSU-D: Sound-Source Understanding with Description Only, SSU-S: Sound-Source Understanding with Segmentation Only.
f-Lifescene
Methods
mIoU
F-score
BLEU
ROUGE-L
METEOR
SSS [ 75 ] AAAI′24
0.6340
0.7064
-
-
-
AVIS [ 77 ] CVPR′25
0.5968
0.5697
-
-
-
AVSBench [ 76 ] IJCV′25
0.5697
0.6029
-
-
-
SSU-S
0.6320
0.7256
-
-
-
AVC [ 10 ] CVPR′19
-
-
0.3590
0.3423
0.3618
TABLE III: Quantitative comparisons with single-task models on f-Lifescene. (Abbreviation similar to TABLE. III).
Methods
BLEU
ROUGE-L
METEOR
NExT-GPT [ 81 ] ICML′24
0.1207
0.1159
0.1569
ModaVerse [ 80 ] CVPR′24
0.1529
0.1476
0.1278
PG-Video-LLaVA [ 78 ] Arxiv
0.2300
0.2649
0.2260
RA-SSU Baseline
0.2661
0.3102
0.2540
SSUFormer
0.4676
0.5168
0.4479
TABLE IV: Comparison with multi-modal Large Models on sound object description task (average of two datasets).
Ablation Dataset: f-Lifescene
Methods
mIoU
F-score
BLEU
ROUGE-L
METEOR
Baseline
0.4892
0.6510
0.1331
0.1285
0.1052
+ Focal
0.5150
0.6862
0.3201
0.2878
0.4531
+ MCM
0.6132
0.6947
0.4429
0.4590
0.4274
+ RCC
0.6141
0.7087
0.4621
0.4728
0.4452
+ MoHE (SSU)
0.6530
0.7368
0.5290
0.5388
0.5050
TABLE V: Quantitative comparisons on ablation analysis.
Fig. 8: Experimental result samples of the SSUFormer on f-Music. The red-highlighted words mean unreasonable output. The green-highlighted words mean satisfactory output. (Zoom in for better details.)
Fig. 9: Experimental result samples of the SSUFormer on f-Lifescene. The red-highlighted words mean unreasonable output. The green words mean accurate perception. (Zoom in for better details.)
Fig. 10: Data samples of the ablation analysis. Due to layout limitations, we present two groups of comparative results in f-Lifescene with more category diversity. MCM and RCC simultaneously improved these two task performances; MoHE further improves the caption results. The red-highlighted words mean unreasonable output. (Zoom in for better details.)
Methods
Computational Efficiency
Params
FPS
FLOPS
Inference Time
AVSBench [ 22 ]
100.14M
251. 45
154.20G
0.039
AVIS [ 77 ]
527.3M
23. 72
2.65T
0.435
PG-Video-LLaVA [ 78 ]
6.76B
475.81
1.32T
0.635
NExT-GPT [ 81 ]
6.74B
455.86
1.62T
0.834
ModaVerse [ 80 ]
7.98B
359.72
1.70T
0.658
TABLE VI: Computational Efficiency of the proposed Method with comparisons. (Params: model parameters. FPS: frame rate inferred per second. FLOPS: floating point operations per second. Inference Time: inference time for a single sample.)