AV-Unified: A Unified Framework for Audio-visual Scene Understanding

Guangyao Li; Xin Wang; Wenwu Zhu

cs.CVMar 6, 2026

AV-Unified: A Unified Framework for Audio-visual Scene Understanding

Authors: Guangyao Li, Xin Wang, Wenwu Zhu

Organizations: Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · Beijing National Research Center for Information Science and Technology, Beijing 100084, China

Abstract

When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing, segmentation and question answering are mostly explored individually, making it challenging to comprehensively understand complex audio-visual scenes and explore inter-task relationships. Hence, we propose AV-Unified, a unified framework that enables joint learning across a wide range of audio-visual scene understanding tasks. AV-Unified standardizes the diverse input-output formats of each task and incorporates a multi-scale spatiotemporal perception network to effectively capture audio-visual associations. Specifically, we unify the inputs and outputs of all supported tasks by converting them into sequences of discrete tokens, establishing a shared representation that allows a single architecture to be trained jointly across heterogeneous varied datasets. Considering the varying temporal granularity of audio-visual events, a multi-scale temporal perception module is designed to capture key cues. Meanwhile, to overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module that models spatial audio-visual associations. Furthermore, task-specific text prompts are employed to enhance the model's adaptability and task-awareness. Extensive experiments on benchmark datasets (e.g., AVE, LLP, MUSIC-AVQA, VGG-SS and AVS) demonstrate the effectiveness of AV-Unified across temporal, spatial, and spatiotemporal tasks.

Figures & tables

Fig. 1: AV-Unified is a single sequence-to-sequence model that performs a variety of audio-visual tasks using a unified architecture without a need for either task or modality specific branches. A schematic of the model with multiple demonstrative audio-visual tasks: event localization, video parsing, sound source localization, segmentation and question answering.

Fig. 2: The proposed Multi-scale Temporal-Spatial Perception Framework. First, the visual and audio features extracted by the encoder are fed into a temporal perception module to capture key audio-visual temporal cues. Then, a spatial perception module performs cross-modal guidance and interaction based on these temporal cues, uncovering spatial associations between the audio and visual modalities. Next, carefully designed task-specific textual prompts guide the model to focus on features that are most relevant to the current task. Finally, the learned representations are serialized and passed to task-specific decoders to address different downstream audiovisual scene understanding tasks.

Fig. 3: Encoder and decoder for AVS task.

	Segment-level					Event-level
Method	Audio	Visual	Audio-Visual	Type	Event	Audio	Visual	Audio-Visual	Type	Event
AVEL [ 4 ]	47.2	37.1	35.4	39.9	41.6	40.4	34.7	31.6	35.5	36.5
AVSDN [ 59 ]	47.8	52.0	37.1	45.7	50.8	34.1	46.3	26.5	35.6	37.7
HAN [ 6 ]	60.1	52.9	48.9	54.0	55.4	51.3	48.9	43.0	47.7	48.0
MA [ 60 ]	60.3	60.0	55.1	58.9	57.9	53.6	56.4	49.0	53.0	50.6
CVCMS [ 61 ]	60.8	63.5	57.0	60.5	59.5	53.8	58.9	49.5	54.0	52.1

TABLE I: The performance of the AV-Unified Framework on the LLP Dataset for the AVVP Task.

Method	Fully-Supervised	Weakly-Supervised
AVEL [ 4 ]	72.7	66.7
AVIN [ 65 ]	75.2	69.4
AVT [ 66 ]	75.8	70.2
CMRAN [ 67 ]	77.4	73.0
PSP [ 68 ]	77.8	73.5
MM-Pyramid [ 63 ]	77.8	73.2

TABLE II: The performance of the AV-Unified on AVE.

Method	CIoU (%)	AUC (%)
Attention10K [ 8 ]	18.50	30.20
CoarsetoFine [ 69 ]	29.10	34.80
AVObject [ 70 ]	29.70	35.70
LVS [ 71 ]	34.40	38.20
HardPos [ 72 ]	34.60	38.00
EZ-VSL [ 73 ]	38.85	39.54

TABLE III: The performance of the AV-Unified on VGG-SS.

Method	Audio			Visual			Audio-Visual						Avg
Method	Count	Comp	Avg	Count	Local	Avg	Exist	Count	Local	Comp	Temp	Avg	Avg
FCNLSTM [ 75 ]	70.80	65.66	68.90	64.58	48.08	56.23	82.29	59.92	46.20	62.94	47.45	60.42	60.81
GRU [ 76 ]	71.29	63.13	68.28	66.08	68.08	67.09	80.67	61.03	51.74	62.85	57.79	63.03	65.03
Hco_Att [ 77 ]	70.80	54.71	64.87	63.49	67.10	65.32	79.48	59.84	48.80	56.31	56.33	60.32	62.45
MCAN [ 78 ]	78.07	57.74	70.58	71.76	71.76	71.76	80.77	65.22	54.57	56.77	46.84	61.52	65.83
PSAC [ 79 ]	75.02	66.84	72.00	68.00	70.78	69.41	79.76	61.66	55.22	61.13	59.85	63.60	66.62

TABLE IV: The performance of the AVQA task trained jointly with other multiple audio-visual scene understanding tasks.

Method	S4		MS3		AVSS
Method	mIoU	F-Score	mIoU	F-Score	mIoU	F-Score
MSSL [ 69 ]	44.9	66.3	26.1	36.3	-	-
SST [ 86 ]	66.3	80.1	42.6	57.2	-	-
iGAN [ 87 ]	61.6	77.8	42.9	54.4	-
LGVT [ 88 ]	74.9	87.3	40.7	59.3	-	-
TPAVI [ 10 ]	78.7	87.9	54.0	64.5	29.8	35.2

TABLE V: The performance of the S4, MS3 and AVSS task trained individually and jointly with other multiple audio-visual scene understanding tasks in AV-Unified framework.

Method	AVE [ 4 ]		LLP [ 6 ]				VGG-SS [ 71 ]		S4 [ 10 ]		MS3 [ 10 ]		AVSS [ 74 ]		MUSIC-AVQA [ 14 ]
	Fully	Weakly	Segment-level		Event-level		CIoU	AUC	mIoU	F-Score	mIoU	F-Score	mIoU	F-Score	AV	All
	Fully	Weakly	Type	Event	Type	Event	CIoU	AUC	mIoU	F-Score	mIoU	F-Score	mIoU	F-Score	AV	All
AV-Unified w/o. jt	77.4	73.5	62.2	60.9	56.4	53.6	38.94	40.80	82.40	89.00	59.30	68.40	37.10	37.40	71.94	75.32
AV-Unified (Ours)	78.7	74.2	62.8	61.5	56.7	53.9	39.16	41.24	83.20	87.10	59.50	69.30	43.10	41.90	76.96	76.42

TABLE VI: The performance of multi-task trained individually (AV-Unified w/o. jt , jt : joint training) and jointly (AV-Unified) with other multiple audio-visual scene understanding tasks in AV-Unified framework.

Method	Audio	Visual	Audio-Visual	All
w/o MS-STPM	73.93	79.23	70.37	73.35
w/o TPM	77.72	79.81	71.21	74.64
w/o SPM	75.85	80.64	71.84	74.88
w/o TPGL	75.54	79.93	71.74	74.59
AV-Unified (Ours)	72.60	75.61	76.96	76.42

TABLE VII: MS-TSPM’s module configuration results.

Fig. 4: Visualization of the Temporal-Spatial Task (AVQA). The audio-visual representation of the input video is first processed by the AV-Unified framework to obtain a unified representation, corresponding to the w/o task-specific prompt setting. Then, a task-specific prompt is applied to guide the model in selecting the most relevant features required for the task, corresponding to the w/ task-specific prompt setting.

Explore similar work

Crab $^{+}$ : A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Mar 4, 2026Dongnuan Cai, Henghui Du, Chang Zhou +5Audio-Visual Understanding Unified Multimodal Models
Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Artificial Intelligence of China Telecom (TeleAI) · AI Technology Center, Online Video Business Unit, Tencent PCG +2
Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization
Jun 5, 2026Zhe Yang, Ruyi Zhang, Hongtao Chen +4Audio-Visual Understanding Heterogeneous GNNs
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Peng Cheng Laboratory, Shenzhen 518000, China · Harbin Institute of Technology Suzhou Research Institute, Suzhou 215000, China
Audio-Visual Intelligence in Large Foundation Models
May 5, 2026You Qin, Kai Liu, Shengqiong Wu +12Audio-Visual Understanding Audio-Video Generation
National University of Singapore · University of Oxford · University of Toronto +5

Mar 4, 2026J/K move · Enter open · S save

Crab $^{+}$ : A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Dongnuan Cai, Henghui Du, Chang Zhou, Xi Chen+4

Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Artificial Intelligence of China Telecom (TeleAI) · AI Technology Center, Online Video Business Unit, Tencent PCG+2

Audio-Visual Understanding Unified Multimodal Models Multi-Task Learning Multimodal Instruction Tuning Low-Rank Adaptation

Open paper AskPDF arXiv

Developing Audio-Visual Large Language Models (AV-LLMs) for unified scene understanding is pivotal in multimodal intelligence. While instruction tuning enables pre-trained models with multi-task abilities, we observe that conventional multi-task unification methods often suffer from severe negative transfer, where nearly 55% of tasks degrade compared to single-task training. We attribute this phenomenon to audio-visual task heterogeneity, characterized by disparate task granularity and divergent capability demands, which lead to negative interference under joint training. To tackle this, we present Crab

^{+}

, a scalable and unified audio-visual scene understanding model that addresses task heterogeneity through explicit cooperation from both data and model perspectives. On the data side, we introduce AV-UIE v2, a comprehensive Audio-Visual Unified Instruction-tuning dataset with Explicit reasoning processes. It contains approximately 222K samples spanning 17 datasets and 7 tasks, enabling the model to capture cross-task relationships at different levels of granularity. On the model side, we design a unified interface to align heterogeneous task formulations, and propose Interaction-aware LoRA (I-LoRA), which explicitly models inter-task relationships via dynamic routing to coordinate distinct audio-visual interaction patterns, mitigating parameter interference. Extensive experiments show Crab

^{+}

covers broader tasks than existing unified models while outperforming specialized models on various benchmarks. We successfully reverse the negative transfer trend, achieving positive transfer where multi-task learning surpasses single-task baselines in nearly 88% of tasks. These results hold across diverse AV-LLM paradigms and are validated through in-depth visualization, positioning Crab

^{+}

as a robust step towards holistic audio-visual scene understanding.

Abstract

Explore similar work

Crab+^{+}+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization

Audio-Visual Intelligence in Large Foundation Models

Crab $^{+}$ : A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation