Crab$^{+}$: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Dongnuan Cai; Henghui Du; Chang Zhou; Xi Chen; Dan Guo; Hongyuan Zhang; Xuelong Li; Di Hu

cs.CVMar 4, 2026

Crab $^{+}$ : A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Authors: Dongnuan Cai, Henghui Du, Chang Zhou, Xi Chen, Dan Guo, Hongyuan Zhang, Xuelong Li, Di Hu

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Artificial Intelligence of China Telecom (TeleAI) · AI Technology Center, Online Video Business Unit, Tencent PCG · Hefei University of Technology · The University of Hong Kong

Abstract

Developing Audio-Visual Large Language Models (AV-LLMs) for unified scene understanding is pivotal in multimodal intelligence. While instruction tuning enables pre-trained models with multi-task abilities, we observe that conventional multi-task unification methods often suffer from severe negative transfer, where nearly 55% of tasks degrade compared to single-task training. We attribute this phenomenon to audio-visual task heterogeneity, characterized by disparate task granularity and divergent capability demands, which lead to negative interference under joint training. To tackle this, we present Crab $^{+}$ , a scalable and unified audio-visual scene understanding model that addresses task heterogeneity through explicit cooperation from both data and model perspectives. On the data side, we introduce AV-UIE v2, a comprehensive Audio-Visual Unified Instruction-tuning dataset with Explicit reasoning processes. It contains approximately 222K samples spanning 17 datasets and 7 tasks, enabling the model to capture cross-task relationships at different levels of granularity. On the model side, we design a unified interface to align heterogeneous task formulations, and propose Interaction-aware LoRA (I-LoRA), which explicitly models inter-task relationships via dynamic routing to coordinate distinct audio-visual interaction patterns, mitigating parameter interference. Extensive experiments show Crab $^{+}$ covers broader tasks than existing unified models while outperforming specialized models on various benchmarks. We successfully reverse the negative transfer trend, achieving positive transfer where multi-task learning surpasses single-task baselines in nearly 88% of tasks. These results hold across diverse AV-LLM paradigms and are validated through in-depth visualization, positioning Crab $^{+}$ as a robust step towards holistic audio-visual scene understanding.

Figures & tables

Figure 1: Multi-task vs. single-task LoRA on customized LLaMA2 and Qwen2.5-Omni. Green and red cells highlight tasks with performance gains and drops, respectively, under multi-task learning.

Figure 2: Overview of Crab + . (a) Scalable and unified architecture capable of addressing diverse audio-visual scene understanding tasks. (b) Explicit cooperation that captures complex cross-task dependencies to improve overall performance.

Figure 3: Overview of AV-UIE v2 dataset. (a) Construction pipeline. We leverage MLLMs to augment raw data and simple labels with explicit reasoning processes. (b) Statistical distribution. The dataset integrates a comprehensive collection of source data from 17 datasets, covering 7 distinct scene understanding tasks.

Audio-Visual Task	Source Dataset	Domain	Samples (K)
Action Recognition	Kinetics-Sounds [ 26 ]	Human Actions	20.1
Action Recognition	UCF51 [ 27 ]	Human Actions	4.5
Emotion Recognition	MELD [ 28 ]	Conversations	11.0
	MAFW [ 29 ]	Movies & TV	6.4
	DFEW [ 30 ]	Movies & TV	9.3
	MER2024 [ 31 ]	Movies & TV	4.8

Table 1: Data composition of the AV-UIE v2 dataset, organized by task complexity from basic recognition to complex reasoning.

Figure 4: Our unified input-output interface comprises a visual branch, an audio branch, a LLM backbone, and a segmentation module, trained on the AV-UIE v2 dataset with interaction-aware LoRA.

Figure 5: Interaction-aware LoRA comprises a shared matrix

A

and multiple

B

matrices, where inputs are dynamically routed to specific

B

matrices.

Method	CREMA-D	MELD	DFEW	MAFW	KS	UCF51
OGM-GE [ 59 ]	64.34	–	–	–	66.35	–
MMPareto [ 64 ]	70.19	–	–	–	69.13	–
CAL [ 65 ]	79.30	–	–	–	74.82	–
DialogueRNN [ 66 ]	–	56.12	–	–	–	–
DQ-Former [ 67 ]	–	64.88	–	–	–	–
ACoSDN [ 68 ]	–	68.04	–	–	–	–

Table 3: Comparison of methods on recognition tasks.

Method	AVVP		AVE
Method	Segment-level	Event-level	AVE
AVT [ 74 ]	–	–	75.80
MM-Pyramid [ 75 ]	59.20	53.04	77.80
CMBS [ 76 ]	55.00	48.48	–
DHHN [ 77 ]	60.32	55.06	–
MoLT [ 78 ]	–	–	83.50

Table 4: Comparison of methods on temporal localization tasks.

Method	ARIG	S4	MS3	Seen	Unseen
LVS [ 79 ]	23.69	–	–	–	–
FNAC [ 80 ]	27.15	–	–	–	–
AVSBench [ 14 ]	–	78.70	54.00	23.20	32.36
AVSegFormer [ 81 ]	–	83.06	61.33	33.47	36.05
EEMC [ 15 ]	–	–	–	34.20	49.54
SAM2-LOVE [ 82 ]	–	–	–	43.50	66.50

Table 5: Comparison of methods on spatial localization tasks.

Method	MUSIC-AVQA				AVQA
Method	Audio	Visual	Audio-Visual	Overall	AVQA
ST-AVQA [ 17 ]	73.87	74.40	69.53	71.59	–
PSTP-Net [ 83 ]	70.91	77.26	72.57	73.52	90.20
SaSR-Net [ 84 ]	73.56	73.28	74.66	74.21	89.90
QA-TIGER [ 85 ]	78.58	85.14	73.74	77.62	–
HCRN+HAVF [ 86 ]	–	–	–	–	89.00

Table 6: Comparison of methods on AVQA tasks.

Figure 6: Comparative analysis of synergy in multi-task vs. single-task learning across three representative AV-LLM paradigms. For each model, we report the proportion of tasks exhibiting positive gains and the net score, defined as the difference between the percentage of tasks benefiting from multi-task learning and those suffering negative transfer. The results demonstrate that our method effectively mitigates task heterogeneity, promoting inter-task synergy.

Figure 7: Visualization of router weights. Head pair comparisons across (a)–(b) datasets and (c)–(d) tasks. Tasks sharing similar demands form distinct clusters, indicating capacity to clarify complex audio-visual patterns.

Figure 8: Router activation analysis. (a) Average activations show balanced contributions ( $\approx 0.3$ ). (b) Relative patterns reveal task-specific head preferences.

Figure 9: Similarity analysis of routing matrices

B

. (a) Overall similarity (mean 0.630). (b) Layer-wise similarity, showing divergence in deeper layers.

Method	CREMA-D	AVQA	MUSIC	AVVP	KS	ARIG	A $\rightarrow$ V	V $\rightarrow$ A
drop head-1	74.06	86.57	71.11	51.91	91.01	40.22	68.14	66.15
drop head-2	72.37	85.30	71.55	54.65	91.57	43.94	67.01	67.83
drop head-3	71.76	84.34	69.80	39.72	90.76	44.40	67.88	67.14
no drop	74.12	86.92	74.73	59.77	92.21	45.78	68.81	69.28

Table 7: Results of dropping heads.

Figure 10: Ablation results about LoRA. While LoRA shows gains on a subset of tasks, I-LoRA significantly increases the number of tasks with synergy.

Figure 11: Ablation on the number of

B

heads. Our method remains stable without significant fluctuations across this hyperparameter.

Figure 12: Qualitative results. Crab + is capable of performing multiple different audio-visual scene understanding tasks on a single sample.

Explore similar work

Mar 6, 2026

AV-Unified: A Unified Framework for Audio-visual Scene Understanding

When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing, segmentation and question answering are mostly explored individually, making it challenging to comprehensively understand complex audio-visual scenes and explore inter-task relationships. Hence, we propose AV-Unified, a unified framework that enables joint learning across a wide range of audio-visual scene understanding tasks. AV-Unified standardizes the diverse input-output formats of each task and incorporates a multi-scale spatiotemporal perception network to effectively capture audio-visual associations. Specifically, we unify the inputs and outputs of all supported tasks by converting them into sequences of discrete tokens, establishing a shared representation that allows a single architecture to be trained jointly across heterogeneous varied datasets. Considering the varying temporal granularity of audio-visual events, a multi-scale temporal perception module is designed to capture key cues. Meanwhile, to overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module that models spatial audio-visual associations. Furthermore, task-specific text prompts are employed to enhance the model's adaptability and task-awareness. Extensive experiments on benchmark datasets (e.g., AVE, LLP, MUSIC-AVQA, VGG-SS and AVS) demonstrate the effectiveness of AV-Unified across temporal, spatial, and spatiotemporal tasks.

Guangyao Li, Xin Wang, Wenwu Zhu

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · Beijing National Research Center for Information Science and Technology, Beijing 100084, China

Audio-Visual Understanding Unified Multimodal Models Multimodal Fusion

Ask PDF

May 5, 2026

Audio-Visual Intelligence in Large Foundation Models

Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era of large foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-vision architectures that learn from massive multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g., speech recognition, sound localization) to generation (e.g., audio-driven video synthesis, video-to-audio) and interaction (e.g., dialogue, embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.

You Qin, Kai Liu, Shengqiong Wu +12

National University of Singapore · University of Oxford · University of Toronto +5

Audio-Visual Understanding Audio-Video Generation Multimodal Pretraining Audio-Visual Alignment Multimodal Foundation Models

Ask PDF

Apr 17, 2026

AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers

Recent advances in reasoning models have shown remarkable progress in text-based domains, but transferring those capabilities to multimodal settings, e.g., to allow reasoning over audio-visual data, still remains a challenge, in part because of the limited availability of high-quality reasoning data in targeted multimodal combinations. To address this problem, we introduce AVRT, a novel framework that generates high-quality audio-visual reasoning traces from single-modality teacher models. We generate independent vision- and audio-reasoning traces via models specialized to reason over their respective modalities and merge the resulting traces with an LLM merger model. The resulting multimodal traces are used in a supervised fine-tuning (SFT) cold start to adapt the target model to audio-visual reasoning traces first, before training it in a second reinforcement learning stage on larger-scale data. Evaluated on seven audio-visual and audio benchmarks, our 3B and 7B parameter models achieve state-of-the-art results among models of comparable size including OmniBench and DailyOmni for audio-visual and MMAR for audio-only reasoning, showing that cross-modal training also transfers to single-modality tasks and establishing a new training pipeline for multimodal reasoning models.

Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza +5

University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA +2

Cross-Modal Learning Audio-Visual Understanding Multimodal Reasoning Multimodal Instruction Tuning Multimodal Understanding

Ask PDF