GAANet: Global-guided Asymmetric Attention Network for Audio-Visual Speech Separation

Zhiyuan Zhang; Jingyuan Xu; Yiming Tang; Liu Liu; Dan Guo

cs.SDOct 2, 2026

GAANet: Global-guided Asymmetric Attention Network for Audio-Visual Speech Separation

Authors: Zhiyuan Zhang, Jingyuan Xu, Yiming Tang, Liu Liu, Dan Guo

Organizations: School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui 230601, China

Abstract

Multi-scale design is crucial for efficient audio-visual speech separation, yet effectively modeling multi-scale information for audio-visual feature fusion remains challenging. We argue that the limited capacity of existing approaches primarily arises from: 1) treating features from different modalities in the same manner, and 2) overlooking the role of global features. To address these issues, we propose a Global-guided Asymmetric Attention Network (GAANet). Our model introduces two core innovations: first, an asymmetric multi-scale fusion framework that allows audio and visual streams to extract and interact with features at their respective optimal temporal resolutions, removing the need for symmetric temporal downsampling; second, a global-guided attention mechanism that compresses each modality into a compact global token with a temporal dimension of one, which then provides high-level semantic cues to guide both intra- and inter-modal fusion across scales. Experiments on LRS2 and VoxCeleb2 demonstrate that GAANet achieves state-of-the-art performance, reaching 16.5 dB SI-SNRi on LRS2 and 14.0 dB on VoxCeleb2, while maintaining a lightweight computational profile with only 3.3M parameters and 19.8G MACs. These results highlight the strong potential of asymmetric temporal modeling and global guidance for efficient and robust multimodal fusion. The source code is publicly accessible at https://github.com/redizzy/GAANet

Figures & tables

Fig. 1 : Illustration of 1) Existing multi-scale methods with symmetric fusion framework and 2) our proposed GAANet with asymmetric multi-scale fusion framework and global-token guidance.

Fig. 2 : Overview of GAANet, which consists of an audio encoder, a video encoder, an A-V Fusion Block combining the asymmetric multi-scale fusion framework and the global-guided attention mechanism, a lightweight auditory refinement module, and an audio decoder.Here, “ $\odot$ ” denotes the element-wise product.

Fig. 3 : Architecture of one iteration of the proposed A-V Fusion Block. In a single iteration, the audio and visual multi-scale features are first processed by the asymmetric multi-scale fusion framework to produce the cumulative representations $S_{A}$ and $S_{V}$ . The global-guided attention mechanism (a) then generates modality-specific global tokens $G_{A}$ and $G_{V}$ , which are used in (b) to modulate features across all scales via global token modulation. The refined multi-scale audio and visual features are finally consolidated and passed to the next iteration. This figure illustrates the case of $D=3$ .

Methods	LRS2			VoxCeleb2			Params (M)	MACs (G)
Methods	SI-SNRi	SDRi	PESQ	SI-SNRi	SDRi	PESQ	Params (M)	MACs (G)
AVConvTasNet [ 25 ]	12.5	12.8	2.69	9.2	9.8	2.17	16.5	23.8
LWTNet [ 2 ]	-	10.8	-	-	-	-	69.4	-
VisualVoice [ 4 ]	11.5	11.8	2.78	9.3	10.2	2.45	77.8	9.7
CaffNet-C [ 7 ]	-	10.0	1.15	-	7.6	-	-	-
AVLiT-8 [ 15 ]	12.8	13.1	2.56	9.4	9.9	2.23	5.8	18.2

TABLE I : Separation results of different AVSS methods on LRS2 and VoxCeleb2 datasets. These metrics represent the average values for all speakers in each test set, where larger SI-SNRi, SDRi and PESQ values are better. ”-” denotes results not reported in the original paper. Bolds indicate the best.

GGA	AMF	SI-SNRi(dB)	SDRi(dB)	MACs (G)	Params (M)
✗	✗	15.5	15.65	18.4	3.1
✓	✗	16.2	16.31	19.6	3.3
✗	✓	15.9	16.31	19.1	3.2
✓	✓	16.5	16.64	19.8	3.3

TABLE II : Ablation study on the global-guided attention (GGA) and asymmetric multi-scale fusion (AMF) framework on the LRS2 dataset.

Fig. 4 : Time–frequency representations of two mixtures from the LRS2 dataset. For each example (1) and (2) , we show the ground-truth spectrogram, the output of GAANet, and IIANet. The red boxes highlight regions in which IIANet retains noticeable interference or loses harmonic structure, whereas GAANet recovers more target-consistent spectral patterns.

Fig. 5 : t-SNE visualization of the global audio tokens $G_{A}$ . We randomly select 10 target speakers from the test set. Each point corresponds to one $G_{A}$ embedding, and colors indicate different target speakers.

Setting	$N_{1}$	$D$	SI-SNRi (dB)	MACs (G)	Params (M)
Varying $N_{1}$ (fixed $D=4$ )
A	1	4	15.2	18.1	3.3
B	2	4	15.7	18.7	3.3
C	3	4	16.1	19.2	3.3
D	4	4	16.5	19.8	3.3
E	5	4	16.5	20.4	3.3

TABLE III : Impact of the iteration number

N_{1}

and multi-scale depth

D

on the LRS2 dataset.

Audio Source Separation

Audio source separation isolates individual speakers, instruments, or sound events from mixtures for listening, analysis, and audio editing.

See all 66 papers

Cinematic Audio Source Separation Using Visual Cues
Mar 27, 2026Kang Zhang, Suyeon Lee, Arda Senocak +1Audio Source Separation
School of Electrical Engineering, KAIST · Graduate School of Artificial Intelligence, UNIST
Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning
Apr 9, 2026Linge Wang, Yingying Chen, Bingke Zhu +2Audio-Visual Representation Learning Multimodal Pretraining
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Objecteye Inc., Beijing, China
AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning
Oct 22, 2026Chuan Han Li, Yun Chien ChengMultimodal Emotion Recognition Multimodal Fusion
Department of Mechanical Engineering, College of Engineering, National Yang Ming Chiao Tung University, Hsin-Chu, Taiwan · Department of Electrical Engineering, College of Engineering, National Taiwan University, Taipei, Taiwan

Mar 27, 2026J/K move · Enter open · S save

Cinematic Audio Source Separation Using Visual Cues

Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son Chung

School of Electrical Engineering, KAIST · Graduate School of Artificial Intelligence, UNIST

Audio Source Separation

Open paper AskPDF arXiv

Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams (e.g., facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at https://cass-flowmatching.github.io.

Abstract

Audio Source Separation

Explore similar work

Cinematic Audio Source Separation Using Visual Cues

Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning