SIREN: Spatially-Informed Reconstruction of Binaural Audio with Vision

Mingyeong Song; Seoyeon Ko; Junhyug Noh

cs.SDMar 31, 2026

SIREN: Spatially-Informed Reconstruction of Binaural Audio with Vision

Authors: Mingyeong Song, Seoyeon Ko, Junhyug Noh

Organizations: Ewha Womans University, Seoul, Korea

Abstract

Binaural audio delivers spatial cues essential for immersion, yet most consumer videos are monaural due to capture constraints. We introduce SIREN, a visually guided mono to binaural framework that explicitly predicts left and right channels. A ViT-based encoder learns dual-head self-attention to produce a shared scene map and end-to-end L/R attention, replacing hand-crafted masks. A soft, annealed spatial prior gently biases early L/R grounding, and a two-stage, confidence-weighted waveform-domain fusion (guided by mono reconstruction and interaural phase consistency) suppresses crosstalk when aggregating multi-crop and overlapping windows. Evaluated on FAIR-Play and MUSIC-Stereo, SIREN yields consistent gains on time-frequency and phase-sensitive metrics with competitive SNR. The design is modular and generic, requires no task-specific annotations, and integrates with standard audio-visual pipelines.

Figures & tables

Figure 1: Overview. (a, b) SIREN takes mono STFT $S_{M}$ and video $V$ ; a ViT with dual-head attention produces a shared map $v$ and L/R features that FiLM-condition an audio U-Net to predict $\hat{S}_{D}$ and $(\hat{S}_{L},\hat{S}_{R})$ . (c) At test time, two-stage confidence-weighted refinement fuses crop-wise and overlap-wise waveforms (post-iSTFT) using mono and interaural-phase scores.

Method	FAIR-Play (10-split)				MUSIC-Stereo
Method	STFT $\downarrow$	ENV $\downarrow$	Phs $\downarrow$	SNR $\uparrow$	STFT $\downarrow$	ENV $\downarrow$	Phs $\downarrow$	SNR $\uparrow$
Mono2Binaural [ 2 ]	0.889	0.137	1.438	6.232	0.942	0.138	1.550	8.255
Sep-Stereo [ 5 ]	0.879	0.136	1.429	–	0.929	0.135	1.544	8.306
CMC [ 8 ]	0.849	0.133	1.423	–	0.759	0.113	1.502	–
CC-Stereo [ 9 ]	0.823	0.132	–	7.144	0.624	0.097	1.578	12.985
SIREN (Ours)	0.820	0.132	1.550	7.219	0.417	0.091	1.006	10.872

Table 1: Comparison with prior work on FAIR-Play (10-split) and MUSIC-Stereo. Lower is better for STFT/ENV/Phs; higher is better for SNR. “–” denotes not reported.

$L_{\text{prior}}$	$W$	STFT $\downarrow$	ENV $\downarrow$	Phs $\downarrow$	SNR $\uparrow$
✗	✗	0.941	0.141	1.599	6.345
✓	✗	0.928	0.140	1.584	6.224
✓	✓	0.888	0.136	1.589	6.798

Table 2: Ablation on FAIR-Play (5-split): impact of the spatial prior loss (

L_{\text{prior}}

) and confidence-weighted refinement (

W

FiLM	DualHead	STFT $\downarrow$	ENV $\downarrow$	Phs $\downarrow$	SNR $\uparrow$
✗	✗	0.935	0.141	1.582	6.379
✓	✗	0.925	0.140	1.576	6.432
✗	✓	0.913	0.139	1.577	6.475
✓	✓	0.888	0.136	1.589	6.798

Table 3: Ablation on FAIR-Play (5-split): effect of FiLM conditioning and dual-head attention.

Spatial Audio

Spatial audio studies how sound is captured, represented, generated, and perceived with direction, distance, motion, and room acoustics.

See all 100 papers

Binaural Audio-Visual Instance Segmentation
Sep 26, 2026Saijun Wang, Guanfeng Tang, Hongbo Zhao +4Audio-Visual Understanding Instance Segmentation
College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China
PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation
Sep 30, 2026Tiernon Riesenmy, You Zhang, Gautam Bhattacharya +1Text-to-Audio Generation Audio-Video Generation
The Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Advanced Technology Group, Dolby Laboratories, San Francisco, CA, USA
BinauralVAE: Spatial Audio Reconstruction For World Models
Sep 6, 2026Luis Vitor Zerkowski, Luiz VelhoSpatial Audio Audio Representation Learning
VISGRAF IMPA

Sep 26, 2026J/K move · Enter open · S save

Binaural Audio-Visual Instance Segmentation

Saijun Wang, Guanfeng Tang, Hongbo Zhao, Zhicheng Lei+3

College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China

Audio-Visual Understanding Instance Segmentation

Open paper AskPDF arXiv

Audio-visual segmentation (AVS) aims to segment sounding objects at the pixel level by integrating auditory and visual cues. However, existing methods are predominantly developed under the monaural setting and primarily rely on cross-modal semantic correspondence, which limits their ability to distinguish visually similar instances of the same semantic class. In contrast, humans naturally exploit binaural hearing, where interaural differences and direction-dependent acoustic filtering introduced by the head and pinnae provide physically grounded spatial cues for accurate sound source localization. Motivated by this observation, we introduce binaural audio-visual instance segmentation (BiAVIS), a new task that leverages synchronized binaural audio and video frames to segment sounding instances. To advance research on this task, we establish two benchmarks by manually annotating an existing binaural audio-visual dataset and collecting a new real-world dataset, BiAVIS-Bench, in more challenging and diverse scenarios. The proposed BiAVIS model (BiAVISM) features an audio-only sound source localization network that directly projects binaural spatial cues onto the image plane to learn spatial and semantic priors for sounding instances. A query-level audio-visual fusion strategy is subsequently introduced to inject these informative priors into the instance segmentation decoder. Extensive experiments conducted on the two proposed benchmarks demonstrate the superior performance of the BiAVISM over previous monaural AVS methods, especially in resolving instance-level intra-class ambiguity. On the more challenging BiAVIS-Bench, BiAVISM achieves 26.65 mAP and 45.90 FSLA, corresponding to relative improvements of approximately 35.35% over AVISM in mean average precision (mAP) and 11.62% over ACVIS in frame-level sound localization accuracy (FSLA).

Abstract

Spatial Audio

Explore similar work

Binaural Audio-Visual Instance Segmentation

PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation

BinauralVAE: Spatial Audio Reconstruction For World Models