AudioGS: Spectrogram-Based Audio Gaussian Splatting for Sound Field Reconstruction

Chunhao Bi; Houqiang Zhong; Zhixin Xu; Li Song; Zhengxue Cheng

cs.SDApr 10, 2026

AudioGS: Spectrogram-Based Audio Gaussian Splatting for Sound Field Reconstruction

Authors: Chunhao Bi, Houqiang Zhong, Zhixin Xu, Li Song, Zhengxue Cheng

Organizations: School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China · Institute of Cultural and Creative Industry, Shanghai Jiao Tong University, Shanghai, China

Abstract

Spatial audio is fundamental to immersive virtual experiences, yet synthesizing high-fidelity binaural audio from sparse observations remains a significant challenge. Existing methods typically rely on implicit neural representations conditioned on visual priors, which often struggle to capture fine-grained acoustic structures. Inspired by 3D Gaussian Splatting (3DGS), we introduce AudioGS, a novel visual-free framework that explicitly encodes the sound field as a set of Audio Gaussians based on spectrograms. AudioGS associates each time-frequency bin with an Audio Gaussian equipped with dual Spherical Harmonic (SH) coefficients and a decay coefficient. For a target pose, we render binaural audio by evaluating the SH field to capture directionality, incorporating geometry-guided distance attenuation and phase correction, and reconstructing the waveform. Experiments on the Replay-NVAS dataset demonstrate that AudioGS successfully captures complex spatial cues and outperforms state-of-the-art visual-dependent baselines. Specifically, AudioGS reduces the magnitude reconstruction error (MAG) by over 14% and reduces the perceptual quality metric (DPAM) by approximately 25% compared to the best performing visual-guided method.

Figures & tables

Fig. 1: AudioGS framework and performance. Top: Inspired by 3D Visual Gaussian Splatting, we propose AudioGS, a visual-free framework that models the sound field as a set of learnable 3D Audio Gaussians. Bottom: Quantitative comparison on the Replay-NVAS dataset. AudioGS significantly outperforms baselines in reconstruction quality (MAG) and perceptual fidelity (DPAM).

Fig. 2: Overview of the AudioGS framework. We explicitly represent the sound field by encoding the source spectrogram into a set of Audio Gaussians, where each Gaussian corresponds to a specific time-frequency bin. For a target listener pose, the rendering pipeline separates into two streams: (1) Magnitude Modulation , which utilizes spherical harmonics and distance attenuation to model directional energy and spatial decay; and (2) Phase Correction , which computes inter-aural time differences (ITD) based on wave propagation delays. Finally, the modulated magnitudes and corrected phases are combined via inverse STFT (iSTFT) to synthesize high-fidelity binaural audio.

Method	Visual	SC-1044				SC-1052				SC-1074
Method	Visual	MAG $\downarrow$	ENV $\downarrow$	LRE $\downarrow$	DPAM $\downarrow$	MAG $\downarrow$	ENV $\downarrow$	LRE $\downarrow$	DPAM $\downarrow$	MAG $\downarrow$	ENV $\downarrow$	LRE $\downarrow$	DPAM $\downarrow$
Source Binaural	✗	0.2468	0.1030	2.5450	0.1279	0.0966	0.0318	3.0069	0.1477	0.0547	0.0187	1.9118	0.1247
Mono	✗	0.2029	0.0836	0.5400	0.1280	0.0836	0.0269	0.8100	0.1477	0.0460	0.0158	0.5287	0.1247
ViGAS [ 4 ]	✓	0.1945	0.0608	0.5517	0.1105	0.0842	0.0225	0.4802	0.1062	0.0511	0.0147	0.4700	0.1095
AV-NeRF [ 12 ]	✓	0.2354	0.0942	0.6315	0.2278	0.0883	0.0289	0.8424	0.2946	0.0456	0.0142	0.5770	0.4999
AV-Cloud [ 7 ]	✓	0.2004	0.0707	0.4380	0.2271	0.0665	0.0204	0.6658	0.1621	0.0439	0.0143	0.4357	0.2546

TABLE I: Results on Replay-NVAS dataset. Background colors indicate performance ranking: First , Second , and Third best results.

Fig. 3: Comparison of binaural waveform synthesis. Left: Images of source and target viewpoints provided for spatial context. Note that unlike visual-dependent baselines, AudioGS does not utilize these RGB images. Right: Synthesized waveforms compared against the Ground Truth (GT). The red boxes highlight regions with distinct transient acoustic events. Baseline methods (ViGAS, AV-NeRF) often produce attenuated or over-smoothed responses in these high-frequency regions. In contrast, AudioGS accurately reconstructs the signal envelope, matching the GT closely. The embedded table reports the scene-specific metrics, where AudioGS achieves the best spatial accuracy (LRE) and perceptual quality (DPAM).

Methods	MAG $\downarrow$	ENV $\downarrow$	LRE $\downarrow$	DPAM $\downarrow$
AudioGS	0.0892	0.0300	0.5269	0.0887
w/o DA	0.1018	0.0341	0.8618	0.0930
w/o SH	0.0911	0.0309	0.6437	0.0896
w/o PC	0.0906	0.0302	0.7932	0.0893

TABLE II: Ablation study of key components in AudioGS.

Fig. 4: MUSHRA listening test on Replay-NVAS dataset. Mean Basic Audio Quality (BAQ) ratings (0–100) from 12 participants on six representative novel-view samples. Error bars denote the standard error of the mean (SEM).

Fig. 5: The spatial distribution of Audio Gaussians filtered by STFT magnitudes. Left/Middle: Gaussians with the lowest 20% and 50% magnitudes remain clustered near the initialization center, representing silence or noise. Right: Overall distribution shows that high-energy Gaussians spread out spatially, effectively encoding the directional sound field structure.

Spatial Audio

Spatial audio studies how sound is captured, represented, generated, and perceived with direction, distance, motion, and room acoustics.

See all 100 papers

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
Aug 1, 2026Masaki Yoshida, Ren Togo, Takahiro Ogawa +1Spatial Audio Audio Generation
Hokkaido University
BinauralVAE: Spatial Audio Reconstruction For World Models
Sep 6, 2026Luis Vitor Zerkowski, Luiz VelhoSpatial Audio Audio Representation Learning
VISGRAF IMPA
DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos
Apr 3, 2026Ziyu Luo, Lin Chen, Qiang Qu +2Spatial Audio Audio-Video Generation
School of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing, China · School of Computer Science, The University of Sydney, Sydney, NSW, Australia · School of Software, Shandong University, Jinan, China

Aug 1, 2026J/K move · Enter open · S save

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds

Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama

Hokkaido University

Spatial Audio Audio Generation

Open paper AskPDF arXiv

3D Gaussian Splatting (3DGS) turns captured or generated imagery into photorealistic 3D world simulations that users can freely explore, yet these worlds remain silent. Because existing audio generation methods condition on a single image or viewpoint, their sound is tied to that observation and cannot stay consistent while a listener moves. We introduce the task of generating a spatially consistent soundscape for a given 3DGS world through auditory grounding, identifying which objects in the world should emit sound and anchoring each to a persistent 3D position, and present Scene2Sound, a training-free framework built on this grounding. From the input world alone, our pipeline selects viewpoints that jointly cover the scene, identifies sound-emitting objects with a vision-language model, and associates the multi-view detections into 3D instances through Gaussian set matching, which measures the overlap between the Gaussian sets that render each detection. Each source then receives generated audio that a standard object-based audio engine spatializes in real time at arbitrary listener poses. We further propose two spatial-consistency metrics, one testing whether rendered audio responds consistently to listener motion and one testing whether the claimed sources are supported by views held out from their placement. On a curated set of generated 3DGS worlds and on 3DGS scenes generated from real-world 360-degree captures, Scene2Sound preserves the audio quality of strong per-viewpoint baselines while remaining spatially consistent where per-viewpoint and single-panorama pipelines do not, and a user study confirms the perceptual benefit. Project page: https://masaki-lmd.github.io/scene2sound/.

Abstract

Spatial Audio

Explore similar work

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds

BinauralVAE: Spatial Audio Reconstruction For World Models

DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos