SonoWorld: From One Image to a 3D Audio-Visual Scene

Derong Jin; Xiyi Chen; Ming C. Lin; Ruohan Gao

cs.CVMar 30, 2026

SonoWorld: From One Image to a 3D Audio-Visual Scene

Authors: Derong Jin, Xiyi Chen, Ming C. Lin, Ruohan Gao

Organizations: University of Maryland, College Park

Abstract

Tremendous progress in visual scene generation now turns a single image into an explorable 3D world, yet immersion remains incomplete without sound. We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image, and present SonoWorld, the first framework to tackle this challenge. From one image, our pipeline outpaints a 360° panorama, lifts it into a navigable 3D scene, places language-guided sound anchors, and renders ambisonics for point, areal, and ambient sources, yielding spatial audio aligned with scene geometry and semantics. Quantitative evaluations on a newly curated real-world dataset and a controlled user study confirm the effectiveness of our approach. Beyond free-viewpoint audio-visual rendering, we also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation. Project website: https://humathe.github.io/sonoworld/

Figures & tables

Figure 1: From one image, SonoWorld generates an explorable 3D audio-visual scene, where you can navigate to novel views and locations, while listening to spatial audio aligned with scene semantics and the 3D locations of heterogeneous sound sources.

Figure 2: Illustration of real-world audio-visual scene data collection and curation for SonoScene360 .

Figure 3: Given a single image $I$ , SonoWorld jointly generates a 3D visual scene $\mathbf{V}$ and a semantically and geometrically aligned audio scene ${\bf A}$ . It consists of: 1) Visual Scene Generation (Sec. 4.1 ): single-image calibration and warping followed by panorama outpainting to obtain the a full $360^{\circ}$ panorama image $I_{\mathrm{pano}}$ , which is further lifted into a 3D Gaussian scene via a panorama-to-3D reconstruction model $\mathcal{G}_{\mathrm{\textbf{V}}}$ ; 2) 360 ∘ Semantic Grounding (Sec. 4.2 ): a VLM extracts the categories $\mathcal{C}$ of potential sounding sources, which are used to generate panoramic instance masks M with the help and coordination of both an open-vocabulary segmentation model (OVS) and a class-agnostic segmentation model (SAM2 [ 63 ] ); 3) Ambisonics Encoding (Sec. 4.3 ): based on the audio prompt and equalization parameters from the VLM model, a text-to-audio (T2A) model generates per-source waveforms that are equalized, and mapped to ambisonics coefficients based on the 3D locations and its source type; and 4) Free-Viewpoint Rendering (Sec. 4.4 ): the ambisonics coefficients are decoded into pose-dependent binaural audio $\textbf{b}(\textbf{p})$ using head related transfer function (HRTF) and synchronized with Gaussian rendering ${\bf V}(\textbf{p})$ .

Figure 4: Per-scene results on our SonoScene360 dataset. Results on representative metrics show that our method consistently outperforms baselines on all scenes.

Figure 5: User Study of Spatial Audio Quality. Preference rates across synthetic and real scenes for our method, MMAudio [ 9 ] , and OmniAudio [ 54 ] . Each bar shows the average per-user, per-scene preference (%); error bars indicate the interquartile range (25 th –75 th percentile) across 50 participants.

	Spatial Metrics					Semantic Metrics
Method	$\Delta_{\text{abs}}\theta$ $\downarrow$	$\Delta_{\text{abs}}\varphi$ $\downarrow$	$\Delta_{\text{Anglular}}$ $\downarrow$	CC $\uparrow$	AUC $\uparrow$	D-CLAP ${}_{\text{R}}$ $\uparrow$	D-CLAP ${}_{\text{A}}$ $\uparrow$	D-CLAP ${}_{\text{T}}$ $\uparrow$
MMAudio [ 9 ]	—	—	—	—	—	33.8%	0.345	0.322
SEE-2-SOUND [ 14 ]	1.354	0.254	1.397	0.194	0.603	22.1%	0.221	0.156
ViSAGe [ 42 ]	1.598	0.426	1.649	0.142	0.624	22.1%	0.088	0.019
OmniAudio [ 54 ]	1.508	0.317	1.449	0.148	0.588	39.7%	0.101	0.104
Ours (Open-source)	0.976	0.251	0.975	0.491	0.753	52.9%	0.464	0.413

Table 1: Quantitative comparison on SonoScene360 . We report two versions of our method: Ours (Open-source) , a fully reproducible version using only open-source models, and Ours (Proprietary) , which leverages proprietary models for the best performance.

Method	$\Delta_{\text{Angular}}$ $\downarrow$	MAG $\downarrow$	ENV $\downarrow$
NAF [ 56 ]	1.76	3.96	3.60
INRAS [ 68 ]	1.64	5.06	5.97
AV-NeRF [ 49 ]	1.58	4.58	1.89
Ours	0.22	3.46	1.22

Table 2: Results on one-shot room acoustic learning.

Figure 6: Ambisonics energy map visualization on real-world scenes. Top row: panoramic renderings from our method at novel views (microphone poses) on SonoScene360 . Second row: ground-truth FOA energy maps. Remaining rows: FOA predictions from our method and OmniAudio [ 54 ] . Our maps most closely follow the ground-truth reference spatial patterns, while the baselines produce oversmoothed or misaligned energy. For more qualitative baseline comparisons, please refer to Supp.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Spatial Metrics					Semantic Metrics
Method	$\Delta_{\text{abs}}\theta$ $\downarrow$	$\Delta_{\text{abs}}\varphi$ $\downarrow$	$\Delta_{\text{Anglular}}$ $\downarrow$	CC $\uparrow$	AUC $\uparrow$	D-CLAP ${}_{\text{R}}$ $\uparrow$	D-CLAP ${}_{\text{A}}$ $\uparrow$	D-CLAP ${}_{\text{T}}$ $\uparrow$
Ours (Point)	1.264	0.614	1.203	0.273	0.629	38.2%	0.521	0.419
Ours (w/o EQ)	0.815	0.196	0.814	0.602	0.806	58.8%	0.467	0.443
Ours (No merge)	0.800	0.359	0.843	0.586	0.807	39.7%	0.271	0.324
Ours (All merge)	1.032	0.331	1.012	0.445	0.719	45.6%	0.461	0.329
Ours (Boundary perturb)	0.760	0.228	0.791	0.627	0.824	69.1%	0.483	0.456

Appendix

Table 3: Ablation studies on SonoScene360 . We analyze the effect of source representation, equalization, grounding/merging strategy, and robustness to imperfect visual inputs. Ours (Point) replaces region-based sources with point sources. Ours (w/o EQ) removes the equalization module. Ours (No merge) treats tile-level masks independently, while Ours (All merge) merges all masks from the same category. Ours (Boundary perturb) and Ours (Depth perturb) evaluate robustness to noisy mask boundaries and depth estimates, respectively. Results show that each component contributes to the final performance, with our full model achieving the strongest overall spatial and semantic quality.

Figure 7: The sound energy by second order ambisonics (SOA) in real scenes.

Figure 8: Qualitative results on synthetic scenes. From a single diffusion-generated image, SonoWorld reconstructs a 3D Gaussian scene and predicts a spatial audio field that supports free-viewpoint exploration. We show two synthetic scenes ( Garden and Riverside Market ), sample views along a camera trajectory (top row), and the corresponding FOA / SOA spherical energy maps for our method and OmniAudio. The energy smoothly tracks visually grounded sources such as the waterfall, streams, and market stalls.

Figure 9: Qualitative comparison on real scenes. Additional qualitative results on six real environments from SonoScene360 . For each scene, we show the rendered panorama from our 3D Gaussian reconstruction, the reference spherical energy map derived from the recorded FOA (second row), and the predicted energy maps from our method (FOA), OmniAudio, SEE-2-SOUND (S2S), and ViSAGe. Our method produces spatial patterns that best align with the reference both in azimuth and elevation and in the spatial extent of the energy, especially around extended water and ambient sources. Please refer to the supplementary video for audio.

Figure 10: Comparison of 3D scene backbones. Given an input 360 ∘ panorama (left column), we compare novel views generated by HunyuanWorld 1.0 and Marble for several real scenes ( Kitchen , Stream , Siren ). HunyuanWorld produces globally coherent but sometimes distorted geometry (e.g., warped floors and façades), whereas Marble yields sharper, more faithful reconstructions that preserve local spatial layout, which is crucial for accurate spatial audio anchoring and free-viewpoint navigation.

Figure 11: Panoramic instance merging strategies. We compare three strategies for grouping class-agnostic region proposals into semantic instances in SonoScene360 : All-Merge (left), No-Merge (middle), and our voting-based strategy Vote (Ours) (right). For each scene, we overlay the resulting instances on the panorama and indicate their counts. All-Merge collapses large structures into a single instance, losing spatial detail; No-Merge over-fragments the scene into many small pieces. Our voting scheme strikes a balance, producing a small number of coherent instances that align with visually meaningful sound sources.

Figure 12: Calibration annotation interface. Our tool for camera–microphone calibration. Left: annotating the azimuth and elevation of the FOA microphone in the panorama. Middle: annotating azimuth and elevation for ground markers ( i.e . , the marker right below the microphone position) that define a world reference frame. Right: visualizing the relative rotation between AprilTag marker and camera, which is used to estimate the microphone orientation in the world coordinate system.

Figure 13: Effect of calibration on rendered scenes. For two real scenes ( Siren and River & Train ), we show the original camera view with the microphone visible, the inpainted camera view, and renderings of the reconstructed 3D scene when aligned using the AprilTag-based pose estimate vs. our refined pose. The refined calibration produces renderings whose layout and perspective are more consistent with the original capture, leading to more accurate alignment between the FOA reference frame and the visual scene.

Scene (setup)	# Mics	Clips / Mic	Total Clips
Fountain	10	2	20
Kitchen	5	2	10
Pool-left	3	2	6
Pool-right	2	2	4
Bridge-river	1	2	2
Bridge-train	1	2	2

Appendix

Table 4: Statistics of the SonoScene360 dataset. Each scene subset corresponds to one environment; we vary microphone layouts within a scene to capture diverse listening positions.

Figure 15: User Study Interface. An example pairwise comparison shown to participants. Each question presents two spatial-audio videos (“Method 1” and “Method 2”) for the same scene. Participants listen with headphones and select the method that provides better spatial coherence and audio–visual alignment.

Figure 16: Physics-aware occlusion add-on: as a clustered source becomes partially and then fully blocked between $t_{0}$ and $t_{1}$ , visibility-based point reweighting yields smooth attenuation of the rendered signal.

Figure 17: Dynamic source rendering on a plane fly-by clip: a time-varying 3D anchor from SAM3 [ 5 ] and Depth Anything 3 [ 50 ] separates dynamic and static sound components, with the dynamic layer peaking near closest approach and attenuating with distance

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 154 papers

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
Aug 1, 2026Masaki Yoshida, Ren Togo, Takahiro Ogawa +1Spatial Audio Audio Generation
Hokkaido University
Audible World Models: Spatially Aware Sound Generation for 3D Worlds
Sep 29, 2026Duowen Chen, Jinjin He, Gouthaman KV +2Sound Effects Generation 3D Scene Generation
Georgia Institute of Technology · Dolby Laboratories
Enabling Immersive Audio-Visual Experience from Any Video
Sep 28, 2026Zitong Lan, Mutian Tong, Jiatao Gu +1Audio-Video Generation Audio-Visual Alignment
University of Pennsylvania

Aug 1, 2026J/K move · Enter open · S save

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds

Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama

Hokkaido University

Spatial Audio Audio Generation

Open paper AskPDF arXiv

3D Gaussian Splatting (3DGS) turns captured or generated imagery into photorealistic 3D world simulations that users can freely explore, yet these worlds remain silent. Because existing audio generation methods condition on a single image or viewpoint, their sound is tied to that observation and cannot stay consistent while a listener moves. We introduce the task of generating a spatially consistent soundscape for a given 3DGS world through auditory grounding, identifying which objects in the world should emit sound and anchoring each to a persistent 3D position, and present Scene2Sound, a training-free framework built on this grounding. From the input world alone, our pipeline selects viewpoints that jointly cover the scene, identifies sound-emitting objects with a vision-language model, and associates the multi-view detections into 3D instances through Gaussian set matching, which measures the overlap between the Gaussian sets that render each detection. Each source then receives generated audio that a standard object-based audio engine spatializes in real time at arbitrary listener poses. We further propose two spatial-consistency metrics, one testing whether rendered audio responds consistently to listener motion and one testing whether the claimed sources are supported by views held out from their placement. On a curated set of generated 3DGS worlds and on 3DGS scenes generated from real-world 360-degree captures, Scene2Sound preserves the audio quality of strong per-viewpoint baselines while remaining spatially consistent where per-viewpoint and single-panorama pipelines do not, and a user study confirms the perceptual benefit. Project page: https://masaki-lmd.github.io/scene2sound/.

Abstract

Audio-Video Generation

Explore similar work

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds

Audible World Models: Spatially Aware Sound Generation for 3D Worlds

Enabling Immersive Audio-Visual Experience from Any Video