Tremendous progress in visual scene generation now turns a single image into an explorable 3D world, yet immersion remains incomplete without sound. We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image, and present SonoWorld, the first framework to tackle this challenge. From one image, our pipeline outpaints a 360° panorama, lifts it into a navigable 3D scene, places language-guided sound anchors, and renders ambisonics for point, areal, and ambient sources, yielding spatial audio aligned with scene geometry and semantics. Quantitative evaluations on a newly curated real-world dataset and a controlled user study confirm the effectiveness of our approach. Beyond free-viewpoint audio-visual rendering, we also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation. Project website: https://humathe.github.io/sonoworld/
Figures & tables
Figure 1: From one image, SonoWorld generates an explorable 3D audio-visual scene, where you can navigate to novel views and locations, while listening to spatial audio aligned with scene semantics and the 3D locations of heterogeneous sound sources.
Figure 2: Illustration of real-world audio-visual scene data collection and curation for SonoScene360 .
Figure 3: Given a single image I , SonoWorld jointly generates a 3D visual scene V and a semantically and geometrically aligned audio scene A . It consists of: 1) Visual Scene Generation (Sec. 4.1 ): single-image calibration and warping followed by panorama outpainting to obtain the a full 360∘ panorama image Ipano , which is further lifted into a 3D Gaussian scene via a panorama-to-3D reconstruction model GV ; 2) 360 ∘ Semantic Grounding (Sec. 4.2 ): a VLM extracts the categories C of potential sounding sources, which are used to generate panoramic instance masks M with the help and coordination of both an open-vocabulary segmentation model (OVS) and a class-agnostic segmentation model (SAM2 [ 63 ] ); 3) Ambisonics Encoding (Sec. 4.3 ): based on the audio prompt and equalization parameters from the VLM model, a text-to-audio (T2A) model generates per-source waveforms that are equalized, and mapped to ambisonics coefficients based on the 3D locations and its source type; and 4) Free-Viewpoint Rendering (Sec. 4.4 ): the ambisonics coefficients are decoded into pose-dependent binaural audio b(p) using head related transfer function (HRTF) and synchronized with Gaussian rendering V(p) .
Figure 4: Per-scene results on our SonoScene360 dataset. Results on representative metrics show that our method consistently outperforms baselines on all scenes.
Figure 5: User Study of Spatial Audio Quality. Preference rates across synthetic and real scenes for our method, MMAudio [ 9 ] , and OmniAudio [ 54 ] . Each bar shows the average per-user, per-scene preference (%); error bars indicate the interquartile range (25 th –75 th percentile) across 50 participants.
Spatial Metrics
Semantic Metrics
Method
Δabsθ↓
Δabsφ↓
ΔAnglular↓
CC ↑
AUC ↑
D-CLAP R↑
D-CLAP A↑
D-CLAP T↑
MMAudio [ 9 ]
—
—
—
—
—
33.8%
0.345
0.322
SEE-2-SOUND [ 14 ]
1.354
0.254
1.397
0.194
0.603
22.1%
0.221
0.156
ViSAGe [ 42 ]
1.598
0.426
1.649
0.142
0.624
22.1%
0.088
0.019
OmniAudio [ 54 ]
1.508
0.317
1.449
0.148
0.588
39.7%
0.101
0.104
Ours (Open-source)
0.976
0.251
0.975
0.491
0.753
52.9%
0.464
0.413
Table 1: Quantitative comparison on SonoScene360 . We report two versions of our method: Ours (Open-source) , a fully reproducible version using only open-source models, and Ours (Proprietary) , which leverages proprietary models for the best performance.
Method
ΔAngular↓
MAG ↓
ENV ↓
NAF [ 56 ]
1.76
3.96
3.60
INRAS [ 68 ]
1.64
5.06
5.97
AV-NeRF [ 49 ]
1.58
4.58
1.89
Ours
0.22
3.46
1.22
Table 2: Results on one-shot room acoustic learning.
Figure 6: Ambisonics energy map visualization on real-world scenes. Top row: panoramic renderings from our method at novel views (microphone poses) on SonoScene360 . Second row: ground-truth FOA energy maps. Remaining rows: FOA predictions from our method and OmniAudio [ 54 ] . Our maps most closely follow the ground-truth reference spatial patterns, while the baselines produce oversmoothed or misaligned energy. For more qualitative baseline comparisons, please refer to Supp.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Spatial Metrics
Semantic Metrics
Method
Δabsθ↓
Δabsφ↓
ΔAnglular↓
CC ↑
AUC ↑
D-CLAP R↑
D-CLAP A↑
D-CLAP T↑
Ours (Point)
1.264
0.614
1.203
0.273
0.629
38.2%
0.521
0.419
Ours (w/o EQ)
0.815
0.196
0.814
0.602
0.806
58.8%
0.467
0.443
Ours (No merge)
0.800
0.359
0.843
0.586
0.807
39.7%
0.271
0.324
Ours (All merge)
1.032
0.331
1.012
0.445
0.719
45.6%
0.461
0.329
Ours (Boundary perturb)
0.760
0.228
0.791
0.627
0.824
69.1%
0.483
0.456
Appendix
Table 3: Ablation studies on SonoScene360 . We analyze the effect of source representation, equalization, grounding/merging strategy, and robustness to imperfect visual inputs. Ours (Point) replaces region-based sources with point sources. Ours (w/o EQ) removes the equalization module. Ours (No merge) treats tile-level masks independently, while Ours (All merge) merges all masks from the same category. Ours (Boundary perturb) and Ours (Depth perturb) evaluate robustness to noisy mask boundaries and depth estimates, respectively. Results show that each component contributes to the final performance, with our full model achieving the strongest overall spatial and semantic quality.
Figure 7: The sound energy by second order ambisonics (SOA) in real scenes.
Figure 8: Qualitative results on synthetic scenes. From a single diffusion-generated image, SonoWorld reconstructs a 3D Gaussian scene and predicts a spatial audio field that supports free-viewpoint exploration. We show two synthetic scenes ( Garden and Riverside Market ), sample views along a camera trajectory (top row), and the corresponding FOA / SOA spherical energy maps for our method and OmniAudio. The energy smoothly tracks visually grounded sources such as the waterfall, streams, and market stalls.
Figure 9: Qualitative comparison on real scenes. Additional qualitative results on six real environments from SonoScene360 . For each scene, we show the rendered panorama from our 3D Gaussian reconstruction, the reference spherical energy map derived from the recorded FOA (second row), and the predicted energy maps from our method (FOA), OmniAudio, SEE-2-SOUND (S2S), and ViSAGe. Our method produces spatial patterns that best align with the reference both in azimuth and elevation and in the spatial extent of the energy, especially around extended water and ambient sources. Please refer to the supplementary video for audio.
Figure 10: Comparison of 3D scene backbones. Given an input 360 ∘ panorama (left column), we compare novel views generated by HunyuanWorld 1.0 and Marble for several real scenes ( Kitchen , Stream , Siren ). HunyuanWorld produces globally coherent but sometimes distorted geometry (e.g., warped floors and façades), whereas Marble yields sharper, more faithful reconstructions that preserve local spatial layout, which is crucial for accurate spatial audio anchoring and free-viewpoint navigation.
Figure 11: Panoramic instance merging strategies. We compare three strategies for grouping class-agnostic region proposals into semantic instances in SonoScene360 : All-Merge (left), No-Merge (middle), and our voting-based strategy Vote (Ours) (right). For each scene, we overlay the resulting instances on the panorama and indicate their counts. All-Merge collapses large structures into a single instance, losing spatial detail; No-Merge over-fragments the scene into many small pieces. Our voting scheme strikes a balance, producing a small number of coherent instances that align with visually meaningful sound sources.
Figure 12: Calibration annotation interface. Our tool for camera–microphone calibration. Left: annotating the azimuth and elevation of the FOA microphone in the panorama. Middle: annotating azimuth and elevation for ground markers ( i.e . , the marker right below the microphone position) that define a world reference frame. Right: visualizing the relative rotation between AprilTag marker and camera, which is used to estimate the microphone orientation in the world coordinate system.
Figure 13: Effect of calibration on rendered scenes. For two real scenes ( Siren and River & Train ), we show the original camera view with the microphone visible, the inpainted camera view, and renderings of the reconstructed 3D scene when aligned using the AprilTag-based pose estimate vs. our refined pose. The refined calibration produces renderings whose layout and perspective are more consistent with the original capture, leading to more accurate alignment between the FOA reference frame and the visual scene.
Scene (setup)
# Mics
Clips / Mic
Total Clips
Fountain
10
2
20
Kitchen
5
2
10
Pool-left
3
2
6
Pool-right
2
2
4
Bridge-river
1
2
2
Bridge-train
1
2
2
Appendix
Table 4: Statistics of the SonoScene360 dataset. Each scene subset corresponds to one environment; we vary microphone layouts within a scene to capture diverse listening positions.
Figure 15: User Study Interface. An example pairwise comparison shown to participants. Each question presents two spatial-audio videos (“Method 1” and “Method 2”) for the same scene. Participants listen with headphones and select the method that provides better spatial coherence and audio–visual alignment.
Figure 16: Physics-aware occlusion add-on: as a clustered source becomes partially and then fully blocked between t0 and t1 , visibility-based point reweighting yields smooth attenuation of the rendered signal.
Figure 17: Dynamic source rendering on a plane fly-by clip: a time-varying 3D anchor from SAM3 [ 5 ] and Depth Anything 3 [ 50 ] separates dynamic and static sound components, with the dynamic layer peaking near closest approach and attenuating with distance
3D Gaussian Splatting (3DGS) turns captured or generated imagery into photorealistic 3D world simulations that users can freely explore, yet these worlds remain silent. Because existing audio generation methods condition on a single image or viewpoint, their sound is tied to that observation and cannot stay consistent while a listener moves. We introduce the task of generating a spatially consistent soundscape for a given 3DGS world through auditory grounding, identifying which objects in the world should emit sound and anchoring each to a persistent 3D position, and present Scene2Sound, a training-free framework built on this grounding. From the input world alone, our pipeline selects viewpoints that jointly cover the scene, identifies sound-emitting objects with a vision-language model, and associates the multi-view detections into 3D instances through Gaussian set matching, which measures the overlap between the Gaussian sets that render each detection. Each source then receives generated audio that a standard object-based audio engine spatializes in real time at arbitrary listener poses. We further propose two spatial-consistency metrics, one testing whether rendered audio responds consistently to listener motion and one testing whether the claimed sources are supported by views held out from their placement. On a curated set of generated 3DGS worlds and on 3DGS scenes generated from real-world 360-degree captures, Scene2Sound preserves the audio quality of strong per-viewpoint baselines while remaining spatially consistent where per-viewpoint and single-panorama pipelines do not, and a user study confirms the perceptual benefit. Project page: https://masaki-lmd.github.io/scene2sound/.
Text- and image-conditioned world generators can create visually rich 3D environments, yet these worlds often remain silent or rely on soundtracks synthesized solely from text or rendered video. Although such audio can convey what should be heard, it lacks an explicit representation of where sound sources are located and how their perceived sound should vary with listener movement. We introduce Audible World Models, a training-free framework that incorporates sound into the generated world state. Starting from a text prompt, our system constructs a panoramic 3D proxy, separates it into semantic layers, identifies sound-producing foreground objects and ambient background regions, and synthesizes dry audio for each sound label. It then anchors these sources to reconstructed geometry and renders listener-dependent spatial audio using geometric acoustic propagation. By explicitly linking semantics, geometry, and sound propagation, the framework maintains persistent source locations while adapting the rendered audio to changes in listener viewpoint and motion. Experiments across 80 generated scenes demonstrate substantial gains in spatial consistency over text-, video-, and panorama-conditioned baselines, while preserving competitive semantic alignment. VLM-based assessments and human evaluations further indicate that our soundtracks are preferred for their audio-visual consistency, spatial plausibility, and motion-dependent behavior.
Duowen Chen, Jinjin He, Gouthaman KV +2
Georgia Institute of Technology · Dolby Laboratories
Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded visual worlds remain incomplete. This paper presents OmniDream, a training-free framework that transforms a silent monocular video into an immersive audiovisual experience, where viewers can freely look around while sounds remain spatially aligned with the visual scene. At the core of OmniDream is an object-centric audio representation that disentangles each sound source's intrinsic audio content from its scene-dependent acoustic effects, enabling independent audio generation, physics-based simulation of propagation effects, and flexible spatial audio rendering. Experiments show improved audio-visual alignment, spatial correctness, and perceptual immersiveness over baselines. Examples are available on https://huggingface.co/spaces/CuriousAlien000/spatial-audio-360-demo