SonoWorld4D: From One Video to a 4D Audio-Visual Scene via Diffusion Synchronization

Derong Jin; Xiyi Chen; Ming C. Lin; Ruohan Gao

cs.CVOct 29, 2026

SonoWorld4D: From One Video to a 4D Audio-Visual Scene via Diffusion Synchronization

Authors: Derong Jin, Xiyi Chen, Ming C. Lin, Ruohan Gao

Organizations: University of Maryland, College Park

Abstract

Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and present the SonoWorld4D framework for this task. Given a monocular video, our method generates a 360 static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time novel-view visual and spatial audio rendering. Evaluation on real-world dynamic scenes and a user study demonstrates SonoWorld4D generates 4D audio-visual scenes with perceptually realistic, physically grounded spatial audio consistent with scene geometry and visual content.

Figures & tables

Figure 1: From a single monocular video, SonoWorld4D generates an explorable 4D audio-visual scene, where you can navigate to novel viewpoints and follow arbitrary listener trajectories while hearing spatial audio synchronized with dynamic events and grounded in moving sound sources (e.g., a racing car), static sources (e.g., birds and rustling trees), and background ambience.

Figure 2: Overview of SonoWorld4D . Given a monocular video, we extract depth maps, camera poses, foreground masks, and VLM-derived audio prompts. For visual scene generation (Sec. 4.2 ), the framework reconstructs per-object dynamic Gaussians, lifts an outpainted $360^{\circ}$ panorama into background Gaussians, and aligns them into a common world frame via scale correction and local displacement correction. For dynamic sound field generation (Sec. 4.3 ), static sources are synthesized via text-to-audio and anchored to the background. Dynamic sources are generated via dual-space diffusion synchronization: at each sampling step, forward rendering ( $\mathcal{R}_{i}$ ) maps object-space predictions to camera space, while source separation and inverse rendering ( $\mathcal{Q}_{i}$ ) map camera-space mixtures back to individual object spaces. The resulting scene enables real-time novel-pose visual and spatial audio rendering (Sec. 4.4 ).

	Spatial Metrics					Semantic Metrics
Method	$\Delta_{\text{abs}}\theta$ $\downarrow$	$\Delta_{\text{abs}}\varphi$ $\downarrow$	$\Delta_{\text{Angular}}$ $\downarrow$	CC $\uparrow$	AUC $\uparrow$	CLAP ${}_{\text{A}}$ $\uparrow$	CLAP ${}_{\text{T}}$ $\uparrow$
ViSAGe	74.1	24.5	75.9	0.189	0.615	0.078	$-$ 0.059
OmniAudio	80.9	30.0	83.1	0.124	0.579	0.234	0.057
SonoWorld	65.7	24.3	66.6	0.333	0.672	0.365	0.154
Sonic4D	54.3	17.2	55.5	0.446	0.728	0.223	0.103
Ours (T2A)	45.7	16.8	48.4	0.566	0.787	0.349	0.185

Table 1: Quantitative comparison on SonoScene4D for video-to-4D audio-visual scene generation. We compare our full method against baselines and two ablated variants: Ours (T2A) and Ours (V2A), which use text-to-audio and video-to-audio generation for dynamic-source signals, respectively. Angular errors are in degrees.

\downarrow

\uparrow

indicate whether lower/higher is better.

Figure 3: Time-azimuth RMS visualization of ambisonics. Top: mid-frame panoramas of each video rendered by our method at microphone poses in SonoScene4D with faces blurred to protect privacy. Remaining rows compare the reference FOA recordings with baselines and ours. Each map shows the peak-normalized directional RMS response over time and azimuth after spherical-area-weighted aggregation over elevation. Our method more closely follows the reference azimuthal trajectories and temporal structure, while the baselines exhibit directional mismatches or reduced spatiotemporal variation.

Figure 4: STFT visualization of spatial audio rendered at a novel-listener position. The top two rows show input video frames and our novel-view renderings on a shared timeline. Highlighted intervals (a) and (b) indicate when the airplane is closest to the input camera and the novel listener, respectively. Correct spatialization should concentrate energy near (b) with a downward frequency shift due to the Doppler effect. Sonic4D retains strong energy near (a). Ours (V2A) responds near both (a) and (b). Ours (T2A) responds near (b) but lacks clear spectral structure. Our full method produces a dominant response with a visible Doppler shift around (b), correctly adapting both amplitude and frequency to the novel-listener position.

Figure 5: Pairwise user preferences . Bars show mean per-participant preference rates, and error bars indicate 95% confidence intervals.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Motion-type-specific instantiations for dynamic foreground generation. Each branch combines object motion estimation with canonical Gaussian generation and a local deformation model: identity for rigid objects, linear blend skinning ( Magnenat-Thalmann et al., 1989 ) for humans, and learned per-Gaussian center offsets ( Litman et al., 2026 ) for non-rigid objects.

Panel of Figure 7 — Figure 7: Qualitative ablation on visual components. Top: Ablation of displacement correction in foreground-background alignment. Both variants include global scale alignment; adding local displacement corrections reduces foreground interpenetration with the ground. Bottom: Both panorama-to-3D backbones produce usable scenes, but Marble yields fewer visible reconstruction artifacts in the illustrated example.

4D Scene Generation

4D scene generation creates dynamic 3D worlds with coherent geometry, appearance, and motion across time.

See all 54 papers

SonoWorld: From One Image to a 3D Audio-Visual Scene
Mar 30, 2026Derong Jin, Xiyi Chen, Ming C. Lin +1Audio-Video Generation 3D Scene Generation
University of Maryland, College Park
CP4D: Compositional Physics-aware 4D Scene Generation
Jun 8, 2026Hanxin Zhu, Cong Wang, Tianyu He +4Physical Consistency in Video Generation 3D Scene Generation
School of Information Science and Technology, University of Science and Technology of China · Zhongguancun Academy · the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences. +2
Full-4D: Generating Full-Scope 4D Scenes from a Single-View Video
May 25, 2026Tingxi Chen, Ke Hao, Yabo Chen +6Camera-Conditioned Video Generation 4D Reconstruction
Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom (TeleAI)

Mar 30, 2026J/K move · Enter open · S save

SonoWorld: From One Image to a 3D Audio-Visual Scene

Derong Jin, Xiyi Chen, Ming C. Lin, Ruohan Gao

University of Maryland, College Park

Audio-Video Generation 3D Scene Generation Image-to-3D Scene Generation Spatial Audio

Open paper AskPDF arXiv

Tremendous progress in visual scene generation now turns a single image into an explorable 3D world, yet immersion remains incomplete without sound. We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image, and present SonoWorld, the first framework to tackle this challenge. From one image, our pipeline outpaints a 360° panorama, lifts it into a navigable 3D scene, places language-guided sound anchors, and renders ambisonics for point, areal, and ambient sources, yielding spatial audio aligned with scene geometry and semantics. Quantitative evaluations on a newly curated real-world dataset and a controlled user study confirm the effectiveness of our approach. Beyond free-viewpoint audio-visual rendering, we also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation. Project website: https://humathe.github.io/sonoworld/

Abstract

4D Scene Generation

Explore similar work

SonoWorld: From One Image to a 3D Audio-Visual Scene

CP4D: Compositional Physics-aware 4D Scene Generation

Full-4D: Generating Full-Scope 4D Scenes from a Single-View Video