Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement

Zhengxian Yang; Shengqi Wang; Shi Pan; Hongshuai Li; Haoxiang Wang; Lin Li; Guanjun Li; Zhengqi Wen

cs.CVApr 10, 2026

Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement

Authors: Zhengxian Yang, Shengqi Wang, Shi Pan, Hongshuai Li, Haoxiang Wang, Lin Li, Guanjun Li, Zhengqi Wen, +3 more

Organizations: Tsinghua University, Beijing, China. · the Institute of Automation, Chinese Academy of Sciences, Beijing, China. · the Migu Beijing Research Institute, Beijing, China · the BNRist, Tsinghua University, Beijing, China. · the School of Architecture, Tsinghua University, Beijing, China. · the Department of Automation, Tsinghua University, Beijing, China.

Abstract

Fully immersive experiences that tightly integrate 6-DoF visual and auditory interaction are essential for virtual and augmented reality. While such experiences can be achieved through computer-generated content, constructing them directly from real-world captured videos remains largely unexplored. We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces, audiovisual feedback, and high-resolution, high-frame-rate dynamic content. To support IVV construction, we present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy. Our custom capture rig enables synchronized multi-view video-audio acquisition during motion, facilitating efficient capture of complex indoor and outdoor scenes with rich foreground--background interactions and challenging dynamics. The dataset provides 5K-resolution videos at 60 FPS with durations of 1-5 minutes, offering richer spatial, temporal, and multimodal coverage than existing benchmarks. Leveraging this dataset, we develop a dynamic light field reconstruction framework built upon a Gaussian-based spatio-temporal representation, incorporating flow-guided sparse initialization, joint camera temporal calibration, and multi-term spatio-temporal supervision for robust and accurate modeling of complex motion. We further propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data. Together, these components form a unified pipeline for immersive volumetric video production. Extensive benchmarks and immersive VR experiments demonstrate that our pipeline generates high-quality, temporally stable audiovisual volumetric content with large 6-DoF interaction spaces. This work provides both a foundational definition and a practical construction methodology for immersive volumetric videos.

Figures & tables

Fig. 1 : Equipped with a) Multimodal Capture, b) a State-of-the-art Audiovisual Reconstruction Framework, and c) Seamless 6-DoF VR Interaction, we achieve high-fidelity Immersive Volumetric Videos (IVV) construction.

Fig. 2 : We introduce ImViD , a dataset for immersive volumetric videos. ImViD records dynamic scenes using a multi-view audio-video capture rig moving in a space-oriented manner, which provides a new benchmark for immersive volumetric video reconstruction and its application.

Datasets	No. Scene	Outdoor/ Indoor	Cameras	Mobility	Resolution	Angles	Duration	FPS	Multi- modality	Content
PanopticSports [ 7 ]	65	Indoor	480 cameras	Static	640 $\times$ 480	$360^{\circ}$	5mins	25	✗	Human-centric actions
Technicolor [ 37 ]	12	Indoor	16 cameras	Static	2048 $\times$ 1088	Frontal	2s	30	✗	Has a number of close-ups sequences, captured medium angle scenes and other animated scenes where the movement does not come from a human
Immersive-Lightfield [ 3 ]	15	both	46 cameras	Static	2560 $\times$ 1920	Frontal	10–30s	30	✗	Simple and slow motion of human, animals, objects
HyperNeRF [ 33 ]	17	Indoor	1 hand-held phone	Fixed-point Waving	1920 $\times$ 1080	Frontal	30–60s	30	✗	Waving a mobile phone in front of a moving scene, object-centric
Dynamic Scene Datasets (NVIDIA) [ 34 ]	8	Outdoor	1 Mobile phone /12 cameras	Fixed-point Waving/Static	1920 $\times$ 1080	Frontal	5s	60	✗	Simple body motions (facial, jump, etc.)
UCSD Dynamic [ 38 ]	96	Outdoor	10 cameras	Static	1920 $\times$ 1080	Frontal	1–2mins	120	✗	Various visual effects and human interactions

TABLE I : Existing real-world datasets for dynamic novel view synthesis.

Fig. 3 : The pipeline to realize the multimodal 6-DoF immersive VR experiences. We applied a carefully designed rig to (a) simultaneously capture multi-view video and audio. The (b1) presents our proposed dynamic light field reconstruction framework while (b2) demonstrates the construction process of sound field, details can be seen in Sec. IV . Ultimately, we achieve a 6-DoF audiovisual experience (c) , demonstrate the effectiveness of both our dataset and the proposed pipeline.

Fig. 4 : Our rig support two kinds of capturing strategies for high resolution, high frame rate and $360^{\circ}$ dynamic data.

Name	#Camera	#Static Views	#Take	#Strategy	Density ( $m^{3}/s$ )	Viewing Space	Avg. Duration	Storage (GB)
Scene 1 Opera	39	1152	2	1	–	$180^{\circ}$	3min22s	226
Scene 2 Laboratory	39	1225	2	2	0.10	$360^{\circ}$	1min42s	137.3
Scene 3 Classroom	39	1223	2	2	0.10	$360^{\circ}$	4min42s	497
Scene 4 Meeting	39	1223	1	1	–	$360^{\circ}$	3min16s	114
Scene 5 Rendition	39	1620	4	2	0.10	$360^{\circ}$	2min02s	516
Scene 6 Puppy	39	1404	3	2	0.10	$360^{\circ}$	1min50s	359

TABLE II : Detailed Statistics of the ImViD Dataset.

Fig. 5 : Calculation method for spatiotemporal capture density . 1) the capture strategy of handheld monocular camera 2) represents the fixed camera array 3) Our rig moving covers a volume of $0.6\pi\times 0.5^{2}\ \mathrm{m}^{3}$ over 5 seconds.

Fig. 6 : Overview of the proposed Dynamic Light Field Reconstruction method. The pipeline starts with Flow-Guided Sparse Initialization (a) , which leverages SfM geometry and optical flow priors to decouple static and dynamic regions, initializing static primitives globally and dynamic ones per frame to reduce redundancy. The scene is then represented using a Gaussian-Based Spatio-Temporal Representation (b) , encoding spatial geometry, appearance, and temporal dynamics, where motion is modeled by linear velocity and visibility by Gaussian-modulated temporal opacity. In the final stage, we jointly optimize the scene parameters and perform Joint Camera Temporal Calibration (c) to refine per-camera temporal offsets for sub-frame alignment. Rendered color, depth, and optical flow maps are further supervised through Spatio-Temporal Supervision (d) , enforcing photometric, geometric, and motion consistency.

Fig. 7 : Overview of the proposed sound field reconstruction framework. The pipeline consists of two main modules. Sound Localization estimates the 3D coordinates of the sound source. Spatial Sound Generation then synthesizes binaural audio based on the VR user’s real-time pose.

(a) ImViD	Gaussian4D			4DGS			STG			Ex4DGS			Ours
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Scene 1 Opera	25.61	0.873	0.206	25.01	0.877	0.174	26.30	0.899	0.169	23.08	0.835	0.219	33.51	0.916	0.070
Scene 2 Laboratory	22.15	0.908	0.099	21.02	0.910	0.081	23.40	0.920	0.067	20.29	0.896	0.106	31.10	0.941	0.034
Scene 5 Rendition	25.78	0.848	0.206	26.23	0.865	0.158	26.21	0.861	0.145	24.73	0.857	0.159	27.84	0.884	0.073
Scene 6 Puppy	20.15	0.408	0.553	21.03	0.568	0.299	21.25	0.597	0.231	21.21	0.586	0.226	21.53	0.607	0.135
Average	23.42	0.759	0.266	23.32	0.805	0.178	24.29	0.819	0.153	22.33	0.794	0.178	28.50	0.837	0.078

TABLE III : Quantitative comparison on Gaussian-based dynamic light field reconstruction method across multiple datasets. All metrics are calculated on the test views across all frames (300 frames for each scene). The best , second and third results are highlighted with color blocks.

Method		Scene 1 Opera	Scene 2 Laboratory	Scene 5 Rendition	Scene 6 Puppy	Average
Gaussian4D	PSNR $\uparrow$	25.61	22.15	25.78	20.15	23.42
Gaussian4D	Size (MB) $\downarrow$	107	174	92	183	139.0
4DGS	PSNR $\uparrow$	25.01	21.02	26.23	21.03	23.32
4DGS	Size (MB) $\downarrow$	1802	4832	1403	5233	3317.5
STG	PSNR $\uparrow$	26.30	23.40	26.21	21.25	24.29
STG	Size (MB) $\downarrow$	540	542	366	1493	735.3

TABLE IV : Quantitative results and storage footprint on the ImViD dataset (300 frames for each scene).

Panel of Figure 12 — Fig. 8 : Qualitative Comparison on ImViD Dataset.

Setting	Init. Pts (Ratio)	Final Pts	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o Flow Init.	2.48M (100%)	4.52M	33.13	0.915	0.070
w Flow Init. (Ours)	0.29M ( 11.6% )	2.50M	33.51	0.916	0.070

TABLE V : Ablation on Flow-Guided Initialization. Our strategy maintains high reconstruction fidelity while significantly reducing initial primitive redundancy.

Panel of Figure 14 — Fig. 9 : Ablation on Spatio-temporal Supervision. Qualitative results on Cam 10 show that removing depth and flow constraints leads to severe artifacts. Rendered flow maps reveal that lacking flow supervision induces chaotic background motion, directly causing severe temporal flickering.

Panel of Figure 15 — Fig. 10 : Ablation on Joint Camera Temporal Calibration. Qualitative results on the test view (Cam 00) show that jointly optimizing camera temporal offsets effectively improves reconstruction quality in fast-moving regions, yielding a clearer texture and sharper motion boundaries.

Setting	$\lambda_{reg}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o $\Delta\gamma$ Opt.	-	30.59	0.910	0.035
w/ $\Delta\gamma$ Opt.	0	29.93	0.903	0.033
w/ $\Delta\gamma$ Opt.	$10^{-2}$	30.50	0.911	0.031
w/ $\Delta\gamma$ Opt. (Ours)	$10^{-4}$	30.94	0.917	0.028

TABLE VI : Ablation on Joint Camera Temporal Calibration. Jointly optimizing camera temporal offsets (

\Delta\gamma

) improves reconstruction quality. Metrics are reported exclusively for the dynamic foreground involving the opera performers.

Loss Component		Peripheral View Metrics
$\mathcal{L}_{flow}$	$\mathcal{L}_{depth}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
-	-	24.43	0.729	0.287
✓	-	30.03	0.748	0.153
-	✓	30.45	0.760	0.134
✓	✓	30.98	0.772	0.124

TABLE VII : Ablation on Spatio-temporal Supervision. Metrics are evaluated on Cam 10 to reveal the contribution of geometric and motion constraints.

Rating	Spatial Perception	Sound Quality	Immersiveness
Very poor	0.00%	0.00%	0.00%
Poor	0.00%	0.00%	0.00%
Fair	14.28%	19.04%	9.52%
Good	23.80%	33.33%	42.85%
Excellent	61.90%	47.61%	47.61%

TABLE VIII : User study for the sound field construction. 21 participants rated the sound field reconstruction.

Method	Direction Score $\uparrow$	Distance Score $\uparrow$
w/o SFR	1.17	1.67
SFR + distance	1.69	3.02
SFR + direction	3.81	3.03
SFR + distance + direction	3.91	3.07

TABLE IX : Ablation studies on the Sound Field Reconstruction (SFR) baseline. 58 participants rated sense of direction and distance (Scale 1–5).

Fig. 11 : Visualization of the interaction trajectory and corresponding visual & auditory results. The multi-modal feedback changes consistently with the positions and orientations.

Explore similar work

Enabling Immersive Audio-Visual Experience from Any Video
Sep 28, 2026Zitong Lan, Mutian Tong, Jiatao Gu +1Audio-Video Generation Audio-Visual Alignment
University of Pennsylvania
PackUV: Packed Gaussian UV Maps for 4D Volumetric Video
Feb 26, 2026Aashish Rai, Angela Xing, Anushka Agarwal +54D Reconstruction 3D Gaussian Splatting
Brown University · UMass Amherst · Meta
HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video
Jun 28, 2026Yiming Jiang, Hanzhang Tu, Wenfeng Song +53D Gaussian Splatting Streaming 3D Reconstruction
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Tsinghua University, Beijing, China · College of Computer Science, Beijing Information Science and Technology University, Beijing, China +1

Sep 28, 2026J/K move · Enter open · S save

Enabling Immersive Audio-Visual Experience from Any Video

Zitong Lan, Mutian Tong, Jiatao Gu, Mingmin Zhao

University of Pennsylvania

Audio-Video Generation Audio-Visual Alignment Spatial Audio

Open paper AskPDF arXiv

Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded visual worlds remain incomplete. This paper presents OmniDream, a training-free framework that transforms a silent monocular video into an immersive audiovisual experience, where viewers can freely look around while sounds remain spatially aligned with the visual scene. At the core of OmniDream is an object-centric audio representation that disentangles each sound source's intrinsic audio content from its scene-dependent acoustic effects, enabling independent audio generation, physics-based simulation of propagation effects, and flexible spatial audio rendering. Experiments show improved audio-visual alignment, spatial correctness, and perceptual immersiveness over baselines. Examples are available on https://huggingface.co/spaces/CuriousAlien000/spatial-audio-360-demo

Abstract

Explore similar work

Enabling Immersive Audio-Visual Experience from Any Video

PackUV: Packed Gaussian UV Maps for 4D Volumetric Video

HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video