Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
Authors: Zhengxian Yang, Shengqi Wang, Shi Pan, Hongshuai Li, Haoxiang Wang, Lin Li, Guanjun Li, Zhengqi Wen, +3 more
Organizations: Tsinghua University, Beijing, China. · the Institute of Automation, Chinese Academy of Sciences, Beijing, China. · the Migu Beijing Research Institute, Beijing, China · the BNRist, Tsinghua University, Beijing, China. · the School of Architecture, Tsinghua University, Beijing, China. · the Department of Automation, Tsinghua University, Beijing, China.
Fully immersive experiences that tightly integrate 6-DoF visual and auditory interaction are essential for virtual and augmented reality. While such experiences can be achieved through computer-generated content, constructing them directly from real-world captured videos remains largely unexplored. We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces, audiovisual feedback, and high-resolution, high-frame-rate dynamic content. To support IVV construction, we present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy. Our custom capture rig enables synchronized multi-view video-audio acquisition during motion, facilitating efficient capture of complex indoor and outdoor scenes with rich foreground--background interactions and challenging dynamics. The dataset provides 5K-resolution videos at 60 FPS with durations of 1-5 minutes, offering richer spatial, temporal, and multimodal coverage than existing benchmarks. Leveraging this dataset, we develop a dynamic light field reconstruction framework built upon a Gaussian-based spatio-temporal representation, incorporating flow-guided sparse initialization, joint camera temporal calibration, and multi-term spatio-temporal supervision for robust and accurate modeling of complex motion. We further propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data. Together, these components form a unified pipeline for immersive volumetric video production. Extensive benchmarks and immersive VR experiments demonstrate that our pipeline generates high-quality, temporally stable audiovisual volumetric content with large 6-DoF interaction spaces. This work provides both a foundational definition and a practical construction methodology for immersive volumetric videos.
Figures & tables
Fig. 1 : Equipped with a) Multimodal Capture, b) a State-of-the-art Audiovisual Reconstruction Framework, and c) Seamless 6-DoF VR Interaction, we achieve high-fidelity Immersive Volumetric Videos (IVV) construction.
Fig. 2 : We introduce ImViD , a dataset for immersive volumetric videos. ImViD records dynamic scenes using a multi-view audio-video capture rig moving in a space-oriented manner, which provides a new benchmark for immersive volumetric video reconstruction and its application.
Datasets
No. Scene
Outdoor/ Indoor
Cameras
Mobility
Resolution
Angles
Duration
FPS
Multi- modality
Content
PanopticSports [ 7 ]
65
Indoor
480 cameras
Static
640 × 480
360∘
5mins
25
✗
Human-centric actions
Technicolor [ 37 ]
12
Indoor
16 cameras
Static
2048 × 1088
Frontal
2s
30
✗
Has a number of close-ups sequences, captured medium angle scenes and other animated scenes where the movement does not come from a human
Immersive-Lightfield [ 3 ]
15
both
46 cameras
Static
2560 × 1920
Frontal
10–30s
30
✗
Simple and slow motion of human, animals, objects
HyperNeRF [ 33 ]
17
Indoor
1 hand-held phone
Fixed-point Waving
1920 × 1080
Frontal
30–60s
30
✗
Waving a mobile phone in front of a moving scene, object-centric
Dynamic Scene Datasets (NVIDIA) [ 34 ]
8
Outdoor
1 Mobile phone /12 cameras
Fixed-point Waving/Static
1920 × 1080
Frontal
5s
60
✗
Simple body motions (facial, jump, etc.)
UCSD Dynamic [ 38 ]
96
Outdoor
10 cameras
Static
1920 × 1080
Frontal
1–2mins
120
✗
Various visual effects and human interactions
TABLE I : Existing real-world datasets for dynamic novel view synthesis.
Fig. 3 : The pipeline to realize the multimodal 6-DoF immersive VR experiences. We applied a carefully designed rig to (a) simultaneously capture multi-view video and audio. The (b1) presents our proposed dynamic light field reconstruction framework while (b2) demonstrates the construction process of sound field, details can be seen in Sec. IV . Ultimately, we achieve a 6-DoF audiovisual experience (c) , demonstrate the effectiveness of both our dataset and the proposed pipeline.
Fig. 4 : Our rig support two kinds of capturing strategies for high resolution, high frame rate and 360∘ dynamic data.
Name
#Camera
#Static Views
#Take
#Strategy
Density ( m3/s )
Viewing Space
Avg. Duration
Storage (GB)
Scene 1 Opera
39
1152
2
1
–
180∘
3min22s
226
Scene 2 Laboratory
39
1225
2
2
0.10
360∘
1min42s
137.3
Scene 3 Classroom
39
1223
2
2
0.10
360∘
4min42s
497
Scene 4 Meeting
39
1223
1
1
–
360∘
3min16s
114
Scene 5 Rendition
39
1620
4
2
0.10
360∘
2min02s
516
Scene 6 Puppy
39
1404
3
2
0.10
360∘
1min50s
359
TABLE II : Detailed Statistics of the ImViD Dataset.
Fig. 5 : Calculation method for spatiotemporal capture density . 1) the capture strategy of handheld monocular camera 2) represents the fixed camera array 3) Our rig moving covers a volume of 0.6π×0.52m3 over 5 seconds.
Fig. 6 : Overview of the proposed Dynamic Light Field Reconstruction method. The pipeline starts with Flow-Guided Sparse Initialization (a) , which leverages SfM geometry and optical flow priors to decouple static and dynamic regions, initializing static primitives globally and dynamic ones per frame to reduce redundancy. The scene is then represented using a Gaussian-Based Spatio-Temporal Representation (b) , encoding spatial geometry, appearance, and temporal dynamics, where motion is modeled by linear velocity and visibility by Gaussian-modulated temporal opacity. In the final stage, we jointly optimize the scene parameters and perform Joint Camera Temporal Calibration (c) to refine per-camera temporal offsets for sub-frame alignment. Rendered color, depth, and optical flow maps are further supervised through Spatio-Temporal Supervision (d) , enforcing photometric, geometric, and motion consistency.
Fig. 7 : Overview of the proposed sound field reconstruction framework. The pipeline consists of two main modules. Sound Localization estimates the 3D coordinates of the sound source. Spatial Sound Generation then synthesizes binaural audio based on the VR user’s real-time pose.
(a) ImViD
Gaussian4D
4DGS
STG
Ex4DGS
Ours
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Scene 1 Opera
25.61
0.873
0.206
25.01
0.877
0.174
26.30
0.899
0.169
23.08
0.835
0.219
33.51
0.916
0.070
Scene 2 Laboratory
22.15
0.908
0.099
21.02
0.910
0.081
23.40
0.920
0.067
20.29
0.896
0.106
31.10
0.941
0.034
Scene 5 Rendition
25.78
0.848
0.206
26.23
0.865
0.158
26.21
0.861
0.145
24.73
0.857
0.159
27.84
0.884
0.073
Scene 6 Puppy
20.15
0.408
0.553
21.03
0.568
0.299
21.25
0.597
0.231
21.21
0.586
0.226
21.53
0.607
0.135
Average
23.42
0.759
0.266
23.32
0.805
0.178
24.29
0.819
0.153
22.33
0.794
0.178
28.50
0.837
0.078
TABLE III : Quantitative comparison on Gaussian-based dynamic light field reconstruction method across multiple datasets. All metrics are calculated on the test views across all frames (300 frames for each scene). The best , second and third results are highlighted with color blocks.
Method
Scene 1 Opera
Scene 2 Laboratory
Scene 5 Rendition
Scene 6 Puppy
Average
Gaussian4D
PSNR ↑
25.61
22.15
25.78
20.15
23.42
Size (MB) ↓
107
174
92
183
139.0
4DGS
PSNR ↑
25.01
21.02
26.23
21.03
23.32
Size (MB) ↓
1802
4832
1403
5233
3317.5
STG
PSNR ↑
26.30
23.40
26.21
21.25
24.29
Size (MB) ↓
540
542
366
1493
735.3
TABLE IV : Quantitative results and storage footprint on the ImViD dataset (300 frames for each scene).
Fig. 8 : Qualitative Comparison on ImViD Dataset.
Setting
Init. Pts (Ratio)
Final Pts
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Flow Init.
2.48M (100%)
4.52M
33.13
0.915
0.070
w Flow Init. (Ours)
0.29M ( 11.6% )
2.50M
33.51
0.916
0.070
TABLE V : Ablation on Flow-Guided Initialization. Our strategy maintains high reconstruction fidelity while significantly reducing initial primitive redundancy.
Fig. 9 : Ablation on Spatio-temporal Supervision. Qualitative results on Cam 10 show that removing depth and flow constraints leads to severe artifacts. Rendered flow maps reveal that lacking flow supervision induces chaotic background motion, directly causing severe temporal flickering.
Fig. 10 : Ablation on Joint Camera Temporal Calibration. Qualitative results on the test view (Cam 00) show that jointly optimizing camera temporal offsets effectively improves reconstruction quality in fast-moving regions, yielding a clearer texture and sharper motion boundaries.
Setting
λreg
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Δγ Opt.
-
30.59
0.910
0.035
w/ Δγ Opt.
0
29.93
0.903
0.033
w/ Δγ Opt.
10−2
30.50
0.911
0.031
w/ Δγ Opt. (Ours)
10−4
30.94
0.917
0.028
TABLE VI : Ablation on Joint Camera Temporal Calibration. Jointly optimizing camera temporal offsets ( Δγ ) improves reconstruction quality. Metrics are reported exclusively for the dynamic foreground involving the opera performers.
Loss Component
Peripheral View Metrics
Lflow
Ldepth
PSNR ↑
SSIM ↑
LPIPS ↓
-
-
24.43
0.729
0.287
✓
-
30.03
0.748
0.153
-
✓
30.45
0.760
0.134
✓
✓
30.98
0.772
0.124
TABLE VII : Ablation on Spatio-temporal Supervision. Metrics are evaluated on Cam 10 to reveal the contribution of geometric and motion constraints.
Rating
Spatial Perception
Sound Quality
Immersiveness
Very poor
0.00%
0.00%
0.00%
Poor
0.00%
0.00%
0.00%
Fair
14.28%
19.04%
9.52%
Good
23.80%
33.33%
42.85%
Excellent
61.90%
47.61%
47.61%
TABLE VIII : User study for the sound field construction. 21 participants rated the sound field reconstruction.
Method
Direction Score ↑
Distance Score ↑
w/o SFR
1.17
1.67
SFR + distance
1.69
3.02
SFR + direction
3.81
3.03
SFR + distance + direction
3.91
3.07
TABLE IX : Ablation studies on the Sound Field Reconstruction (SFR) baseline. 58 participants rated sense of direction and distance (Scale 1–5).
Fig. 11 : Visualization of the interaction trajectory and corresponding visual & auditory results. The multi-modal feedback changes consistently with the positions and orientations.
Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded visual worlds remain incomplete. This paper presents OmniDream, a training-free framework that transforms a silent monocular video into an immersive audiovisual experience, where viewers can freely look around while sounds remain spatially aligned with the visual scene. At the core of OmniDream is an object-centric audio representation that disentangles each sound source's intrinsic audio content from its scene-dependent acoustic effects, enabling independent audio generation, physics-based simulation of propagation effects, and flexible spatial audio rendering. Experiments show improved audio-visual alignment, spatial correctness, and perceptual immersiveness over baselines. Examples are available on https://huggingface.co/spaces/CuriousAlien000/spatial-audio-360-demo
Volumetric videos offer immersive 4D experiences, but remain difficult to reconstruct, store, and stream at scale. Existing Gaussian Splatting based methods achieve high-quality reconstruction but break down on long sequences, temporal inconsistency, and fail under large motions and disocclusions. Moreover, their outputs are typically incompatible with conventional video coding pipelines, preventing practical applications. We introduce PackUV, a novel 4D Gaussian representation that maps all Gaussian attributes into a sequence of structured, multi-scale UV atlas, enabling compact, image-native storage. To fit this representation from multi-view videos, we propose PackUV-GS, a temporally consistent fitting method that directly optimizes Gaussian parameters in the UV domain. A flow-guided Gaussian labeling and video keyframing module identifies dynamic Gaussians, stabilizes static regions, and preserves temporal coherence even under large motions and disocclusions. The resulting UV atlas format is the first unified volumetric video representation compatible with standard video codecs (e.g., FFV1) without losing quality, enabling efficient streaming within existing multimedia infrastructure. To evaluate long-duration volumetric capture, we present PackUV-2B, the largest multi-view video dataset to date, featuring more than 50 synchronized cameras, substantial motion, and frequent disocclusions across 100 sequences and 2B (billion) frames. Extensive experiments demonstrate that our method surpasses existing baselines in rendering fidelity while scaling to sequences up to 30 minutes with consistent quality.
Uncalibrated volumetric video streaming for human reconstruction is essential for holographic communication and AR/VR, yet remains challenging due to the need for temporal consistency and computational efficiency from sparse-view inputs. Existing methods rely on per-scene optimization or calibrated cameras, while recent feed-forward models are limited to low-resolution (0.5K) single-frame synthesis. We present HiReFF, a feed-forward method for 2K-resolution 360° human video reconstruction from uncalibrated sparse-view videos. Our framework decomposes the problem into two key tasks: foreground 3D Gaussian reconstruction from sparse-view videos (four views separated by 90°) and computationally efficient high-resolution synthesis. To enable the former, we propose Scale-synchronized Camera Calibration to resolve scale ambiguity for multi-view supervision, and Gaussian-wise Foreground Masking to reconstruct clean foregrounds by modulating Gaussian parameters. For efficient high-resolution synthesis, our High-resolution Side-tuning achieves 2K rendering by augmenting the Gaussian head with supplementary features while keeping the backbone at 0.5K, drastically reducing computational overhead. Experiments demonstrate that HiReFF significantly outperforms existing methods in high-resolution streaming volumetric video reconstruction. https://iridescentjiang.github.io/HiReFF
Yiming Jiang, Hanzhang Tu, Wenfeng Song +5
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Tsinghua University, Beijing, China · College of Computer Science, Beijing Information Science and Technology University, Beijing, China +1