RealMe: Immersive Volumetric Video from Minimal Captures

Chenxu Zhou; Zhiyuan Yu; Yudong Jin; Yinji Shentu; Yu Zhang; Jiaming Sun; Xiaowei Zhou

cs.CVOct 21, 2026

RealMe: Immersive Volumetric Video from Minimal Captures

Authors: Chenxu Zhou, Zhiyuan Yu, Yudong Jin, Yinji Shentu, Yu Zhang, Jiaming Sun, Xiaowei Zhou

Organizations: State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou, China · 4DV.ai, Hangzhou, China

Abstract

This paper addresses the problem of creating immersive volumetric videos from sparse and low-resolution capture resources. Conventional dense multi-view reconstruction methods can produce high-quality volumetric videos, but their reliance on costly capture setups makes them impractical in real-world scenarios. Recent approaches use generative models to relax capture requirements, but still suffer from limited resolution and 4D inconsistency. To overcome these constraints, we propose RealMe, a unified framework built around a high-resolution video diffusion model that enhances limited captures into high-resolution (2K), 4D-consistent videos for volumetric video reconstruction. Specifically, RealMe constructs auxiliary data from capture videos and encodes them into latents along with layout-aware positional embeddings. The video diffusion model adapts factorized spatio-temporal attention to the multi-view capture layout, denoises these latents across viewpoints and time, and decodes them into enhanced videos to refine the 4D representation. This adaptation helps the model capture temporal dynamics and multi-view relationships, leading to more 4D-consistent generation. To reduce the data gap between generated videos and real captures, we apply a latent frequency loss that provides frequency-domain supervision in latent space. Extensive experiments on the public DNA-Rendering benchmark and our self-captured RealVV dataset demonstrate that RealMe is able to create immersive volumetric videos and outperforms state-of-the-art methods in visual fidelity and 4D consistency.

Figures & tables

Figure 1. RealMe creates immersive volumetric video from sparse and low-quality captures. When directly optimized from sparse observations, FTGS ( Wang et al., 2025e ) struggles to maintain consistent reconstruction quality (left). In contrast, RealMe generates 2K multi-view videos with high visual quality and strong 4D consistency (middle), which further supports high-quality volumetric video reconstruction (right). Teaser comparison. The left part shows FTGS reconstruction from sparse observations. The middle part shows RealMe generation with 2K visual quality and 4D consistency. The right part shows RealMe reconstruction as volumetric video.

Figure 2. Overview of our framework. Given sparse and low-quality captures, RealMe first initializes a coarse 4DGS ( Wang et al., 2025e ) representation and uses it to render novel-view videos; in parallel, we detect ROIs with an offline detector ( Guo and Deng, 2024 ) and crop the input captures to preserve local detail cues. These auxiliary videos are fed into a high-resolution video diffusion model to produce enhanced videos, which are then used together with the input captures to optimize the 4DGS representation for volumetric video reconstruction. The detailed model architecture is illustrated in Fig. 3 . Overview of the RealMe framework, from limited input captures through auxiliary data construction and video enhancement to 4D reconstruction.

Figure 3. Model architecture. We use a ControlNet ( Zhang et al., 2023 ) branch to inject control signals from condition latents and text features. DiT blocks use these signals with layout-aware positional embeddings to denoise noisy latents via factorized spatio-temporal attention. A diagram of the video diffusion model architecture.

Figure 4. Factorized spatio-temporal attention. Full attention activates all sample pairs in the attention pattern, while the factorized variant keeps only pairs with the same temporal or spatial index. The horizontal and vertical axes denote key/value samples and query samples, respectively. Colored blocks indicate active attention regions, and gray blocks indicate inactive regions. Comparison of full and factorized attention patterns over temporal and spatial sample indices.

Method	Type	RealVV			DNA-Rendering ( Cheng et al., 2023 )
Method	Type	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
FTGS ( Wang et al., 2025e )	Optimization	19.0491	0.8142	0.2492	20.6439	0.7986	0.2158
DIFIX3D+ ( Wu et al., 2025b )	Generation	22.6373	0.8267	0.2890	21.9532	0.7920	0.1666
Diffuman4D ( Jin et al., 2025 )	Generation	24.4387	0.9113	0.2105	23.2704	0.7822	0.1427
Ours	Generation	27.2706	0.9393	0.1621	25.5956	0.8979	0.1261

Table 1. Quantitative comparison. We compare RealMe with optimization-based and generation-based baselines on the RealVV and DNA-Rendering ( Cheng et al., 2023 ) datasets. RealMe outperforms the baselines in visual quality across both datasets. Best and second-best results are shown in bold and underlined , respectively.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
4C4D ( Zhou et al., 2026 )	19.0405	0.8682	0.3716
4K4D ( Xu et al., 2024b )	15.6413	0.8181	0.5090
Ours	24.9304	0.8735	0.2278

Table 2. Representative novel view synthesis results on RealVV with four cameras. All methods use four input views. Best results are shown in bold .

Method	PSNR ${}_{c}\uparrow$	SSIM ${}_{c}\uparrow$	LPIPS ${}_{c}\downarrow$
Diffuman4D ( Jin et al., 2025 )	26.7098	0.8437	0.1913
Ours	28.0756	0.9246	0.1385

Table 3. Quantitative comparison on 4D-consistent reconstruction. PSNR c , SSIM c , and LPIPS c denote the consistency metrics defined in Sec. 5.1.3 . Best results are shown in bold .

Method	TWE $\downarrow$
FTGS ( Wang et al., 2025e )	0.00299
DIFIX3D+ ( Wu et al., 2025b )	0.04533
Diffuman4D ( Jin et al., 2025 )	0.03170
Ours	0.00551

Table 4. Temporal consistency comparison. TWE is defined in Sec. 5.1.3 . Best and second-best results are shown in bold and underlined , respectively.

Input Views	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
4	24.0490	0.8049	0.2705
8 (default)	27.2706	0.9393	0.1621
16	27.3614	0.9159	0.1389

Table 5. Ablation on the number of input views. We evaluate reconstruction quality on RealVV under different input-view settings. Best results are shown in bold .

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR ${}_{c}\uparrow$
w/o LA-PE	22.7354	0.9095	0.2355	25.2596
w/o FSA	24.3178	0.8995	0.2166	25.8435
Ours	24.8924	0.9226	0.1383	27.2688

Table 6. Ablation on model components. LA-PE denotes layout-aware positional embedding, FSA denotes factorized spatio-temporal attention, and PSNR c denotes the consistency metric defined in Sec. 5.1.3 .

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS ${}_{c}\downarrow$	GPU VRAM $\downarrow$
w/o $\mathcal{L}_{f}$	22.6017	0.8497	0.3113	72.4 GB
w/ $\mathcal{L}_{\mathrm{img}}$	24.7123	0.8644	0.2849	OOM (2K)
w/ $\mathcal{L}_{\mathrm{lat}}$	24.9641	0.8891	0.2201	73.5 GB

Table 7. Ablation on latent frequency loss.

\mathcal{L}_{f}

denotes frequency loss, while

\mathcal{L}_{\mathrm{img}}

and

\mathcal{L}_{\mathrm{lat}}

denote image-space and latent-space frequency loss, respectively. LPIPS c denotes the consistency metric defined in Sec. 5.1.3 . Metrics for

\mathcal{L}_{\mathrm{img}}

are computed at 1K resolution because 2K training runs out of memory.

Figure 5. Qualitative comparison for novel view synthesis on RealVV . Given the same sparse input captures, RealMe produces higher visual quality and more consistent novel views than optimization-based and generation-based baselines. Diffuman4D ( Jin et al., 2025 ) cannot generate scene backgrounds and generalizes poorly to zoomed-in long-focus views, while RealMe preserves both human appearance and surrounding scene content. Qualitative comparison for novel view synthesis on RealVV.

Figure 6. Qualitative comparison for detail enhancement on RealVV . Starting from low-resolution inputs, STAR ( Xie et al., 2025 ) and FlashVSR ( Zhuang et al., 2025 ) tend to produce over-smoothed regions or artificial local textures. In contrast, RealMe restores richer facial details, hair structures, and clothing textures with a more natural appearance. Please zoom in to better inspect the local visual quality. Qualitative comparison for detail enhancement on RealVV.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Runtime	Peak Memory
135.1 s	48.1 GB

Appendix

Table 8. Diffusion inference efficiency. We measure one 4-frame

2560\times 1440

batch with 30 denoising steps on a single NVIDIA H100 GPU.

Figure 7. Additional qualitative comparisons for novel view synthesis on RealVV . We compare FTGS ( Wang et al., 2025e ) , DIFIX3D+ ( Wu et al., 2025b ) , Diffuman4D ( Jin et al., 2025 ) , RealMe, and the ground truth under the same sparse-capture setting. The examples cover both short-focus and long-focus views. RealMe better preserves scene backgrounds, human appearance, and view-dependent details. Additional qualitative comparisons for novel view synthesis on RealVV.

Figure 8. Additional qualitative comparisons for novel view synthesis on DNA-Rendering. We compare FTGS ( Wang et al., 2025e ) , DIFIX3D+ ( Wu et al., 2025b ) , Diffuman4D ( Jin et al., 2025 ) , RealMe, and the ground truth using eight input views. The top and bottom four rows show two evaluation sequences across different held-out viewpoints and timestamps. Additional qualitative comparisons for novel view synthesis on DNA-Rendering.

Figure 9. Additional qualitative comparisons on long-focus views. We compare FTGS ( Wang et al., 2025e ) , DIFIX3D+ ( Wu et al., 2025b ) , generated views from RealMe, 4D reconstruction from RealMe, and the ground truth. Long-focus views stress the ability to recover high-resolution appearance under limited input coverage. RealMe produces cleaner details and more faithful scene appearance in these zoomed-in views. Additional qualitative comparisons on long-focus views.

Figure 10. Qualitative ablation on latent frequency loss. We compare generated RGB results with and without latent frequency loss (LFL), the RGB error map, and the corresponding low-frequency and high-frequency latent visualizations. The latent maps show that LFL better preserves high-frequency structures such as silhouettes, facial details, and clothing boundaries, complementing the quantitative ablation in the main paper. Qualitative ablation on latent frequency loss.

Figure 11. RealVV dataset overview. We show representative samples from RealVV , including short-focus captures, long-focus captures, foreground masks, and camera-rig views across different scenes. RealVV dataset overview with representative short-focus captures, long-focus captures, foreground masks, and camera-rig views.

4D Reconstruction

4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.

See all 140 papers

Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
Apr 10, 2026Zhengxian Yang, Shengqi Wang, Shi Pan +8Virtual Reality Dynamic Scene Reconstruction
Tsinghua University, Beijing, China. · the Institute of Automation, Chinese Academy of Sciences, Beijing, China. · the Migu Beijing Research Institute, Beijing, China +3
4D Human-Scene Reconstruction from Low-Overlap Captures
Jul 10, 2026Minhyuk Hwang, Sangmin Kim, Seunguk Do +24D Reconstruction Dynamic Scene Reconstruction
Seoul National University, Republic of Korea
Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation
Aug 11, 2025Xiaoyan Liu, Kangrui Li, Jiaxin Liu +2Camera-Controlled Video Generation Image-to-Video Generation
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales +1

Apr 10, 2026J/K move · Enter open · S save

Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement

Zhengxian Yang, Shengqi Wang, Shi Pan, Hongshuai Li+7

Tsinghua University, Beijing, China. · the Institute of Automation, Chinese Academy of Sciences, Beijing, China. · the Migu Beijing Research Institute, Beijing, China+3

Virtual Reality Dynamic Scene Reconstruction Image Reconstruction Multi-View 3D Reconstruction

Open paper AskPDF arXiv

Fully immersive experiences that tightly integrate 6-DoF visual and auditory interaction are essential for virtual and augmented reality. While such experiences can be achieved through computer-generated content, constructing them directly from real-world captured videos remains largely unexplored. We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces, audiovisual feedback, and high-resolution, high-frame-rate dynamic content. To support IVV construction, we present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy. Our custom capture rig enables synchronized multi-view video-audio acquisition during motion, facilitating efficient capture of complex indoor and outdoor scenes with rich foreground--background interactions and challenging dynamics. The dataset provides 5K-resolution videos at 60 FPS with durations of 1-5 minutes, offering richer spatial, temporal, and multimodal coverage than existing benchmarks. Leveraging this dataset, we develop a dynamic light field reconstruction framework built upon a Gaussian-based spatio-temporal representation, incorporating flow-guided sparse initialization, joint camera temporal calibration, and multi-term spatio-temporal supervision for robust and accurate modeling of complex motion. We further propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data. Together, these components form a unified pipeline for immersive volumetric video production. Extensive benchmarks and immersive VR experiments demonstrate that our pipeline generates high-quality, temporally stable audiovisual volumetric content with large 6-DoF interaction spaces. This work provides both a foundational definition and a practical construction methodology for immersive volumetric videos.

Abstract

4D Reconstruction

Explore similar work

Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement

4D Human-Scene Reconstruction from Low-Overlap Captures

Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation