Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
Figures & tables
Table 1: Different video generation schemes for 3D reconstruction. Long-Bi VDMs produce long trajectories in a single pass to cover diverse viewpoints. AR models sequentially generate long videos in an autoregressive manner. Multi-Bi-Mem (ours) achieves multiple consistent generations based on a powerful open-released VDM [ 78 ] with complementary viewpoints and memory mechanisms for integrated reconstruction.
Figure 2: Overview of WorldStereo. WorldStereo comprises two ControlNet branches. The camera branch ensures precise camera control and Global-Geometric Memory (GGM), depending on global point clouds; the Spatial-Stereo Memory (SSM) branch leverages retrieved reference frames and pointmap (3D correspondence) guidance obtained from the 3D cache to further preserve fine-grained consistency. We omit the diffusion noise part for simplicity.
Figure 3: Spatial-Stereo Memory (SSM). Reference views are retrieved from the memory bank, while pointmaps for both target and reference views are constructed based on the 3D cache. In SSM attention, we horizontally stitch each target-reference pair and rearrange the tensor shape to make each target frame’s features focus on the specifically retrieved reference. B, F, H, W, C indicate dimensions of batch, frame, height, width, and channels.
RotErr ↓
TransErr ↓
ATE ↓
Q-Align-I ↑
Q-Align-V ↑
CLIP-I ↑
CLIP-T ↑
CLIP-IQA+ ↑
Laion-Aes ↑
Time (sec)
Voyager [ 39 ]
0.678
0.630
1.343
3.279
0.664
82.63
11.399
0.414
4.994
343
SEVA [ 118 ]
0.171
0.540
1.023
3.907
0.782
91.74
11.194
0.514
5.331
90
Gen3C [ 66 ]
0.220
0.275
1.071
4.094
0.820
91.78
11.270
0.518
5.332
158
Uni3C [ 13 ]
0.155
0.192
0.572
4.202
0.846
91.63
11.115
0.549
5.350
162
WorldStereo*
0.132
0.178
0.542
4.273
0.860
91.94
11.145
0.559
5.352
162
WorldStereo-GGM
0.129
0.162
0.706
4.339
0.875
92.24
11.189
0.572
5.408
162
Table 2: Quantitative results of OOD benchmark with WorldScore [ 21 ] images . ∗ indicates the baseline version of our method without any memory mechanism, while the ‘full’ version denotes adding both GGM and SSM. The ‘DMD’ version is based on ‘WorldStereo-full’. Inference times are all tested with 8 H20 GPUs. † † † All methods are evaluated under their pre-defined resolutions and frame numbers, while the settings of Uni3C and WorldStereo series are the same for fairness (512p and 81-frame).
Method
F1-Score ↑
AUC ↑
RotErr ↓
TransErr ↓
ATE ↓
Tanks&Temples
Uni3C [ 13 ]
0.424
0.378
0.362
0.1017
0.1572
Gen3C [ 66 ]
0.416
0.380
0.342
0.0949
0.1704
SEVA [ 118 ]
0.286
0.293
0.379
0.0949
0.1815
Lyra [ 2 ]
0.227
0.193
–
–
–
VMem [ 49 ]
0.386
0.375
0.533
0.1510
0.1922
WorldStereo ∗
0.447
0.389
0.377
0.0990
0.1545
Table 3: Quantitative results of 3D reconstruction based on Tanks-and-Temples [ 45 ] and MipNeRF360 [ 4 ] . ∗ indicates the baseline version of our method without any memory mechanism.
Figure 4: Results of 3D reconstruction benchmark. The column (a) shows input views and ground-truth point clouds with pre-defined four trajectories (up, left, right rotations, and orbit). We compare the qualitative results of reconstructed point clouds (left) and generated novel views (right) for each method.
Figure 5: Ablation studies of memory components. Please see the red-framed regions to check the consistency compared to retrieved references. Baseline results are generated without any memory. GGM can capture coarse structures, but loses fine-grained details. Moreover, the incorporation of pointmap significantly enhances the consistency gained via the reference frames retrieved from the memory bank.
Figure 6: Results of 3D panorama generation. Please zoom-in for more details of reconstructed point clouds and novel views.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Train Processes
Scene Number
Camera
GGM
SSM
DMD
Total
Epoch
DL3DV
✓
✓
✓
✓
9,174
19,868
Re10K
✓
✓
26,544
12,000
Tartainair
✓
✓
✓
✓
2,214
2,142
Map-Free-Reloc
✓
✓
✓
✓
960
910
WildRGBD
✓
✓
✓
✓
23,005
3,000
Appendix
Table 4: Dataset details of WorldStereo. The training datasets used for various training processes include DL3DV [ 56 ] , Re10K [ 119 ] , Tartainair [ 86 ] , Map-Free-Reloc [ 1 ] , WildRGBD [ 92 ] , and UCo3D [ 57 ] . We dynamically sample subsets for each dataset across training epochs.
RotErr ↓
TransErr ↓
ATE ↓
PSNR ↑
SSIM ↑
LPIPS ↓
WorldStereo*
1.300
0.112
0.237
14.64
0.443
0.412
WorldStereo-GGM
0.699
0.067
0.131
17.45
0.532
0.288
WorldStereo-Full
0.748
0.079
0.142
18.40
0.561
0.283
WorldStereo-DMD
0.772
0.069
0.130
18.04
0.546
0.289
Appendix
Table 5: Quantitative ablation for memory components. ∗ indicates our baseline without any memory mechanism, while the ‘full’ version denotes adding both GGM and SSM. The ‘DMD’ version is based on ‘WorldStereo-full’.
RotErr ↓
TransErr ↓
ATE ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Uni3C
1.816
0.122
0.359
14.28
0.475
0.449
WorldStereo
1.384
0.115
0.347
14.27
0.473
0.441
WorldStereo-DMD
1.435
0.085
0.211
13.78
0.436
0.463
Appendix
Table 6: Quantitative results for camera control.
Figure 7: Qualitative Results of different resolutions. High-resolution (720 × 1280) yields sharper images with richer details than the low-resolution ones (480 × 768), as demonstrated in the red-boxed regions.
orbit
up
right
left
all
reference prop.
pano.
gen.
only panorama
46.0
43.4
38.4
38.5
167.2
100.0%
0.00%
up → right → left → orbit
50.7
43.4
39.6
41.3
175.0
65.17%
34.83%
orbit → up → right → left
46.9
46.2
41.3
42.8
177.1
54.89%
45.11%
orbit → right → left → up
46.9
46.9
41.1
42.8
177.6
53.66%
46.34%
up → orbit → right → left
49.2
43.4
41.3
42.8
176.7
55.55%
44.45%
Appendix
Table 7: Overlapping FoV scores of 3D panorama generation (trajectory order ablation) . Memory bank settings: ‘only panorama’ uses 24 views split from the panorama; others are incrementally updated with generations from different trajectory orders. Higher scores mean that more relevant frames are retrieved. ‘reference prop.’ indicates the proportion of retrieved frames belonging to panoramic (pano.) or generated (gen.) frames.