ZipMVS: Multi-View Stereo with Compressed Cost Volumes
Authors: Guanglin Jin, Hongshan Yu, Javier Civera, Zhaoxin Li
Organizations: School of Artificial Intelligence and Robotics, Hunan University, Changsha 410082, China · I3A, Universidad de Zaragoza, Zaragoza 50018, Spain · Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081, China Key Laboratory of Agricultural Big Data, Ministry of Agriculture and Rural Affairs, Beijing 100081, China
Multi-view stereo (MVS) methods typically deliver highly accurate 3D reconstructions from multiple registered RGB images, thanks to the highly informative, geometric constraints between them. However, their substantial memory requirements remain a major obstacle for deployment in domains such as aerospace and autonomous systems, where resource efficiency is critical. In this work, we introduce ZipMVS, an MVS method specifically designed for efficient high-quality reconstruction. We propose a novel depth-hypothesis strategy that enables substantial compression of the cost volume, hence greatly reducing GPU memory consumption while preserving reconstruction accuracy. Experiments on the DTU and Tanks and Temples datasets show that ZipMVS achieves competitive reconstruction quality compared with other efficiency-oriented MVS methods, while achieving a competitive balance between reconstruction quality and GPU memory usage. The code is available at https://github.com/JihnGlyn/ZipMVS
Figures & tables
Figure 1: Overview of ZipMVS. Multi-view images are first processed by the feature extractor. Depth estimation begins at level 3, and in subsequent levels, new depth hypotheses are generated using the confidence and depth maps from the previous level. The final refined depth map is produced as the output.
Figure 2: Demonstration of the Advanced Depth Hypothesis Strategy, which contains a Differentiable Depth Range module (DDR) and a GRU-based Depth Speculator module (GDS).
Methods
Acc.(mm)↓
Comp.(mm)↓
Overall (mm)↓
Camp [ 40 ]
0.935
0.554
0.695
Furu [ 11 ]
0.613
0.941
0.777
Tola [ 41 ]
0.342
1.190
0.766
Gipuma [ 18 ]
0.283
0.873
0.578
MVSNet [ 4 ]
0.396
0.527
0.462
R-MVSNet [ 27 ]
0.383
0.452
0.417
Table 1: Results for baselines and ZipMVS on DTU. The lower the better for accuracy (Acc.), completeness (Comp.), and overall. Best result is bolded, and second-best is in italic bold. The first horizontal line separates traditional (above) from learning-based (below) methods.
Figure 3: Qualitative reconstruction results of our ZipMVS on the 22 subsets of the DTU test set.
Figure 4: Depth maps generated by PatchmatchNet [ 8 ] ( 2nd column), IterMVS [ 7 ] ( 3rd column), and our ZipMVS ( 4th column) for scan1 ( 1st row), scan48 ( 2nd row), and scan77 ( 3rd row) of the DTU dataset.
Methods
Intermediate Dataset
Advanced Dataset
Precision↑
Recall↑
F-score↑
Precision↑
Recall↑
F-score↑
Colmap [ 19 ]
43.16
44.48
42.14
33.65
23.96
27.24
MVSNet [ 4 ]
40.23
49.70
43.48
-
-
-
R-MVSNet [ 27 ]
43.74
57.60
48.40
31.47
22.05
24.91
Point-MVSNet [ 6 ]
41.27
60.13
48.27
-
-
-
CasMVSNet [ 10 ]
47.62
74.01
56.84
29.68
35.24
31.12
Table 2: Results on Tanks and Temples for baselines trained only on DTU. The higher the better for precision, recall and F-score.
Figure 5: Depth maps (first row) and point clouds (second row) estimated by PatchmatchNet [ 8 ] (left column), IterMVS [ 7 ] (central column), and our ZipMVS (right column) on the Auditorium set of Tanks and Temples.
Methods
Mem.(MB)
Runtime(s)
Overall(mm)↓
CasMVSNet [ 10 ]
4591
-
0.355
PatchmatchNet [ 8 ]
1629
0.139
0.352
IterMVS [ 7 ]
886
0.093
0.363
ZipMVS (5 views)
1322
0.112
0.340
Table 3: Comparison of several methods in terms of memory usage, run-time on the DTU dataset. Bold and italic bold indicate the best and the second-best, respectively.
Figure 6: Overall error (mm) and GPU memory usage (MB) of several MVS baselines and our ZipMVS on the DTU dataset at an input resolution of 1152×1600 . Points closer to the lower-left corner indicate a better trade-off between reconstruction quality and memory footprint. ZipMVS stands out as a practical alternative: it is surpassed only by MVSFormer++ [ 45 ] , yet requires over 3× less GPU memory. Compared to the remaining baselines, ZipMVS delivers superior accuracy while achieving competitive memory efficiency compared to many existing methods.
Figure 7: Reconstruction results on our in-house synthetic dataset. The first row shows the output of PatchmatchNet [ 8 ] , while the second row shows the reconstruction produced by our ZipMVS. Note the higher quality of ZipMVS results, in particular in the parts framed in red.
Methods
Acc.(mm)↓
Comp.(mm)↓
Overall(mm)↓
w/o AR & CD
0.383
0.383
0.383
w/o AR
0.352
0.351
0.352
w/o CD
0.362
0.366
0.364
ZipMVS (ours)
0.369
0.284
0.327
Table 4: Ablation study evaluating the impact of the Adaptive Range (AR) and Central Densely Interval (CD) modules on depth hypotheses generation.
Figure 8: (a) Reference image. (b) The range map produced by ARNet. (c) The eccentricity map, eccentricity between the depth speculated by the GDS module and the previous estimated depth. The brighter the area, the higher the value, and vice versa.
Level 2
Level 1
Acc.(mm)↓
Comp.(mm)↓
Overall(mm)↓
C
C
0.352
0.373
0.362
S
C
0.342
0.364
0.353
C
S
0.336
0.363
0.349
S
S
0.333
0.346
0.340
Table 5: Ablation study concerning the effects of GRU-based Depth Speculator (GDS) module. Symbols S and C denote configurations with the speculated depth (GDS) and compensated depth, respectively. The number of input views N is set to 5 .
N
Acc.(mm)↓
Comp.(mm)↓
Overall(mm)↓
Mem.(MB)
Runtime(s)
2
0.4617
0.2424
0.3520
1102
0.075
3
0.4650
0.2354
0.3502
1175
0.088
4
0.3692
0.2840
0.3266
1250
0.096
5
0.3331
0.3460
0.3396
1322
0.112
6
0.3341
0.3448
0.3395
1396
0.125
Table 6: Ablation study on the number of input views N on DTU.
Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University · Dim12 AI Inc · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University