Martine: Benchmarking Multi-View 3D Surface Reconstruction Across Viewpoint Coverage, Resolution, and Lighting
Authors: Robin Bruneau, Baptiste Brument, Frederic Giraud, Bastian Sigrist, Sascha Jecklin, Philipp Fürnstahl, Bjoern Menze, Lilian Calvet
Organizations: University of Zurich, Switzerland · IRIT, Université de Toulouse, CNRS, Toulouse, France · Balgrist University Hospital, University of Zurich, Switzerland · Independent researcher
Controlled acquisition setups are widely used for high-accuracy multi-view 3D surface reconstruction, as they allow sensor configuration, image resolution, viewpoint distribution, and illumination to be specified. Yet benchmarking remains difficult, since reconstruction accuracy depends jointly on the reconstruction paradigm, object properties, and acquisition regime. Existing benchmarks often restrict viewpoint coverage, resolution, or illumination, which can bias comparisons towards methods suited to the chosen setting. Others lack accurate 3D ground truth or quantified image-to-geometry alignment, limiting their use for metric surface evaluation. We introduce Martine, a benchmark for controlled 3D object surface reconstruction. Martine comprises 28 objects, including transparent materials and fine structures, captured at 9568 x 6376 pixels from 84 views over a spherical cap under both single-light and multi-light regimes. It provides object-independent camera calibration, accurate 3D ground-truth scans, calibration and image-to-geometry accuracy quantified on test images, and 3D property masks for analysing errors by material properties, visibility, curvature and fine structures. Experiments across photogrammetry, classical multi-view stereo, neural implicit surfaces, Gaussian-splatting-based reconstruction, and multi-view photometric stereo show that Martine enables regime-aware evaluation with explicit control over reference-data accuracy. The benchmark resolves sub-millimetric reconstruction differences, with a best observed Chamfer distance of 0.058 mm over the evaluated visible surface.
Figures & tables
Figure 1: Examples of image–GT misalignment in OpenSubstance [ 20 ] . Inaccurate COLMAP reconstructions of an opaque flower (left) and a transparent glass (right) in red. Registered GT scan meshes in green.
Figure 2: Dataset overview. (a) All objects in the proposed dataset, showing high geometric and optical diversity, including low-texture regions, fine structures, transparent and specular materials, and high-frequency geometric details. (b) Dense viewpoint sampling with wide angular coverage, obtained by coupling a robotic arm with a turntable. (c) Two examples of ground-truth meshes with manually curated challenge-level 3D masks highlighting specular regions (top, in blue) and transparent regions (bottom, in red) for challenge-level evaluation.
Figure 3: Acquisition setup. (a) Single-light acquisition regime, using an ambient illumination setup designed to be rotation-invariant around the turntable axis. (b) Multi-light acquisition regime, using a rig of 30 LEDs rigidly mounted around the camera. At each viewpoint, the LEDs are activated sequentially while the camera pose remains fixed.
rock
vase
hammer head
candle
can
glass
petanque ball
jam pot
1.06
1.08
1.02
0.78
1.21
1.14
0.84
1.26
Table 2: Leave-one-out registration residuals (mean reprojection error in pixels, over 8 control views per object.
Figure 4: Global precision, recall, and F-score curves pooled across all evaluated objects at quarter resolution (d4). Each curve aggregates all point-to-surface distances before computing the metric, so larger objects contribute proportionally more.
Figure 5: F-score at 1 mm on the vase object versus number of input views on a log scale (top) and image resolution (bottom). Crosses plotted at 0% F-score mark failed reconstructions in the 5-view and one-eighth-resolution settings.
Figure 6: Qualitative comparison of accuracy error heatmaps (viridis colour map, 0–4 mm, larger errors saturated) at quarter resolution (d4).
Figure 7: Radar chart of F-score at 0.5 mm per reconstruction challenge, at quarter resolution, for the five representative methods and Neuralangelo. Higher is better.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Calibration
#corners
Mean
Median
RMS
IQR
P95
Max
Textured cloth (ours)
6 480
0.542
0.524
0.584
0.298
0.931
1.489
ArUco cylinder [ 20 ]
6 480
0.666
0.542
0.824
0.555
1.621
4.010
COLMAP
6 480
0.809
0.799
0.913
0.612
1.559
2.917
ChArUco board [ 21 ]
6 480
1.086
0.924
1.312
0.865
2.541
5.199
Appendix
Table 3: Distribution of the per-corner reprojection residual on the test checkerboard on the 20 frontal views, in pixels at the full 9568×6376 resolution.
Figure 8: Calibration setup comparison. All configurations include a test checkerboard used only for evaluation. (a) Our approach uses SfM on the textured black cloth. The inset shows a brightness-enhanced crop of the cloth micro-texture, which provides point correspondences at the ultra-high image resolution. (b) ChArUco board [ 39 ] . (c) ArUco cylinder on the turntable base.
Figure 9: IoU limitations in assessing boundary precision. Visualisation of the discrepancy between SAM (magenta) and ground-truth (cyan) masks, with their intersection in orange. The boundary analysis reveals systematic contour shifts (4.80 and 4.51 px on average) that are critical for 2D–3D registration accuracy.
SAM
SAM + Canny (Ours)
Object
IoU ↑
Dist. (px) ↓
IoU ↑
Dist. (px) ↓
rock
0.9918
4.80
0.9972
1.28
warrior head
0.9913
4.51
0.9984
1.37
Appendix
Table 4: Contour refinement. The proposed Canny-based refinement significantly reduces the mean contour distance (px).
Figure 10: Sample of OpenSubstance objects that are unsuitable for rigorous 3D reconstruction analysis.
Figure 11: (Left) A visualisation of a single isophote (green) on the calibration sphere. (Right) Distribution of pixel intensity variations for each isophote across the sphere’s height. The standard deviation ranges from 0.0054 to 0.0267, with an average of 0.0141, confirming consistent illumination across viewpoints.
Table 6: Chamfer distance (mm, Eq. 7 ) / F-score at 0.5 mm (%) per object, method and image resolution (d1 to d8: full to one-eighth resolution), with all 84 views; Meshroom at d4 only; methods in two sub-tables, one under the other. Bold : best per row over both sub-tables, underline : second best, for each metric separately (lowest Chamfer distance, highest F-score). –: failure of the method (no reconstruction, Table 8 ). Mean : point distances of all objects pooled, as if they formed a single mesh, so that larger objects weigh more; a failed reconstruction keeps its ground-truth points in the pool, all missed: they lower the F-score, and lying beyond the cutoff of Eq. 7 they do not enter the Chamfer distance, so Chamfer means over different numbers of reconstructed objects ( Reconstructed row) are not directly comparable.
Views (d4)
Resolution (84 views)
Method
84
40
20
10
5
d1
d2
d4
d8
RealityScan
26
26
25
24
13
26
26
26
23
COLMAP
26
24
21
14
4
–
26
26
21
PGSR
26
24
21
13
3
–
26
26
22
NeuS2
26
26
26
26
26
–
26
26
26
RNb-NeuS2
26
26
26
26
26
–
26
26
26
Appendix
Table 7: Number of objects reconstructed (out of 26) per method, for each subset of input views at quarter resolution (d4) and for each image resolution with all 84 views. A missing reconstruction is a failure of the method (Table 8 ). –: configuration not run; at full resolution, the methods other than RealityScan fail for lack of GPU memory (Section 3.2 ).
Configuration
Failed
Objects
Meshroom d4
1/26
surgery scissors
RealityScan d4, 20 views
1/26
surgery scissors
RealityScan d4, 10 views
2/26
surgery scissors , whisk
RealityScan d4, 5 views
13/26
candle , glass , petanque ball , moka pot , lego , surgery scissors , spikes ball , surgery saw , whisk , plant , grid , bowl , spine
RealityScan d8
3/26
paw pad , surgery scissors , whisk
COLMAP d4, 40 views
2/26
paw pad , surgery scissors
Appendix
Table 8: Failed reconstructions per configuration. COLMAP and PGSR fail when structure from motion on the images of the configuration alone yields no usable sparse model (single attempt, default settings, no fallback); the other failures are runs that produced no mesh. Failed objects are excluded from all aggregated scores and declared as n/26 in the captions.
Recent advancements in neural surface reconstruction have significantly enhanced 3D reconstruction. However, current real world datasets mainly focus on benchmarking multiview stereo (MVS) based on RGB inputs. Multiview photometric stereo (MVPS) and multiview shape from polarization (MVSfP), though indispensable on high-fidelity surface reconstruction and sparse inputs, have not been quantitatively assessed together with MVS. To determine the working range of different MVX (MVS, MVSfP, and MVPS) techniques, we propose EvalMVX, a real-world dataset containing 25 objects, each captured with a polarized camera under 20 varying views and 17 light conditions including OLAT and natural illumination, leading to 8,500 images. Each object includes aligned ground-truth 3D mesh, facilitating quantitative benchmarking of MVX methods simultaneously. Based on our EvalMVX, we evaluate 13 MVX methods published in recent years, record the best-performing methods, and identify open problems under diverse geometric details and reflectance types. We hope EvalMVX and the benchmarking results can inspire future research on multiview 3D reconstruction.
Zaiyan Yang, Jieji Ren, Xiangyi Wang +5
Beijing University of Posts and Telecommunications, Beijing, China · School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · Independent Researcher +2
Multiview 3D evaluation assumes that the images being scored are observations of one static 3D scene. This assumption can fail in NVS and sparse-view reconstruction: inputs or generated outputs may contain artifacts, outlier frames, repeated views, or noise, yet still receive high 3D consistency scores. Existing reference-based metrics require ground truth, while ground-truth-free metrics such as MEt3R depend on learned reconstruction backbones whose failure modes are poorly characterized. We study this reliability problem by comparing neural reconstruction priors with classical geometric verification. We introduce \benchmark, a controlled robustness benchmark for multiview 3D consistency, and a parametric family that decomposes neural metrics into backbone, residual, and aggregation components. This family recovers MEt3R and yields variants up to 3× more robust. Our analysis shows that VGGT, MASt3R, DUSt3R, and Fast3R can hallucinate dense geometry and cross-view support for unrelated scenes, repeated images, and random noise. We introduce COLMAP-based metrics that use matches, registration, dense support, and reconstruction failure as failure-aware consistency signals. On real NVS outputs and a structured human study, these metrics achieve up to 4× higher correlation with human judgments than MEt3R.
Accurate 3D reconstruction of objects with reflective, transparent, or low-texture surfaces still remains notoriously challenging. Such materials often violate key assumptions in multi-view reconstruction pipelines, such as photometric consistency and the availability on distinct geometric texture cues. Existing datasets primarily focus on diffuse, textured objects, and therefore provide limited insight into performance under real-world material complexities. We introduce 3DReflecNet, a large-scale hybrid dataset exceeding 22 TB that is specifically designed to benchmark and advance 3D vision methods for these challenging materials. 3DReflecNet combines two types of data: over 120,000 synthetic instances generated via physically-based rendering of more than 12,000 shapes, and over 1,000 real-world objects captured using consumer devices. Together, these data consist of more than 7 million multi-view frames. The dataset spans diverse materials, complex lighting conditions, and a wide range of geometric forms, including shapes generated from both real and LLM-synthesized 2D images using diffusion-based pipelines. To support robust evaluation, we design benchmarks for five core tasks: image matching, structure-from-motion, novel view synthesis, reflection removal, and relighting. Extensive experiments demonstrate that state-of-the-art methods struggle to maintain accuracy across these settings, highlighting the need for more resilient 3D vision models.
Zhicheng Liang, Haoyi Yu, Boyan Li +6
The Chinese University of Hong Kong, Shenzhen · Capital Normal University · University of Southern California