RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects

Yue Yin; Enze Tao; Weijian Deng; Dylan Campbell

cs.CVMay 9, 2025

RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects

Authors: Yue Yin, Enze Tao, Weijian Deng, Dylan Campbell

Organizations: Australian National University

Abstract

Modern 3D reconstruction and novel view synthesis approaches have demonstrated strong performance on scenes with opaque, non-refractive objects. However, most assume straight light paths and therefore cannot properly handle refractive and reflective materials. The lack of datasets specialized for these effects has impeded efforts to fairly and thoroughly evaluate performance and thereby make progress in this domain. Most existing datasets focus on opaque scenes, while those targeting refractive objects are often limited to thin glass with negligible ray bending, untinted single-material objects, or backgrounds treated as infinitely-distant, leaving key failure modes under strong refraction untested. To address this gap, we introduce the RefRef dataset. It contains 150 synthetic and 60 real scenes spanning varying levels of geometric complexity and diverse background types that expose the limitations of existing methods. For the purpose of benchmarking, we also provide an oracle method that, given the object geometry and refractive indices, calculates accurate light paths for neural rendering, and a simple two-stage baseline that relaxes these assumptions. We evaluate these against state-of-the-art methods and show that the task is far from solved.

Figures & tables

Datasets	$\|{\mathcal{S}}_{\mathrm{refr}}\|$	$\|{\mathcal{I}}_{\mathrm{refr}}\|$	Strong	Real	Multi	Bounded	Tinted
DTU [ 27 ]	0	0	✗	✗	✗	✗	✗
T & T [ 32 ]	0	0	✗	✗	✗	✗	✗
ShapeNet [ 11 ]	0	0	✗	✗	✗	✗	✗
OmniObject3D [ 59 ]	0	0	✗	✗	✗	✗	✗
Objaverse-XL [ 16 ]	0	0	✗	✗	✗	✗	✗
Shiny [ 56 ]	2	348	✓	✓	✓	✓	✓

Table 1: Comparison of datasets by counts of refractive scenes

|{\mathcal{S}}_{\mathrm{refr}}|

and images

|{\mathcal{I}}_{\mathrm{refr}}|

, and the presence of refractive scenes with predominantly thick or solid refractive objects (Strong), real data (Real), multi-material refractive objects (Multi), bounded backgrounds (Bounded), and tinted refractive materials (Tinted). ∗ Our estimates: the paper does not specify and the publicly released synthetic subset contains no strongly refractive scenes.

Figure 2: Overview of the benchmark’s oracle method. Starting with a straight ray (blue arrow), sample points are generated along its path. The scene’s geometry and refractive index update the ray trajectory, as shown in the red dashed box. Refracted and reflected rays are processed separately, producing updated sample positions and directions. The Zip-NeRF field then predicts color ${\mathbf{c}}_{i}$ and density $\sigma_{i}$ for each sample. Using Eq. 1 , the final color along each ray is rendered and combined via Eq. 6 .

	Method	Convex single-material					Nonconvex single-material					Nonconvex multi-material					Entire dataset
		PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$	PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$	PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$	PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$
Synthetic Scenes	NeuS	19.62	14.91	0.63	0.17	1.50	19.91	14.04	0.65	0.23	1.57	19.34	15.07	0.62	0.20	1.65	19.65	14.65	0.63	0.21	1.58
	Splatfacto	18.63	14.32	0.79	0.33	16.38	19.06	14.32	0.74	0.33	12.23	19.95	15.41	0.77	0.33	10.70	19.39	14.81	0.76	0.33	12.31
	Zip-NeRF	19.84	16.42	0.76	0.35	1.35	21.08	16.03	0.78	0.24	1.05	22.24	16.99	0.82	0.21	1.72	21.50	16.68	0.80	0.24	1.52
	TNSR	18.96	11.79	0.78	0.22	1.26	19.15	11.55	0.83	0.24	1.47	17.68	10.34	0.76	0.18	1.70	18.65	11.15	0.78	0.19	1.49
	MS-NeRF	18.86	15.48	0.73	0.35	2.18	20.06	15.38	0.74	0.29	2.26	21.41	16.41	0.78	0.26	1.86	20.34	15.78	0.76	0.30	2.08

Table 2: Quantitative results on the RefRef test set for novel view synthesis (PSNR, masked PSNR, SSIM, and LPIPS) and geometry estimation (DMAE). A dash denotes a result that cannot be reported due to method or ground-truth limitations. On the synthetic data, there is a large performance gap between the Oracle reference point and the next best approach (our simple baseline R3F), while all methods perform poorly on the real scenes with very blurry results from even the top-performing method Zip-NeRF. This demonstrates that refractive object reconstruction is very far from being solved. † Has access to ground-truth foreground geometry and material properties.

Panel of Figure 4 — Figure 4: Qualitative results on (a) synthetic and (b) real scenes. On the synthetic scenes, the refracted patterns are best recovered by oracle and R3F, while no method successfully recovers them on the real scenes. Reconstruction results are in the appendix ( Sec. F ).

Source	License	#	Objects
BlenderKit	Royalty Free	31	ampoule, beaker, candle holder, cat, cola bottle, conical flask, crystal vase, demijohn vase, diamond, flower vase, fox, household item set, Korken jar, lab equipment set, light bulb, magnifier, man sculpture, perfume red, perfume yellow, plastic bottle, reed diffuser, skull bottle, sleeping dragon, star-shaped bottle, teacup, teapot, Vardagen jar, vial, water pitcher, wine glass, wisolt kettle
	CC0	2	generic sculpture, woman sculpture
Blender	GPL	1	monkey
CGTrader	Royalty Free	3	graduated cylinder, round bottom flask, test tube
Free3D	Royalty Free	1	dog
Keenan’s	CC0	1	cow

Table 3: Source and license information for objects used in our dataset. Objects are obtained from BlenderKit, Blender, CGTrader, Free3D, Keenan’s 3D Model Repository, or created by the authors. Website-sourced objects were further modified ( e.g . , their materials and refractive indices) to better suit our dataset requirements.

Figure 5: R3F pipeline. Given multi-view posed images, R3F performs two-stage reconstruction. In Stage 1, object masks predicted by SAM 3 [ 9 ] are used to mask foreground rays while optimizing a background radiance field with Zip-NeRF [ 4 ] . In Stage 2, the object geometry and refractive index are estimated using NU-NeRF [ 47 ] . These estimates are then used to optimize a separate in-object foreground field via full ray tracing that models complex light interactions such as reflection, multi-bounce refraction, and total internal reflection, while keeping the background field frozen. The learned foreground field captures absorption and tinting effects within the refractive object and is combined with the background field to render novel views.

	Method	Convex single-material					Nonconvex single-material					Nonconvex multi-material					Entire dataset
		PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$	PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$	PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$	PSNR $\uparrow$	PSNR M $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DMAE $\downarrow$
Cube BG	NeuS	18.48	14.94	0.66	0.19	1.20	19.11	14.86	0.67	0.33	1.17	19.19	15.97	0.63	0.20	1.31	19.15	15.33	0.65	0.25	1.23
	Splatfacto	22.32	14.90	0.87	0.11	14.56	24.40	16.02	0.88	0.07	9.42	24.70	17.72	0.86	0.10	11.03	24.26	16.69	0.87	0.09	10.96
	Zip-NeRF	17.98	14.52	0.69	0.37	1.31	20.28	15.20	0.75	0.23	1.57	21.43	16.02	0.79	0.19	1.88	20.40	15.44	0.76	0.24	1.66
	TNSR	19.30	11.92	0.71	0.26	0.90	18.92	11.25	0.83	0.22	1.31	17.66	10.06	0.71	0.16	1.19	18.78	11.02	0.75	0.20	1.10
	MS-NeRF	17.77	14.26	0.67	0.34	0.40	19.76	14.86	0.73	0.24	0.37	21.48	16.61	0.78	0.22	0.45	20.07	15.44	0.74	0.25	0.41

Table 4: Detailed quantitative breakdown of the RefRef test set by background type and object geometry. The synthetic scenes comprise three background types: two categories of bounded backgrounds, including high-frequency cube and sphere patterns, and one category of unbounded backgrounds consisting of multiple environment maps. The indoor real scenes consist of the living room and puzzle table captures, while the outdoor real scenes consist of the courtyard and lookout-deck captures. For environment-map scenes, DMAE is evaluated only within the foreground object mask, as the background has no finite depth. Real-scene DMAE is unavailable. The best and second-best results within each background/geometry block are shown in bold and underlined, respectively. A dash denotes an unavailable result. † Has access to ground-truth foreground geometry and material properties.

Table 8

Panel of Figure 6 — Figure 6: Comparison of the original [ 3 ] distortion loss ${\mathcal{L}}_{\text{dist}}^{\text{orig}}$ and the translucency-corrected distortion loss ${\mathcal{L}}_{\text{dist}}$ . ( 6(a) – 6(b) ) Sample point and ( 6(c) – 6(d) ) weight distributions.

Method	PSNR PSNR M $\uparrow$ [-1pt] $\uparrow$	PSNR M PSNR M $\uparrow$ [-1pt] $\uparrow$	SSIM $\uparrow$ PSNR M $\uparrow$ [-1pt] $\times 10^{2}$	LPIPS $\downarrow$ PSNR M $\uparrow$ [-1pt] $\times 10^{2}$	DMAE $\downarrow$ PSNR M $\uparrow$ [-1pt] $\times 10^{2}$
Oracle	31.64	25.37	96.13	2.76	4.38
w/o corrected ${\mathcal{L}}_{\mathrm{dist}}$	31.59	25.31	96.12	2.78	4.07
w/o R	29.65	22.92	94.69	3.67	2.97
w/o TIR	25.85	19.27	90.84	8.38	15.86
w/o Zip-NeRF	24.68	21.45	87.79	11.63	27.42

Table 7: Ablation study of oracle on convex single-material objects with cube backgrounds. Ablated components include distortion-loss correction, first surface reflection, total internal reflection, and Zip-NeRF [ 4 ] backbone replaced by Nerfacto [ 49 ] .

Panel of Figure 7 — Figure 7: Additional qualitative comparison of novel view synthesis results on bounded synthetic scenes. The object is enclosed by a cube or a sphere textured with a high-frequency pattern image. Each block shows one scene: the top row gives the rendered RGB images and the bottom row the corresponding distance maps, which expose geometric errors not apparent from RGB alone.

Panel of Figure 8 — Figure 8: Additional qualitative comparison of novel view synthesis results on unbounded synthetic scenes. The background is an infinitely distant high-dynamic-range environment map rather than enclosing geometry. Each block shows one scene: the top row gives the rendered RGB images and the bottom row the corresponding distance maps, which expose geometric errors not apparent from RGB alone.

Panel of Figure 9 — Figure 9: Additional qualitative comparison of novel view synthesis results on real scenes. (a) shows results on indoor scenes, including living room and puzzle table backgrounds, while (b) shows results on outdoor scenes, including courtyard and lookout deck backgrounds.

Panel of Figure 10 — Figure 10: Geometry visualization of the reconstruction results across different methods on the test set from the RefRef dataset. Inspecting the reconstructed geometry directly exposes errors that are difficult to observe from the rendered RGB images or the distance maps alone, such as fragmented surfaces and missing structure.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 992 papers

3DReflecNet: A Large-Scale Dataset for 3D Reconstruction of Reflective, Transparent, and Low-Texture Objects
May 11, 2026Zhicheng Liang, Haoyi Yu, Boyan Li +63D Reconstruction Benchmarks 3D Reconstruction
The Chinese University of Hong Kong, Shenzhen · Capital Normal University · University of Southern California
ICTPolarReal: A Polarized Reflection and Material Dataset of Real World Objects
Mar 26, 2026Jing Yang, Krithika Dharanikota, Emily Jia +2Inverse Rendering Intrinsic Image Decomposition
Institute for Creative Technologies, University of Southern California
RefracGS: Novel View Synthesis Through Refractive Water Surfaces with 3D Gaussian Ray Tracing
Mar 23, 2026Yiming Shao, Qiyu Dai, Chong Gao +6Novel View Synthesis 3D Reconstruction
Shandong University, China · Peking University, Beijing, China · Sun Yat-sen University, China +2

May 11, 2026J/K move · Enter open · S save

3DReflecNet: A Large-Scale Dataset for 3D Reconstruction of Reflective, Transparent, and Low-Texture Objects

Zhicheng Liang, Haoyi Yu, Boyan Li, Dayou Zhang+5

The Chinese University of Hong Kong, Shenzhen · Capital Normal University · University of Southern California

3D Reconstruction Benchmarks 3D Reconstruction Multi-View 3D Reconstruction Novel View Synthesis

Open paper AskPDF arXiv

Accurate 3D reconstruction of objects with reflective, transparent, or low-texture surfaces still remains notoriously challenging. Such materials often violate key assumptions in multi-view reconstruction pipelines, such as photometric consistency and the availability on distinct geometric texture cues. Existing datasets primarily focus on diffuse, textured objects, and therefore provide limited insight into performance under real-world material complexities. We introduce 3DReflecNet, a large-scale hybrid dataset exceeding 22 TB that is specifically designed to benchmark and advance 3D vision methods for these challenging materials. 3DReflecNet combines two types of data: over 120,000 synthetic instances generated via physically-based rendering of more than 12,000 shapes, and over 1,000 real-world objects captured using consumer devices. Together, these data consist of more than 7 million multi-view frames. The dataset spans diverse materials, complex lighting conditions, and a wide range of geometric forms, including shapes generated from both real and LLM-synthesized 2D images using diffusion-based pipelines. To support robust evaluation, we design benchmarks for five core tasks: image matching, structure-from-motion, novel view synthesis, reflection removal, and relighting. Extensive experiments demonstrate that state-of-the-art methods struggle to maintain accuracy across these settings, highlighting the need for more resilient 3D vision models.

Abstract

3D Reconstruction

Explore similar work

3DReflecNet: A Large-Scale Dataset for 3D Reconstruction of Reflective, Transparent, and Low-Texture Objects

ICTPolarReal: A Polarized Reflection and Material Dataset of Real World Objects

RefracGS: Novel View Synthesis Through Refractive Water Surfaces with 3D Gaussian Ray Tracing