Detailed Geometry and Appearance from Opportunistic Motion

Ryosuke Hirai; Kohei Yamashita; Antoine Guédon; Ryo Kawahara; Vincent Lepetit; Ko Nishino

Figure 1: We introduce a method to recover high-fidelity 3D geometry and appearance from sparse-view static videos. By leveraging “virtual viewpoints” induced by opportunistic object motion, our method jointly optimizes object pose and geometry through a motion-aware appearance representation to capture detailed surface structure.

Figure 2: Method Overview. Given sparse multi-view videos and an initial set of canonical 2D Gaussians (a), our framework recovers per-frame object poses while iteratively refining the Gaussian geometry via differentiable rendering. To ensure robust convergence under sparse supervision, we employ an alternating optimization that switches between 6-DoF object pose estimation (b) and canonical Gaussian refinement (c) using the aggregated temporal information from all processed frames.

Figure 3: Motion-Aware Appearance Modeling. (a) For moving objects, specular reflection is a function of incident radiance from the reflected viewing direction

\boldsymbol{\omega}_{r}

, which evolves with object pose. (b) Similarly, diffuse reflection depends on the time-varying surface normal

\mathbf{n}

relative to the static environment. (c) Our model factorizes appearance into specular (e) and diffuse (f) components by evaluating the surface normal and reflected viewing directions in the world coordinate system. (d) Unlike standard 3DGS, which optimizes independent Spherical Harmonics (SH) for each primitive, our approach employs shared SH coefficients

\theta_{d}

and

\theta_{s}

across all foreground Gaussians to robustly capture the global illumination field.

Method	drain cleaner			ottoman			bunny			garden gnome			drill			Average
Method	Mean $\downarrow$	Med. $\downarrow$	P80 $\downarrow$	Mean $\downarrow$	Med. $\downarrow$	P80 $\downarrow$	Mean $\downarrow$	Med. $\downarrow$	P80 $\downarrow$	Mean $\downarrow$	Med. $\downarrow$	P80 $\downarrow$	Mean $\downarrow$	Med. $\downarrow$	P80 $\downarrow$	Mean $\downarrow$	Med. $\downarrow$	P80 $\downarrow$
DG-Mesh [ 21 ]	28.3	24.5	41.7	26.9	22.7	39.2	33.8	30.5	49.3	26.0	22.3	38.6	26.9	23.6	39.3	28.4	24.7	41.6
Single-Frame	11.8	7.2	16.3	18.1	13.0	20.0	18.4	14.4	25.9	22.9	16.5	35.2	21.3	15.7	32.5	18.5	13.4	26.0
w/o Motion-Aware Appearance	8.4	4.8	12.0	17.9	13.0	20.0	15.3	12.4	21.6	20.7	15.9	31.6	16.3	10.1	24.7	15.7	11.2	22.0
w/o Alternating Estimation	10.1	7.7	14.1	17.5	12.2	18.5	12.4	9.2	17.2	22.1	15.8	33.9	15.2	9.6	22.1	15.5	10.9	21.1
w/o Specular	8.8	5.6	11.9	18.4	13.8	20.5	16.1	13.0	22.3	21.1	17.0	31.8	16.0	10.9	22.2	16.1	12.1	21.8

Table 1: Surface normal accuracy for synthetic data. We evaluate reconstruction quality on synthetic data using the mean, median, and 80th percentile errors against ground-truth surface normals. The results demonstrate that our proposed components significantly enhance the recovery of fine surface details. While simpler surfaces, such as the drain cleaner, can be handled by baseline methods, they do not fully reflect our method’s capability to reconstruct intricate geometric details.

Method	drain cleaner				ottoman				bunny
Method	PSNR $\uparrow$	L1 $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	L1 $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	L1 $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DG-Mesh [ 21 ]	35.23	0.00140	0.9921	0.0181	28.62	0.00515	0.9873	0.0186	36.26	0.00218	0.9895	0.0206
Single-Frame	40.03	0.00104	0.9947	0.0082	29.52	0.00454	0.9880	0.0148	39.42	0.00149	0.9931	0.0096
w/o Motion-Aware Appearance	37.76	0.00209	0.9961	0.0059	29.45	0.00419	0.9890	0.0148	32.63	0.00388	0.9897	0.0097
w/o Alternating Estimation	41.74	0.00096	0.9961	0.0066	32.88	0.00277	0.9904	0.0145	43.56	0.00094	0.9969	0.0062
w/o Specular	40.13	0.00093	0.9959	0.0067	29.84	0.00406	0.9894	0.0148	40.93	0.00118	0.9963	0.0072

Table 2: Novel view synthesis accuracy on synthetic data. The results highlight that our proposed appearance model is essential for accurately recovering object appearance from sparse multi-view observations.

Method	drain cleaner		ottoman		bunny		garden gnome		drill		Average
Method	CD $\downarrow$	Normal $\downarrow$	CD $\downarrow$	Normal $\downarrow$	CD $\downarrow$	Normal $\downarrow$	CD $\downarrow$	Normal $\downarrow$	CD $\downarrow$	Normal $\downarrow$	CD $\downarrow$	Normal $\downarrow$
DG-Mesh [ 21 ]	0.045	33.78	0.066	30.42	0.036	39.12	0.027	30.67	0.032	29.71	0.041	32.74
Single-Frame	0.051	30.48	0.030	27.92	0.036	31.07	0.021	32.75	0.028	33.71	0.033	31.19
w/o Motion-Aware Appearance	0.013	12.79	0.042	31.75	0.035	26.41	0.038	36.24	0.014	18.75	0.028	25.19
w/o Alternating Estimation	0.012	13.90	0.028	19.77	0.024	18.99	0.024	28.45	0.016	22.32	0.021	20.69
w/o Specular	0.012	14.83	0.034	26.99	0.023	20.98	0.021	28.93	0.013	19.89	0.021	22.32

Table 3: Surface mesh reconstruction on synthetic data. We evaluate geometric accuracy using Chamfer Distance (CD,

\downarrow

) and Normal Error (

\downarrow

). The results demonstrate the superior precision of our reconstruction compared to baseline methods.

Figure 4: Visualization of recovered surface normals. The error maps on the right visualize per-pixel estimation discrepancies. These results demonstrate that proper appearance modeling is essential for effectively leveraging radiometric cues to reconstruct fine surface details.

Figure 5: Novel view Synthesis. Our appearance model significantly improves the fidelity of novel view synthesis when applied to objects with specularity.

Figure 6: Surface normals and mesh models recovered from real-world datasets. Qualitative results on HO3D [ 14 ] and HODome [ 47 ] demonstrate the effectiveness and robustness of our method across diverse human-object interaction scenarios.

Figure 7: Visualization of alternating estimation progress. We show the recovered geometry and object poses (represented as effective camera views) at several timesteps. As the temporal window expands and the effective camera coverage becomes more informative, the geometric accuracy of the reconstruction progressively improves.

Method	HO3D					HODome
Method	CD $\downarrow$	PSNR $\uparrow$	L1 $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD $\downarrow$	PSNR $\uparrow$	L1 $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Single-Frame	0.011	27.60	0.01017	0.9565	0.0412	0.024	39.00	0.00101	0.9946	0.0062
w/o Motion-Aware Appearance	0.004	27.09	0.01344	0.9580	0.0345	0.017	38.04	0.00127	0.9951	0.0057
Ours	0.004	28.11	0.01017	0.9625	0.0296	0.016	39.71	0.00099	0.9957	0.0056

Table 4: Quantitative results on real-world datasets. Our method achieves better geometry accuracy and novel view synthesis quality than the baselines.

Figure 8: Results on in-the-wild capture. Our method reconstructs faithful appearance and geometry from real-world capture, including specular highlight on the object.

Figure 9: Additional Visualization of Recovered Surface Normals. Our recovered surface normals are at least comparable to those of the baselines, suggesting the robustness of our method.

Figure 10: Additional Novel View Synthesis Results. Our appearance model improves the fidelity of novel view synthesis on different objects. The ottoman (the second row) remains challenging due to its limited surface-normal variation.

Figure 11: Qualitative comparison with DG-Mesh [ 21 ] . While DG-Mesh supports deformable objects, it struggles with the sparse inputs provided by static cameras.

Figure 12: Effectiveness of diffuse and specular components for surface normal recovery. Modeling both diffuse and specular components is critical for recovering accurate surface normals on objects with diverse material properties, where view-dependent effects can otherwise bias the reconstruction.

Figure 13: Effectiveness of diffuse and specular components for novel view synthesis. Our full model can faithfully reproduce both specular highlights and diffuse shading.

Figure 14: Visualization of specular and diffuse components. In the second column of the specular components, we show the results with increased brightness for better visibility. Our method successfully decomposes the object appearance into specular and diffuse components, except for the challenging case of the ottoman.

Table 5: Comparison with per-Gaussian SH, a variant of our method that optimizes SH coefficients independently for each Gaussian. We report (a) surface normal accuracy and (b) novel view synthesis quality on synthetic data. Optimizing per-Gaussian SH can be unstable on complex objects ( e.g . , the bunny), leading to degraded performance.

Figure 15: Qualitative comparison with per-Gaussian SH. In particular, our full model yields sharper reconstructions of fine surface details ( e.g . , the wrinkles on the bunny).

Figure 16: Comparison with our method without alternating estimation (“w/o Alternating Estimation”). In the “10 frames,” “15 frames,” and “29 frames” settings, the object rotates by approximately 39, 26, and 13 degrees, respectively, between adjacent frames. Our full method successfully recovers geometry and appearance even in the temporally sparsest setting.

Figure 17: Effectiveness of segmentation mask refinement. We compare our full method with a variant without mask refinement (“w/o Mask Refinement”), where the per-Gaussian mask values are fixed during optimization. Refining the mask values jointly with geometry and appearance is essential for converging to the true fine-scale geometry.

Figure 18: Surface normals and mesh models recovered from real-world datasets. The results demonstrate the effectiveness and robustness of our method across diverse human-object interaction scenarios.

Detailed Geometry and Appearance from Opportunistic Motion

Abstract

Figures & tables

3D Reconstruction

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images

Artic-O: End-to-End Articulated Object Reconstruction via Latent Geometry Learning

4DGS360: 360° Gaussian Reconstruction of Dynamic Objects from a Single Video

Abstract

3D Reconstruction

Explore similar work

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images

Artic-O: End-to-End Articulated Object Reconstruction via Latent Geometry Learning

4DGS360: 360° Gaussian Reconstruction of Dynamic Objects from a Single Video