Detailed Geometry and Appearance from Opportunistic Motion
Organizations: Graduate School of Informatics, Kyoto University, Japan · École Polytechnique, France · LIGM, École Nationale des Ponts et Chaussées, IP Paris, Univ Gustave Eiffel, CNRS, France
Abstract
Reconstructing 3D geometry and appearance from a sparse set of fixed cameras is a foundational task with broad applications, yet it remains fundamentally constrained by the limited viewpoints. We show that this bound can be broken by exploiting opportunistic object motion: as a person manipulates an object~(e.g., moving a chair or lifting a mug), the static cameras effectively ``orbit'' the object in its local coordinate frame, providing additional virtual viewpoints. Harnessing this object motion, however, poses two challenges: the tight coupling of object pose and geometry estimation and the complex appearance variations of a moving object under static illumination. We address these by formulating a joint pose and shape optimization using 2D Gaussian splatting with alternating minimization of 6DoF trajectories and primitive parameters, and by introducing a novel appearance model that factorizes diffuse and specular components with reflected directional probing within the spherical harmonics space. Extensive experiments on synthetic and real-world datasets with extremely sparse viewpoints demonstrate that our method recovers significantly more accurate geometry and appearance than state-of-the-art baselines.
Figures & tables
| Method | drain cleaner | ottoman | bunny | garden gnome | drill | Average | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean | Med. | P80 | Mean | Med. | P80 | Mean | Med. | P80 | Mean | Med. | P80 | Mean | Med. | P80 | Mean | Med. | P80 | |
| DG-Mesh [ 21 ] | 28.3 | 24.5 | 41.7 | 26.9 | 22.7 | 39.2 | 33.8 | 30.5 | 49.3 | 26.0 | 22.3 | 38.6 | 26.9 | 23.6 | 39.3 | 28.4 | 24.7 | 41.6 |
| Single-Frame | 11.8 | 7.2 | 16.3 | 18.1 | 13.0 | 20.0 | 18.4 | 14.4 | 25.9 | 22.9 | 16.5 | 35.2 | 21.3 | 15.7 | 32.5 | 18.5 | 13.4 | 26.0 |
| w/o Motion-Aware Appearance | 8.4 | 4.8 | 12.0 | 17.9 | 13.0 | 20.0 | 15.3 | 12.4 | 21.6 | 20.7 | 15.9 | 31.6 | 16.3 | 10.1 | 24.7 | 15.7 | 11.2 | 22.0 |
| w/o Alternating Estimation | 10.1 | 7.7 | 14.1 | 17.5 | 12.2 | 18.5 | 12.4 | 9.2 | 17.2 | 22.1 | 15.8 | 33.9 | 15.2 | 9.6 | 22.1 | 15.5 | 10.9 | 21.1 |
| w/o Specular | 8.8 | 5.6 | 11.9 | 18.4 | 13.8 | 20.5 | 16.1 | 13.0 | 22.3 | 21.1 | 17.0 | 31.8 | 16.0 | 10.9 | 22.2 | 16.1 | 12.1 | 21.8 |
| Method | drain cleaner | ottoman | bunny | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PSNR | L1 | SSIM | LPIPS | PSNR | L1 | SSIM | LPIPS | PSNR | L1 | SSIM | LPIPS | |
| DG-Mesh [ 21 ] | 35.23 | 0.00140 | 0.9921 | 0.0181 | 28.62 | 0.00515 | 0.9873 | 0.0186 | 36.26 | 0.00218 | 0.9895 | 0.0206 |
| Single-Frame | 40.03 | 0.00104 | 0.9947 | 0.0082 | 29.52 | 0.00454 | 0.9880 | 0.0148 | 39.42 | 0.00149 | 0.9931 | 0.0096 |
| w/o Motion-Aware Appearance | 37.76 | 0.00209 | 0.9961 | 0.0059 | 29.45 | 0.00419 | 0.9890 | 0.0148 | 32.63 | 0.00388 | 0.9897 | 0.0097 |
| w/o Alternating Estimation | 41.74 | 0.00096 | 0.9961 | 0.0066 | 32.88 | 0.00277 | 0.9904 | 0.0145 | 43.56 | 0.00094 | 0.9969 | 0.0062 |
| w/o Specular | 40.13 | 0.00093 | 0.9959 | 0.0067 | 29.84 | 0.00406 | 0.9894 | 0.0148 | 40.93 | 0.00118 | 0.9963 | 0.0072 |
| Method | drain cleaner | ottoman | bunny | garden gnome | drill | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CD | Normal | CD | Normal | CD | Normal | CD | Normal | CD | Normal | CD | Normal | |
| DG-Mesh [ 21 ] | 0.045 | 33.78 | 0.066 | 30.42 | 0.036 | 39.12 | 0.027 | 30.67 | 0.032 | 29.71 | 0.041 | 32.74 |
| Single-Frame | 0.051 | 30.48 | 0.030 | 27.92 | 0.036 | 31.07 | 0.021 | 32.75 | 0.028 | 33.71 | 0.033 | 31.19 |
| w/o Motion-Aware Appearance | 0.013 | 12.79 | 0.042 | 31.75 | 0.035 | 26.41 | 0.038 | 36.24 | 0.014 | 18.75 | 0.028 | 25.19 |
| w/o Alternating Estimation | 0.012 | 13.90 | 0.028 | 19.77 | 0.024 | 18.99 | 0.024 | 28.45 | 0.016 | 22.32 | 0.021 | 20.69 |
| w/o Specular | 0.012 | 14.83 | 0.034 | 26.99 | 0.023 | 20.98 | 0.021 | 28.93 | 0.013 | 19.89 | 0.021 | 22.32 |
| Method | HO3D | HODome | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| CD | PSNR | L1 | SSIM | LPIPS | CD | PSNR | L1 | SSIM | LPIPS | |
| Single-Frame | 0.011 | 27.60 | 0.01017 | 0.9565 | 0.0412 | 0.024 | 39.00 | 0.00101 | 0.9946 | 0.0062 |
| w/o Motion-Aware Appearance | 0.004 | 27.09 | 0.01344 | 0.9580 | 0.0345 | 0.017 | 38.04 | 0.00127 | 0.9951 | 0.0057 |
| Ours | 0.004 | 28.11 | 0.01017 | 0.9625 | 0.0296 | 0.016 | 39.71 | 0.00099 | 0.9957 | 0.0056 |
3D Reconstruction
3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.
See all 955 papers