Hand in Silhouette: Synthetic-to-Real Multi-View Hand Reconstruction via Binary Masks
Organizations: Future Lab, HEC Group
Abstract
Calibrated multi-view hand reconstruction has advanced rapidly, yet the effectiveness of existing methods depends strongly on real captured data, which is costly to acquire, limited in hand-geometry diversity, and often affected by imperfect annotations. Synthetic data offers a scalable alternative with accurate supervision, but a substantial synthetic-to-real appearance gap hinders its use. To address this, we introduce Hand in Silhouette (SilHand), a calibrated multi-view hand reconstruction framework that operates on binary hand masks, obtainable at low cost either automatically or manually, to reduce the synthetic-to-real observation gap. Rather than relying on RGB features for 3D lifting, SilHand projects metric 3D hypotheses into each view to query and fuse mask-ray evidence for hand reconstruction. This mask-centric formulation enables training on a large-scale synthetic dataset comprising 46.8k grasping poses and 60k free-hand samples, providing broad coverage of hand geometry while directly transferring to real captured hand geometry. Across five real-world benchmarks, SilHand improves relative reconstruction accuracy from 49.2% to 63.4% using masks rendered from reference hands. Moreover, reconstruction accuracy improves consistently as the synthetic training set grows, indicating that broader geometric coverage translates into better real-world performance. While achieving such diversity is difficult with real captured data, it becomes practical under our mask-centric formulation. Finally, SilHand remains robust under mask corruption, with reconstruction error increasing by at most 1.63 mm. Together, these results support binary masks as an effective interface for learning hand reconstruction from synthetic geometry.
Figures & tables
| Dataset | Method | MPVPE | RR-V | PA-V | MPJPE | RR-J | PA-J |
| DexYCB- MV | POEM | 6.13 | 7.21 | 4.00 | 6.06 | 7.30 | 3.93 |
| MLPHand | 6.20 | 7.30 | 4.15 | 6.23 | 7.41 | 4.11 | |
| POEMv2 | 6.50 | 7.50 | 3.84 | 6.33 | 7.54 | 3.68 | |
| JUMP-Hand | 5.45 | 6.41 | 3.77 | 5.31 | 6.41 | 3.61 | |
| Visual Hull | – | – | 12.95 | – | – | 12.42 | |
| Ours | 2.07 | 1.63 | 1.12 | 2.12 | 1.69 | 1.16 |
| Dataset | POEMv2 | GT | Ours |
| ARCTIC-MV | 79.0 | 80.1 | 81.9 |
| InterHand-MV | 81.8 | 83.9 | 83.1 |
| Metric | Synthetic Corruption Preset | SAM3 | |||
| C1 | C2 | C3 | C4 | ||
| iou | 0.97 | 0.93 | 0.89 | 0.86 | 0.90 |
| Boundary F1 | 0.90 | 0.86 | 0.81 | 0.76 | 0.84 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| (m) | Grid spacing (mm) | Vertex coverage (%) | Whole-hand coverage (%) |
| 0.150 | 9.3750 | 88.5755 / 83.4799 | 10.98 / 5.22 |
| 0.200 | 12.5000 | 99.1255 / 98.1773 | 78.78 / 65.72 |
| 0.225 | 14.0625 | 99.8931 / 99.7016 | 94.28 / 89.36 |
| 0.250 | 15.6250 | 99.9962 / 99.9822 | 99.78 / 98.92 |
| 0.275 | 17.1875 | 99.9996 / 99.9986 | 99.98 / 99.94 |
| 0.300 | 18.7500 | 100.0000 / 99.9999 | 100.00 / 99.98 |
| Mask condition | Mean-shape template | Reference template |
| Clean | 1.64 | 1.54 |
| 1.79 | 1.70 | |
| 1.95 | 1.86 | |
| 2.13 | 2.05 | |
| 2.41 | 2.33 |
| Mask condition | Before projection | After projection | |
| Clean | 1.0250 | 1.0251 | +0.000099 |
| 1.0955 | 1.0956 | +0.000064 | |
| 1.1473 | 1.1474 | +0.000093 | |
| 1.2008 | 1.2010 | +0.000127 | |
| 1.2781 | 1.2782 | +0.000155 |
| Component | Configuration |
| Input masks | Resolution of pixels per view. |
| Mask encoder | Four-stage ConvNeXt-V2-style feature pyramid ( Woo et al., 2023 ) with stage widths , depths , and FFN expansion ratio 4. The stem and inter-stage downsampling use strides 4 and 2, respectively. |
| Ray encoder | A convolution with stride 4, followed by a residual block with depthwise and convolutions. |
| Metric grid | grid points discretizing a workspace of extent . |
| Positional encoders | Coordinates concatenated with Fourier features at frequencies , followed by two-layer MLPs of width 64 for cross-view aggregation and 128 for canonical hand state decoding. |
| Cross-view attention | Two attention blocks with feature width 64, 4 heads, and FFN expansion ratio 4. |
| Augmentation | Description | ||
| Hand-region visibility | 0.5 | 1.0 | Occlusion by non-target geometry and partial hand-region removal. |
| Forearm extent | 0.5 | 0.8 | Extending the forearm with length scale . |
| Morphological perturbations | 0.5 | 0.8 | Filling small gaps in the mask using a diamond-shaped kernel with radius pixels. |
| Boundary perturbations | 0.5 | 0.8 | Smoothly displacing the mask boundary with amplitude in pixels. |
| Localized boundary artifacts | 0.5 | 0.8 | Adding square artifacts near the boundary, with side lengths pixels. |
| Random view dropout | 0.5 | 0.2 | Randomly dropping input views while retaining at least three. |
| Variant | Clean | ||||
| method | 2.86 / 1.92 | 3.05 / 2.02 | 3.30 / 2.15 | 3.50 / 2.26 | 3.77 / 2.39 |
| Direct global pose regression | 3.39 / 2.60 | 3.64 / 2.72 | 3.85 / 2.85 | 4.08 / 2.97 | 4.34 / 3.12 |
| Without canonicalization | 3.18 / 2.24 | 3.37 / 2.36 | 3.56 / 2.50 | 3.80 / 2.63 | 4.10 / 2.79 |
| Mean pooling | 3.01 / 2.06 | 3.22 / 2.16 | 3.48 / 2.29 | 3.71 / 2.41 | 4.03 / 2.55 |
| Method | Surface distance | iou |
| method | 3.109 | 83.1 |
| POEMv2 | 3.498 | 80.6 |
| Metric | Nearly identical masks | Distinguishable masks |
| before PA (%) | 29.70 | 7.18 |
| after PA (%) | 25.65 | 5.27 |
| Parameter | ||||
| Forearm length scale | 0.125 | 0.250 | 0.375 | 0.500 |
| Fragment-dropout probability | 0.05 | 0.10 | 0.25 | 0.50 |
| Morphological perturbations (pixels) | 1 | 2 | 3 | 4 |
| Boundary perturbation coefficient | 0 | 1 | ||
| Number of localized boundary artifacts | 1 | 2 | 3 | 4 |
| Dataset | 2 views | 3 views | 4 views | 5 views | 6 views | 7 views | 8 views |
| DexYCB-MV | 34.47 | 8.23 | 4.51 | 2.78 | 2.44 | 2.21 | 2.07 |
| HO3D-MV | 11.23 | 4.44 | 3.49 | 2.79 | – | – | – |
| OakInk-MV | 12.50 | 3.91 | 2.95 | – | – | – | – |
| ARCTIC-MV | 74.94 | 32.13 | 10.00 | 5.81 | 3.54 | 3.14 | 2.99 |
| InterHand-MV | 11.11 | 5.96 | 4.62 | 4.22 | 3.95 | 3.71 | 3.52 |
| Perturbation | L1 | L2 | L3 | L4 |
| Translation (mm) | 0.10 | 0.25 | 0.50 | 1.00 |
| Rotation ( ∘ ) | 0.025 | 0.05 | 0.10 | 0.25 |
| Focal length (%) | 0.10 | 0.25 | 0.50 | 1.00 |
| Principal point (%) | 0.025 | 0.05 | 0.10 | 0.25 |
| Dataset | L0 | L1 | L2 | L3 | L4 |
| DexYCB-MV | 2.07 | 2.09 | 2.05 | 2.31 | 3.62 |
| HO3D-MV | 2.79 | 2.79 | 2.82 | 2.90 | 3.19 |
| OakInk-MV | 2.95 | 3.09 | 3.20 | 3.58 | 5.42 |
| ARCTIC-MV | 2.99 | 3.06 | 3.23 | 4.20 | 8.05 |
| InterHand-MV | 3.52 | 3.55 | 3.62 | 3.69 | 4.54 |