cs.CVMar 30, 2026
SaveSHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild
Organizations: Meta Reality Labs · Yale University
Abstract
Accurate 3D understanding of human hands and objects during manipulation remains a significant challenge for egocentric computer vision. Existing hand-object interaction datasets are predominantly captured in controlled studio settings, which limits both environmental diversity and the ability of models trained on such data to generalize to real-world scenarios. To address this challenge, we introduce a novel marker-less multi-camera system that allows for nearly unconstrained mobility in genuinely in-the-wild conditions, while still having the ability to generate precise 3D annotations of hands and objects. The capture system consists of a lightweight, back-mounted, multi-camera rig that is synchronized and calibrated with a user-worn VR headset. For 3D ground-truth annotation of hands and objects, we develop an ego-exo tracking pipeline and rigorously evaluate its quality. Finally, we present SHOW3D, the first large-scale dataset with 3D annotations that show hands interacting with objects in diverse real-world environments, including outdoor settings. Our approach significantly reduces the fundamental trade-off between environmental realism and accuracy of 3D annotations, which we validate with experiments on several downstream tasks. show3d-dataset.github.io
Figures & tables
Figure 2: Our mobile multi-camera capture rig. Left: Annotated hardware layout showing five MoCap cameras (red), and eight exocentric monochrome cameras mounted in a half-dome configuration (green), and two egocentric monochrome cameras on the Meta Quest 3 headset (blue). The MoCap cameras are only used for headset pose tracking, by tracking optical markers rigidly attached to the headset. The ten total exocentric and egocentric monochrome cameras are used for marker-free annotation of 3D hand and object poses. Right: The rig in use during in-the-wild capture sessions, demonstrating its lightweight (about eight kilograms) and wearable design that allows natural interaction while maintaining stable, synchronized multi-view coverage under mostly unconstrained motion.
Figure 3: Our ego-exo pipeline for 3D hand and object pose annotation. (a) Multi-view fisheye images from our ego and exo cameras. (b) We detect 3D hand keypoints by fusing predictions from Sapiens [ 14 ] and InterNet [ 22 ] , and fit personalized hand mesh via Inverse Kinematics. (c) CAD-based 3D object pose estimation using CNOS [ 25 ] , FoundPose [ 28 ] , and GoTrack [ 24 ] . (d) The resulting 3D ground-truth annotations are projected back into ego cameras, and can be used to train egocentric vision models.
| Dataset | Ego / Exo | Frames | Hand Pose | Object Pose | Objects | Subjects | Environment | Annotation |
| SHOW3D (ours) | 2 / 8 | 4.3M | Both / Mesh | ✓ | 21 | 38 | In-the-wild | Monochrome + optimization |
| GigaHands [ 8 ] | 0 / 51 | 3.7M | Both / Mesh | ✓ | 417 | 51 | Studio | RGB + optimization |
| ARCTIC [ 7 ] | 1 / 8 | 2.1M | Both / Mesh | ✓ | 11 | 10 | Studio | MoCap |
| HOT3D [ 3 ] | 3 / 0 | 1.7M | Both / Mesh | ✓ | 33 | 19 | Studio | MoCap |
| TACO [ 18 ] | 1 / 12 | 363K | Both / Mesh | ✓ | 196 | 14 | Studio | MoCap |
| OakInk2 [ 39 ] | 1 / 3 | 993K | Both / Mesh | ✓ | 75 | 9 | Studio | MoCap |
Table 1: Comparison of SHOW3D against recent hand–object interaction datasets.
Figure 4: Cross-dataset feature embedding. We plot UMAP [ 21 ] embeddings of DINOv2 [ 27 ] features extracted from the raw images across different hand–object interaction datasets. SHOW3D (pink) spans diverse visual domains between datasets collected in controlled environments: GigaHands (blue), HOT3D (green), ARCTIC (yellow). Best viewed in color.
| Train set | Test set | ADE (mm) | ACC (m/s 2 ) |
|---|---|---|---|
| SHOW3D | HOT3D | 14.70 | 4.05 |
| HOT3D | HOT3D | 11.29 | 3.21 |
| HOT3D + SHOW3D | HOT3D | 8.80 | 2.16 |
| HOT3D | SHOW3D | 22.57 | 5.61 |
| SHOW3D | SHOW3D | 13.82 | 3.79 |
| SHOW3D + HOT3D | SHOW3D | 13.50 | 3.84 |
Table 2: Evaluation of interaction field estimation.
| Frames ahead | aria | bowl | cansoup | dinotoy | dumbbell | juice | keyboard | milk | mouse | mug | mustard | vase | waffles | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 30 | w/o text | 58.5 | 43.1 | 28.3 | 45.7 | 89.7 | 30.3 | 51.0 | 31.9 | 26.6 | 32.6 | 57.5 | 24.2 | 36.1 | 42.7 |
| w/ text | 49.0 | 37.5 | 25.0 | 34.0 | 73.8 | 19.3 | 37.3 | 19.5 | 14.5 | 21.6 | 16.3 | 20.3 | 27.0 | 30.4 | |
| 60 | w/o text | 63.6 | 40.5 | 29.0 | 46.4 | 95.9 | 33.1 | 66.5 | 22.6 | 27.5 | 37.4 | 77.7 | 27.3 | 39.9 | 46.7 |
| w/ text | 57.0 | 32.7 | 28.4 | 37.5 | 89.0 | 21.1 | 34.6 | 25.2 | 24.7 | 24.4 | 19.0 | 26.3 | 35.1 | 35.0 |
Table 3: Evaluation of text-driven 6DoF object pose forecasting. Reported are per-object and mean average translation errors (mm, ), with and without text conditioning. Conditioning on text leads to a consistent improvement across objects.
| # | Training set | Test set | MKPE (mm) |
| 1 | UmeTrack | SHOW3D | 22.2 (+55%) |
| 2 | HOT3D | SHOW3D | 19.6 (+37%) |
| 3 | UmeTrack + HOT3D | SHOW3D | 16.4 (+15%) |
| 4 | SHOW3D | SHOW3D | 15.5 (+8%) |
| 5 | UmeTrack + HOT3D + SHOW3D | SHOW3D | 14.3 |
| 6 | HOT3D | HOT3D | 14.0 (+14%) |
Table 4: Evaluation of 3D hand pose estimation. The reported metric is Mean Keypoint Position Error (MKPE, in mm). Models trained on existing datasets, UmeTrack [ 11 ] and HOT3D [ 3 ] , generalize noticeably worse to SHOW3D, highlighting the increased difficulty of in-the-wild data.
Figure 5: Hand pose estimation on the SHOW3D test set. From left to right: results from models trained on UmeTrack + HOT3D + SHOW3D, HOT3D, and UmeTrack, respectively. Including training data from SHOW3D significantly improves robustness against object occlusion and background clutter.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Stationary configuration for inclusive capture. The rig can be mounted statically to allow for seated data collection. This setup eliminates the weight burden of the backpack, accommodating participants with limited mobility or physical endurance.
Figure 8: SHOW3D dataset statistics. Left: participant distribution. Right: number of recordings per each of the 21 objects.
Figure 9: Examples of contact annotations. Egocentric frames from SHOW3D with annotated contact regions (yellow). For visual clarity, contact annotations are only shown between the left hand and the manipulated object.
Figure 10: Simultaneous rig–dome capture for ground truth validation. To evaluate the accuracy of our mobile annotation pipeline, we captured sequences where the subject wears our mobile rig inside a high-fidelity stationary multi-camera dome. This allows for a direct comparison between our mobile-rig-derived annotations and the dome-derived “gold standard.”
| MPJPE (mm) | |||
| Reference | Category | Median | P90 |
| Dome | No interactions | 6.28 | 7.31 |
| Hand–hand interaction | 7.46 | 8.95 | |
| Hand–object interaction | 7.90 | 12.53 | |
| Manual (in the wild) | No interactions | 5.65 | 12.35 |
| Hand–hand interaction | 6.03 | 14.63 | |
Table 5: Quantitative evaluation of our ground-truth hand annotations against two references: a high-coverage 30-camera dome ( Dome ) and manual annotations ( Manual ). We report median and 90th percentile per-frame MPJPE (mm).
Figure 11: Examples of manually annotated hand keypoints. Orange: left hand, blue: right hand. Note that the annotators are instructed to annotate only the keypoints that they have high confidence for. During evaluation, we only evaluate against annotated keypoints that have valid 3D triangulation.
Figure 12: Our automated pipeline for 3D hand keypoint annotation. (a) Sapiens keypoints: we use the triangulated hand and forearm keypoints to define crop regions for hands. (b) Crop regions are used to generate 256x256 perspective crops [ 11 ] for both hands. (c) We apply InterNet [ 22 ] on the perspective crops, and fuse them with the initial Sapiens detections to obtain final 3D hand keypoints.
| Object | Yield (%) | P50 TE (mm) | P50 RE (deg) |
|---|---|---|---|
| birdhouse_toy | 89.5 | 3.51 | 1.73 |
| carton_oj | 92.6 | 3.20 | 1.23 |
| dumbbell_5lb | 86.1 | 1.53 | 1.63 |
| aria_small | 77.3 | 2.79 | 1.81 |
| bottle_bbq | 53.5 | 2.60 | 5.77 |
Table 6: Quantitative evaluation of our ground-truth object annotations against MoCap-derived poses as an independent reference.
| Frames ahead | pouring in | picking up | pouring out | opening | stirring | placing down | raising | shaking | tapping | washing | checking | inspecting | moving | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 30 | w/o text | 45.8 | 35.7 | 56.1 | 30.4 | 49.9 | 38.2 | 31.0 | 33.8 | 29.3 | 64.5 | 46.8 | 61.9 | 56.2 | 44.6 |
| w/ text | 28.3 | 23.6 | 37.3 | 20.5 | 35.0 | 29.4 | 24.5 | 27.5 | 26.1 | 59.7 | 44.9 | 61.2 | 59.1 | 36.7 | |
| Improv. | 38.2% | 33.9% | 33.5% | 32.6% | 29.9% | 23.0% | 21.0% | 18.6% | 10.9% | 7.4% | 4.1% | 1.1% | -5.2% | 17.7% | |
| 60 | w/o text | 53.1 | 41.8 | 60.6 | 34.4 | 48.4 | 40.1 | 37.8 | 40.9 | 29.9 | 71.6 | 55.7 | 63.8 | 69.7 | 49.8 |
| w/ text | 35.4 | 26.7 | 42.8 | 22.6 | 40.2 | 31.9 | 25.7 | 30.7 | 32.2 | 73.6 | 51.2 | 63.9 | 68.6 | 42.0 | |
| Improv. | 33.3% | 36.1% | 29.4% | 34.3% | 16.9% | 20.4% | 32.0% | 24.9% | -7.7% | -2.8% | 8.1% | -0.2% | 1.6% | 15.8% |
Table 7: Per-verb evaluation of text-driven 6DoF object pose forecasting, for a selected set of verbs. Reported are translation errors (mm, ) and the relative percentage improvement provided by text conditioning. The results are sorted by improvement at 30 frames. Our method shows robust gains on predictable actions ( pouring , opening ), but has limited impact on ambiguous ones ( moving , inspecting ).
Figure 13: Additional ground truth examples from SHOW3D: hand pose (red and blue), object pose (green), and text captions.
Figure 14: Hand annotation failure cases. We visualize two common error modes in our automated annotation pipeline. Top row: Examples of inaccurate 3D pose estimation . While all entities are successfully detected, minor errors in the estimated 6DoF pose result in physically implausible interpenetrations (clipping) between the hand and object meshes. Bottom row: Examples of missed hand detections . In scenarios with severe occlusion—where a hand is significantly obstructed by the object or the opposing hand—our pipeline occasionally fails to detect the occluded hand entirely.
Figure 15: Object annotation failure case: Missed object under severe occlusion and outdoor illumination. Left: The raw image from egocentric view. Right: The projected 3D hand meshes (red/blue) overlaying the image. Although the hands are correctly tracked, the object is not detected due to occlusion from both hands and strong outdoor lighting, and thus is entirely not tracked.
Figure 16: Object annotation failure case: Rotational ambiguity in symmetric objects. Left: The raw image from egocentric view. Right: The projected 3D object mesh (green) overlaying the image. Note that the pipeline correctly tracks the 3D position but fails to resolve the azimuthal rotation, incorrectly predicting the handle to be visible (“hallucinated”) despite not being present in the image.