Accurate 3D understanding of human hands and objects during manipulation remains a significant challenge for egocentric computer vision. Existing hand-object interaction datasets are predominantly captured in controlled studio settings, which limits both environmental diversity and the ability of models trained on such data to generalize to real-world scenarios. To address this challenge, we introduce a novel marker-less multi-camera system that allows for nearly unconstrained mobility in genuinely in-the-wild conditions, while still having the ability to generate precise 3D annotations of hands and objects. The capture system consists of a lightweight, back-mounted, multi-camera rig that is synchronized and calibrated with a user-worn VR headset. For 3D ground-truth annotation of hands and objects, we develop an ego-exo tracking pipeline and rigorously evaluate its quality. Finally, we present SHOW3D, the first large-scale dataset with 3D annotations that show hands interacting with objects in diverse real-world environments, including outdoor settings. Our approach significantly reduces the fundamental trade-off between environmental realism and accuracy of 3D annotations, which we validate with experiments on several downstream tasks. show3d-dataset.github.io
Figures & tables
Figure 2: Our mobile multi-camera capture rig. Left: Annotated hardware layout showing five MoCap cameras (red), and eight exocentric monochrome cameras mounted in a half-dome configuration (green), and two egocentric monochrome cameras on the Meta Quest 3 headset (blue). The MoCap cameras are only used for headset pose tracking, by tracking optical markers rigidly attached to the headset. The ten total exocentric and egocentric monochrome cameras are used for marker-free annotation of 3D hand and object poses. Right: The rig in use during in-the-wild capture sessions, demonstrating its lightweight (about eight kilograms) and wearable design that allows natural interaction while maintaining stable, synchronized multi-view coverage under mostly unconstrained motion.
Figure 3: Our ego-exo pipeline for 3D hand and object pose annotation. (a) Multi-view fisheye images from our ego and exo cameras. (b) We detect 3D hand keypoints by fusing predictions from Sapiens [ 14 ] and InterNet [ 22 ] , and fit personalized hand mesh via Inverse Kinematics. (c) CAD-based 3D object pose estimation using CNOS [ 25 ] , FoundPose [ 28 ] , and GoTrack [ 24 ] . (d) The resulting 3D ground-truth annotations are projected back into ego cameras, and can be used to train egocentric vision models.
Dataset
Ego / Exo
Frames
Hand Pose
Object Pose
Objects
Subjects
Environment
Annotation
SHOW3D (ours)
2 / 8
4.3M
Both / Mesh
✓
21
38
In-the-wild
Monochrome + optimization
GigaHands [ 8 ]
0 / 51
3.7M
Both / Mesh
✓
417
51
Studio
RGB + optimization
ARCTIC [ 7 ]
1 / 8
2.1M
Both / Mesh
✓
11
10
Studio
MoCap
HOT3D [ 3 ]
3 / 0
1.7M
Both / Mesh
✓
33
19
Studio
MoCap
TACO [ 18 ]
1 / 12
363K
Both / Mesh
✓
196
14
Studio
MoCap
OakInk2 [ 39 ]
1 / 3
993K
Both / Mesh
✓
75
9
Studio
MoCap
Table 1: Comparison of SHOW3D against recent hand–object interaction datasets.
Figure 4: Cross-dataset feature embedding. We plot UMAP [ 21 ] embeddings of DINOv2 [ 27 ] features extracted from the raw images across different hand–object interaction datasets. SHOW3D (pink) spans diverse visual domains between datasets collected in controlled environments: GigaHands (blue), HOT3D (green), ARCTIC (yellow). Best viewed in color.
Train set
Test set
ADE (mm) ↓
ACC (m/s 2 ) ↓
SHOW3D
HOT3D
14.70
4.05
HOT3D
HOT3D
11.29
3.21
HOT3D + SHOW3D
HOT3D
8.80
2.16
HOT3D
SHOW3D
22.57
5.61
SHOW3D
SHOW3D
13.82
3.79
SHOW3D + HOT3D
SHOW3D
13.50
3.84
Table 2: Evaluation of interaction field estimation.
Frames ahead
aria
bowl
cansoup
dinotoy
dumbbell
juice
keyboard
milk
mouse
mug
mustard
vase
waffles
Mean
30
w/o text
58.5
43.1
28.3
45.7
89.7
30.3
51.0
31.9
26.6
32.6
57.5
24.2
36.1
42.7
w/ text
49.0
37.5
25.0
34.0
73.8
19.3
37.3
19.5
14.5
21.6
16.3
20.3
27.0
30.4
60
w/o text
63.6
40.5
29.0
46.4
95.9
33.1
66.5
22.6
27.5
37.4
77.7
27.3
39.9
46.7
w/ text
57.0
32.7
28.4
37.5
89.0
21.1
34.6
25.2
24.7
24.4
19.0
26.3
35.1
35.0
Table 3: Evaluation of text-driven 6DoF object pose forecasting. Reported are per-object and mean average translation errors (mm, ↓ ), with and without text conditioning. Conditioning on text leads to a consistent improvement across objects.
#
Training set
Test set
MKPE (mm) ↓
1
UmeTrack
SHOW3D
22.2 (+55%)
2
HOT3D
SHOW3D
19.6 (+37%)
3
UmeTrack + HOT3D
SHOW3D
16.4 (+15%)
4
SHOW3D
SHOW3D
15.5 (+8%)
5
UmeTrack + HOT3D + SHOW3D
SHOW3D
14.3
6
HOT3D
HOT3D
14.0 (+14%)
Table 4: Evaluation of 3D hand pose estimation. The reported metric is Mean Keypoint Position Error (MKPE, in mm). Models trained on existing datasets, UmeTrack [ 11 ] and HOT3D [ 3 ] , generalize noticeably worse to SHOW3D, highlighting the increased difficulty of in-the-wild data.
Figure 5: Hand pose estimation on the SHOW3D test set. From left to right: results from models trained on UmeTrack + HOT3D + SHOW3D, HOT3D, and UmeTrack, respectively. Including training data from SHOW3D significantly improves robustness against object occlusion and background clutter.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Stationary configuration for inclusive capture. The rig can be mounted statically to allow for seated data collection. This setup eliminates the weight burden of the backpack, accommodating participants with limited mobility or physical endurance.
Figure 8: SHOW3D dataset statistics. Left: participant distribution. Right: number of recordings per each of the 21 objects.
Figure 9: Examples of contact annotations. Egocentric frames from SHOW3D with annotated contact regions (yellow). For visual clarity, contact annotations are only shown between the left hand and the manipulated object.
Figure 10: Simultaneous rig–dome capture for ground truth validation. To evaluate the accuracy of our mobile annotation pipeline, we captured sequences where the subject wears our mobile rig inside a high-fidelity stationary multi-camera dome. This allows for a direct comparison between our mobile-rig-derived annotations and the dome-derived “gold standard.”
MPJPE (mm)
Reference
Category
Median ↓
P90 ↓
Dome
No interactions
6.28
7.31
Hand–hand interaction
7.46
8.95
Hand–object interaction
7.90
12.53
Manual (in the wild)
No interactions
5.65
12.35
Hand–hand interaction
6.03
14.63
Appendix
Table 5: Quantitative evaluation of our ground-truth hand annotations against two references: a high-coverage 30-camera dome ( Dome ) and manual annotations ( Manual ). We report median and 90th percentile per-frame MPJPE (mm).
Figure 11: Examples of manually annotated hand keypoints. Orange: left hand, blue: right hand. Note that the annotators are instructed to annotate only the keypoints that they have high confidence for. During evaluation, we only evaluate against annotated keypoints that have valid 3D triangulation.
Figure 12: Our automated pipeline for 3D hand keypoint annotation. (a) Sapiens keypoints: we use the triangulated hand and forearm keypoints to define crop regions for hands. (b) Crop regions are used to generate 256x256 perspective crops [ 11 ] for both hands. (c) We apply InterNet [ 22 ] on the perspective crops, and fuse them with the initial Sapiens detections to obtain final 3D hand keypoints.
Object
Yield (%)
P50 TE (mm) ↓
P50 RE (deg) ↓
birdhouse_toy
89.5
3.51
1.73
carton_oj
92.6
3.20
1.23
dumbbell_5lb
86.1
1.53
1.63
aria_small
77.3
2.79
1.81
bottle_bbq
53.5
2.60
5.77
Appendix
Table 6: Quantitative evaluation of our ground-truth object annotations against MoCap-derived poses as an independent reference.
Frames ahead
pouring in
picking up
pouring out
opening
stirring
placing down
raising
shaking
tapping
washing
checking
inspecting
moving
Mean
30
w/o text
45.8
35.7
56.1
30.4
49.9
38.2
31.0
33.8
29.3
64.5
46.8
61.9
56.2
44.6
w/ text
28.3
23.6
37.3
20.5
35.0
29.4
24.5
27.5
26.1
59.7
44.9
61.2
59.1
36.7
Improv.
38.2%
33.9%
33.5%
32.6%
29.9%
23.0%
21.0%
18.6%
10.9%
7.4%
4.1%
1.1%
-5.2%
17.7%
60
w/o text
53.1
41.8
60.6
34.4
48.4
40.1
37.8
40.9
29.9
71.6
55.7
63.8
69.7
49.8
w/ text
35.4
26.7
42.8
22.6
40.2
31.9
25.7
30.7
32.2
73.6
51.2
63.9
68.6
42.0
Improv.
33.3%
36.1%
29.4%
34.3%
16.9%
20.4%
32.0%
24.9%
-7.7%
-2.8%
8.1%
-0.2%
1.6%
15.8%
Appendix
Table 7: Per-verb evaluation of text-driven 6DoF object pose forecasting, for a selected set of verbs. Reported are translation errors (mm, ↓ ) and the relative percentage improvement provided by text conditioning. The results are sorted by improvement at 30 frames. Our method shows robust gains on predictable actions ( pouring , opening ), but has limited impact on ambiguous ones ( moving , inspecting ).
Figure 13: Additional ground truth examples from SHOW3D: hand pose (red and blue), object pose (green), and text captions.
Figure 14: Hand annotation failure cases. We visualize two common error modes in our automated annotation pipeline. Top row: Examples of inaccurate 3D pose estimation . While all entities are successfully detected, minor errors in the estimated 6DoF pose result in physically implausible interpenetrations (clipping) between the hand and object meshes. Bottom row: Examples of missed hand detections . In scenarios with severe occlusion—where a hand is significantly obstructed by the object or the opposing hand—our pipeline occasionally fails to detect the occluded hand entirely.
Figure 15: Object annotation failure case: Missed object under severe occlusion and outdoor illumination. Left: The raw image from egocentric view. Right: The projected 3D hand meshes (red/blue) overlaying the image. Although the hands are correctly tracked, the object is not detected due to occlusion from both hands and strong outdoor lighting, and thus is entirely not tracked.
Figure 16: Object annotation failure case: Rotational ambiguity in symmetric objects. Left: The raw image from egocentric view. Right: The projected 3D object mesh (green) overlaying the image. Note that the pipeline correctly tracks the 3D position but fails to resolve the azimuthal rotation, incorrectly predicting the handle to be visible (“hallucinated”) despite not being present in the image.
In this work, we present HandMvNet, one of the first real-time method designed to estimate 3D hand motion and shape from multi-view camera images. Unlike previous monocular approaches, which suffer from scale-depth ambiguities, our method ensures consistent and accurate absolute hand poses and shapes. This is achieved through a multi-view attention-fusion mechanism that effectively integrates features from multiple viewpoints. In contrast to previous multi-view methods, our approach eliminates the need for camera parameters as input to learn 3D geometry. HandMvNet also achieves a substantial reduction in inference time while delivering competitive results compared to the state-of-the-art methods, making it suitable for real-time applications. Evaluated on publicly available datasets, HandMvNet qualitatively and quantitatively outperforms previous methods under identical settings. Code is available at github.com/pyxploiter/handmvnet.
Muhammad Asad Ali, Nadia Robertini, Didier Stricker
Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany
Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.
Siddhant Bansal, Zhifan Zhu, Shashank Tripathi +3
University of Bristol, Bristol, United Kingdom · Max Planck Institute for Intelligent Systems, Tübingen, Germany
We introduce a novel 3D hand pose estimator that can accurately recover the shape and pose of people's hands in a room from afar, typically from fixed cameras at room corners, in extremely low-resolution and frequently occluded views. Our key idea is to fully leverage hand-body coordination, its temporal progression, and multiview observations. We achieve this with a novel Transformer-based model, in which hand and body configurations are modeled through correlations between their visual features expressed as per-view tokens, and their temporal coordination is exploited in an autoregressive manner. We introduce a novel dataset, which we refer to as REACH, Room-Environment dataset Annotated with Chest cameras for Hand pose estimation, to train and test our method. REACH is a first-of-its-kind large-scale hand pose dataset that captures accurate hand movements of 50 participants across a wide variety of daily activities. In order to avoid interfering with natural movements while annotating the hands with accurate shape and pose, we leverage concealed chest cameras. Through extensive experiments, including comparative studies with existing methods, we show that our model, REACH-Net, achieves highly accurate 3D hand pose estimation from afar. These results broaden the horizon of 3D hand pose estimation, especially towards "in-the-wild" continuous human behavior analysis.
Shu Nakamura, Ryo Kawahara, Genki Kinoshita +4
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology