Fast and Accurate Hand Estimation in Pixels, Interaction and World Space

Jinkun Cao

cs.CVNov 5, 2026

Fast and Accurate Hand Estimation in Pixels, Interaction and World Space

Authors: Jinkun Cao

Organizations: This paper is an independent personal project and not related to Jinkun Cao’s position at Amazon FAR.

Abstract

Monocular hand mesh estimators commonly predict rig parameters and project the mesh to obtain 2D keypoints. We find that directly predicted 3D joints agree better with 2D predictions, and fitting a mesh to predicted points reduces mesh error. We present FAH (Fast and Accurate Hand Estimation), a family of models built on SAM 3D Body's hand branch through fine-tuning and feature distillation. The models predict corresponding 2D and 3D joints and surface points, fit MANO or MHR-Hand, and optionally estimate anatomical parts and contact without object geometry at inference. Causal refinement reduces acceleration error, and camera estimation places the hands in world space. Experiments show improvements in image-space alignment, hand geometry, contact estimation and world-space tracking over the evaluated baselines. The default ViT-L model runs at 65 fps with mesh fitting on an RTX 4070 Ti SUPER, or 49 fps with parts and contact enabled. The smallest ViT-S model reaches 125 fps without mesh fitting, parts or contact. Code: https://github.com/noahcao/FAH.

Figures & tables

Figure 1: Five outputs of FAH-H (FAH with ViT-H backbone) on ten scenes. Left to right: 2D keypoints, visible dense points at predicted 2D locations, a MANO mesh fitted to the 3D points, optional part labels within the detector mask, and contact on the fitted mesh (gray to violet, probability 0–1; contact-side view). Outlines: right hand blue, left orange. Parts: palm gray; thumb to pinky red, orange, green, blue and magenta; darker shades are more distal. Image credits: Appendix D .

Panel of Figure 2 — Figure 2: FAH on a training-disjoint HOT3D frame. The detector supplies boxes, handedness, masks and crops. The heatmap head predicts 2D and root-relative 3D joints and surface points (inset). Dashed arrows: MHR-Hand and MANO fitting; inset: side view, MANO outlined. The part head labels mask pixels; contact is shown on fitted MANO (gray to violet, probability 0–1; inset: palm side). Bottom: causal joint refinement, frames four apart with thumb/index fingertip trails, followed by world-space placement (camera gray, wrists colored). Blue: right hand; orange: left.

per frame
		HOT3D v1							EgoStandard
method	fps	PA	PA-p	PVE	root	MRRPE	accel	jitter	PA	PA-p	PVE	root	accel	jitter
MediaPipe Hands [ 46 ]	—	19.61	67.01	—	87.72	119.79	20.59	20.61	—	—	—	—	—	—
HaMeR [ 6 ]	4.6	9.46	14.48	9.24	41.62	49.26	6.75	6.84	—	—	—	—	—	—
Hamba [ 8 ]	5.0	9.28	14.31	9.02	40.66	47.83	7.22	7.31	—	—	—	—	—	—
WiLoR [ 7 ]	4.7	6.72	11.85	6.49	43.89	48.57	8.30	8.48	7.26	8.59	7.52	64.18	6.64	6.71

Table 1: 3D accuracy on HOT3D v1 and EgoStandard (mm; accel and jitter in mm/frame 2 ; PA-p: coverage-aware PA-MPJPE [ 25 ] ). fps on HOT3D frames on an NVIDIA RTX 4070 Ti SUPER (16 GB) desktop GPU under Windows 11 + WSL2, slower of two rounds (MediaPipe: GPU delegate unavailable under WSL2 / Windows); § our CUDA-graph implementation; crop-based baselines use WiLoR’s detector. FAH-S, FAH-B, FAH-L and FAH-H (ViT-S, ViT-B, ViT-L, ViT-H+ at 448 px) share data, losses and schedule and differ only in encoder initialization and input size ( Sec. C.1 ); their fps is end to end (detection, pose, mesh fit, fingertips; optional parts excluded), all four timed in one session. Evaluation scope: Appendices B , A.6 and A.11 ; full tables: Appendix C .

		bench2d v2				challenging poses			low light			shadows
method	fps	PCK	incl.	3D fit	overlap	recall	PCK	incl.	recall	PCK	incl.	recall	PCK	incl.
MediaPipe Hands [ 46 ]	—	0.65	0.34	0.55	0.28	0.05	0.94	0.05	0.33	0.65	0.22	0.58	0.89	0.51
RTMPose-m [ 47 ]	89	0.71	0.40	—	0.23	0.07	0.82	0.06	0.35	0.70	0.24	0.45	0.94	0.42
RTMW-l [ 48 ]	32	0.68	0.65	—	0.56	0.13	0.84	0.11	0.85	0.65	0.56	0.98	0.89	0.87
HaMeR [ 6 ]	4.6	0.71	0.66	0.65	0.55	0.84	0.93	0.78	0.82	0.70	0.57	0.95	0.97	0.92
Hamba [ 8 ]	5.0	0.72	0.66	0.65	0.55	0.84	0.93	0.78	0.81	0.70	0.57	0.95	0.97	0.92

Table 2: 2D accuracy on bench2d v2 and the challenging-pose, low-light and shadow sets. [email protected] on matched hands; “incl.”: missed hands count as failures; “3D fit”: 3D joints projected at the best-fitting translation; “overlap”: [email protected] incl. misses on 387 overlapping hands; shadows: the 278 scored hands of shadow_v1. Full metric set: Appendix C ; fps as in Tab. 1 .

			HOT3D v1 (5 s clips)					EgoStandard (30 s clips)
			chunks		whole tracks			chunks		whole tracks
hands	camera	causal	W	WA	W	WA	RTE	W	WA	W	WA	RTE	fps
WiLoR [ 7 ]	DROID-SLAM	✗	52.49	20.64	60.05	23.06	10.58	83.31	26.87	267.54	56.99	1.73	2.5
HaWoR [ 11 ]	DROID-SLAM	✗	31.96	13.79	36.45	14.91	7.52	76.44	28.55	210.46	53.62	2.22	4.9
FAH-L video	DROID-SLAM	✗	22.69	8.81	27.17	9.81	3.68	68.20	23.53	260.56	55.02	1.35	4.4
FAH-L world	FAH-camera	✓	26.60	10.87	32.24	12.62	4.73	55.90	21.26	177.65	45.76	1.30	38

Table 3: Hands in world space (W / WA-MPJPE in mm, RTE in %; chunks of 100 frames; mean of 3 runs, SDs in Tab. 21 ; each method’s own true positives). HOT3D scores guarded camera choices; earlier configurations were also scored on EgoStandard test ( Sec. A.11 ). DROID-SLAM: HaWoR’s camera stage (DROID-SLAM masked with HaWoR’s hands, Metric3D scale, no infiller), one trajectory per clip shared by WiLoR, HaWoR and FAH-L video; with FAH-L world, our offline mode: DROID-SLAM masked with our hands, scale fused with FAH-camera’s. FAH-camera: our causal camera model ( Sec. 3.6 ). causal: every stage uses only past and current frames. fps: all stages, desktop RTX 4070 Ti SUPER, 4 EgoStandard test clips, slower of 2 rounds.

	HOT3D			DexYCB s0
method	F1	AP	flag	F1	AP	flag
HACO [ 29 ]	0.58	0.47	0.82	0.66	0.61	0.86
FAH-S	0.77	0.83	0.98	0.75	0.81	0.98
FAH-B	0.79	0.85	0.98	0.76	0.82	0.98
FAH-L	0.80	0.85	0.98	0.76	0.82	0.98
FAH-H	0.80	0.86	0.98	0.77	0.82	0.98

Table 4: Hand contact on held-out HOT3D and DexYCB s0 test. Pooled vertex F1/AP and hand-level F1 (

\uparrow

), with 10 mm contact labels. Ground-truth boxes unless marked † (full pipeline); distance rules use known object geometry. HACO is zero-shot on HOT3D. Protocol and selection details: Appendices B and A.6 .

Figure 3: Contact on held-out HOT3D and DexYCB hands. Each example shows the input, ground truth, HACO [ 29 ] and FAH-L on the same ground-truth MANO mesh. Gray to violet indicates contact probability, with each method’s validation threshold mapped to 0.5; scores are per-hand vertex F1. HACO wins on the fourth example; the last shows a missed fingertip contact. Selection: Sec. C.7 ; credits: Appendix D .

	dense	param.
3D joints vs 2D (%)	1.43	3.07
mesh joints vs 2D (%)	2.28	3.07
mesh PA-MPVPE (mm)	4.09	6.79
mesh vs dense joints (mm)	2.52	4.15
mesh vs dense surface (mm)	1.64	3.51

Table 5: Dense points versus MANO parameters in one network on HOT3D (3D joints 4.41 mm PA-MPJPE): mesh fitted to the dense points (dense) vs the network’s own MANO-parameter head (param.). 2D: median per hand vs the direct 2D joints, % of hand size. Last rows: mesh vs the dense prediction, rigidly aligned (a diagnostic of the fit, not of accuracy).

	256 px	320 px	448 px
ViT-S	5.50/0.80/ 117	5.40/0.80/109	5.35/0.80/91
ViT-B	4.82/0.82/104	4.75/0.82/93	4.70/0.82/73
ViT-L	4.53/ 0.83 /80	4.46/ 0.83 /68	4.41/ 0.83 /47
ViT-H+	—	—	4.34 / 0.83 /28

Table 6: Backbone and input resolution. Controlled 12k-step ablation with one recipe and no external distillation: HOT3D PA-MPJPE (mm) / bench2d v2 [email protected] / desktop fps, without the fingertip step.

ViT-B	HOT3D	DexYCB
control	4.43	5.69
+ dense part terms	4.43	5.68
+ part head, union loss	4.47	5.72
FAH-L, continued	HOT3D	mined
without parts	4.30	0.80
+ joint part head ( $\times$ 0.3)	4.61	0.81

Table 7: Part-supervision ablation. Matched 2k-step joint training: PA-MPJPE (mm) on HOT3D and DexYCB for ViT-B; HOT3D PA-MPJPE and mined-set [email protected] for continued FAH-L training. Union loss combines part-union/mask and dense-part terms;

\times

0.3 scales pose gradients.

	2D [email protected] $\uparrow$			PA-MPJPE $\downarrow$
method	2D	3D proj.	mesh proj.	3D	mesh
FAH, image	0.83	0.79	0.77	4.41	5.14
FAH, video	0.83	0.79	0.77	4.24	5.04

Table 8: Direct versus projected FAH outputs. Internal comparison using a ViT-B development model on matched HOT3D hands, with true intrinsics. [email protected] (

\uparrow

) compares direct 2D, projected camera-frame joints and projected mesh joints; PA-MPJPE (

\downarrow

, mm) compares direct 3D with mesh-regressed MANO joints plus tips.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

source	crops	labels	share (%)
HOT3D train, every 2nd frame ∗ [ 17 ]	150k	2D, 3D, mesh	13.8
COCO-WholeBody [ 49 ] †	58k	2D, mask, KD	10.6
CMU Panoptic hands [ 50 ] †	1.9k	2D, KD	3.2
HInt train [ 6 ] †	12.4k	2D, KD	6.3
Halpe-FullBody hands [ 51 ] †	21.7k	2D, KD	8.5
FreiHAND [ 14 ] †	130k	2D, 3D, mesh	5.3

Appendix

Table 9: Pose training data at the last development milestone: crops, labels and sampling shares. 2D/3D: joints; mesh: MANO vertices; mask: SAM 2.1; KD: teacher targets. Released stages add HaGRID pseudo-labels (4.9%) and 885 hard crops (2.9%; Sec. A.8 ). † Converted to the MANO convention ( Sec. A.4 ); ∗ benchmark sequences excluded.

Panel of Figure 4 — Figure 4: Custom benchmark examples. Images 1–4 show shadows across the hand or beside it; 5–8 show synthetic low light with the original image inset. Ground-truth keypoints are overlaid. Challenging-pose stock photos are omitted because their licenses do not permit redistribution. All shown photos are CC BY 2.0; sources, authors and modifications: Appendix D .

per frame
method	recall $\uparrow$	rr (mm) $\downarrow$
MediaPipe Hands [ 46 ]	0.58	54.43
HaMeR [ 6 ]	0.96	57.03
HaMeR + AnyHand [ 9 ]	0.96	49.56
Hamba [ 8 ]	0.96	56.49
WiLoR [ 7 ]	0.96	17.80

Appendix

Table 10: Additional HOT3D metrics. Detection recall and root-relative MPJPE complement the primary errors in Tab. 1 .

method	tips	HOT3D	CMU	COCO	HInt-ND	HInt-EPIC	recall	overlap recall
MediaPipe Hands [ 46 ]	0.59	0.61	0.73	0.64	0.66	0.68	0.51	0.43
RTMPose-m [ 47 ]	0.67	0.69	0.73	0.75	0.72	0.72	0.56	0.31
RTMW-l [ 48 ]	0.60	0.64	0.66	0.69	0.71	0.72	0.94	0.94
HaMeR [ 6 ]	0.64	0.72	0.73	0.75	0.72	0.66	0.92	0.80
HaMeR + AnyHand [ 9 ]	0.60	0.72	0.67	0.71	0.64	0.58	0.92	0.80
Hamba [ 8 ]	0.64	0.72	0.72	0.75	0.72	0.69	0.92	0.80

Appendix

Table 11: Additional bench2d v2 metrics. We report fingertip and per-dataset [email protected], overall recall and recall on overlapping hands. Aggregate PCK and accuracy including misses are in Tab. 2 .

	challenging poses		low light		shadows
method	[email protected]	tips	[email protected]	tips	[email protected]	tips
MediaPipe Hands [ 46 ]	0.75	0.91	0.34	0.57	0.70	0.88
RTMPose-m [ 47 ]	0.60	0.85	0.40	0.64	0.83	0.92
RTMW-l [ 48 ]	0.72	0.76	0.36	0.57	0.71	0.83
HaMeR [ 6 ]	0.77	0.92	0.36	0.62	0.90	0.95
HaMeR + AnyHand [ 9 ]	0.71	0.87	0.26	0.55	0.82	0.91

Appendix

Table 12: Additional stress-test metrics. [email protected] and fingertip [email protected] (tips) complement the results in Tab. 2 .

	Dex-		hard	EgoDex	Ego-
model	YCB	HOT3D	poses	p. 2	Std.
original mix	5.68	4.41	0.91	6.27	7.08
+ 3rd person	5.61	4.39	0.92	6.99	7.42

Appendix

Table 13: Third-person data against tracking ground truth (ViT-B development setup; PA-MPJPE in mm; hard poses: [email protected]). EgoStandard exposure: Sec. 5 .

		HOT3D	DexYCB		hard	dark
enc.	px	PVE	PA	PVE	PCK	PCK
ViT-S	256	5.09	6.45	6.11	0.98	0.76
	320	4.99	6.42	6.09	0.98	0.76
	448	4.94	6.39	6.06	0.98	0.76
ViT-B	256	4.45	5.84	5.52	0.99	0.79
	320	4.38	5.82	5.51	0.99	0.79

Appendix

Table 14: Additional backbone and resolution metrics. The configurations are those of Tab. 6 . Errors are in mm; challenging-pose and low-light scores use [email protected].

	HOT3D v1				bench2d v2		hard	dark	mined	EgoStd.
milestone	PA	PA-p	PVE	root	PCK	3D fit	PCK	PCK	PCK	PA	PVE	fps
pose network
SAM 3D Body hand, off the shelf	8.30	—	8.49	40.00	0.61	—	—	—	—	—	—	10 §
fine-tuned hand-only teacher (DINOv3-H+)	4.97	—	5.67	11.25	0.81	0.84	—	—	—	—	—	18 †
ViT-B student, MHR decoder and keypoint head	4.85	5.44	5.70	11.59	0.83	0.85	1.00	0.80	—	7.59	8.09	31
decoder-free dense head and root depth	4.79	5.37	5.69	9.81	0.84	0.76	0.99	0.82	—	—	—	54

Appendix

Table 15: Image-model milestones. Each row adds to the preceding configuration ( Appendix A ). Errors are in mm; 2D scores use [email protected]. Top rows use YOLOv8m-pose at 800 px, except mined and EgoStandard evaluations, which use the one-stage detector. Bottom rows use the one-stage detector throughout. Dense-row PVE evaluates the raw surface. Timing uses CUDA graphs; † marks compiled 384 px inference.

detector	fps	HOT3D	2D	hard	dark
WiLoR’s v8m-pose @640	123	0.96	0.92	0.84	0.81
v8m-pose + verification	100	1.00	0.95	0.93	0.89
v8s-pose, low light	158	1.00	0.97	0.93	0.95
v8s-seg	143	1.00	0.97	0.92	0.95
v8m-seg	87	1.00	0.97	0.94	0.95
v8m-seg, mask-bone boxes	93	1.00	0.97	0.95	0.95

Appendix

Table 16: Detector milestones. Recall with an early dense pose model, verification and 800 px input; s/m: small/medium, pose/seg: keypoint/segmentation detector. Mask-bone boxes combine masks and keypoint bones ( Sec. A.7 ). fps measures box-only detection on 1408 2 frames; WiLoR uses 640 px.

	HOT3D v1					EgoStandard				DexYCB
milestone	PA	PA-p	PVE	accel	jitter	PA	PVE	accel	jitter	PA	PVE	accel	fps
per-frame hands	4.40	4.67	5.66	4.13	4.46	7.08	9.08	4.53	4.60	5.72	6.82	6.94	46
+ online refiner	4.25	4.43	5.67	2.37	2.71	7.10	9.08	2.28	2.31	5.58	6.82	5.07	—
+ One Euro	4.25	4.45	5.64	1.73	1.95	7.12	9.08	1.81	1.77	5.59	6.81	4.38	42
+ MANO fit on every frame	4.25	4.46	4.00	1.72	1.95	7.12	7.30	1.81	1.77	5.60	5.39	4.43	43
+ fast-motion, 2D filter	4.23	4.39	3.99	1.75	2.01	7.13	7.30	1.80	1.77	5.53	5.32	4.29	80

Appendix

Table 17: Video-model milestones. Errors are in mm; acceleration and jitter are in mm/frame 2 . Top rows add components in order, using the image model’s hands; MANO fitting begins at the fourth row. Bottom rows replace one design choice at a time. “Blend 1” uses fully refined articulation, and “det. 2nd frame” runs detection on alternate frames. p Results use stored per-frame predictions.

training	PA	PA-p	accel	PA	PA-p	accel	3rd p.
	online			offline
per frame	4.79	5.37	4.23	4.79	5.37	4.23	—
refiner	4.34	4.71	2.11	4.24	4.54	2.47	13–16
+ geometric aug.	4.23	4.60	2.18	4.14	4.45	2.46	13.4
+ rotation aug.	4.35	4.74	2.22	—	—	—	1.7

Appendix

Table 18: Video module design on an early dense ViT-B model (HOT3D v1 PA-MPJPE / PA-MPJPE-p in mm, accel in mm/frame 2 ). 3rd p.: root correction (cm) applied to a downward-pointing hand in a third-person video.

camera (EgoStandard dev)	100-frame chunks	whole tracks
causal camera, 5 scale calls	60.30 / 21.81	189.38 / 53.58
+ GT rotations	64.33 / 22.79	195.52 / 54.85
+ GT metric scale per clip	48.27 / 19.47	121.71 / 46.36
+ GT scale per 81-frame segment	42.87 / 18.60	108.69 / 38.99
GT cameras	32.50 / 13.96	64.64 / 16.37

Appendix

Table 19: Camera error diagnostics (W / WA, mm; EgoStandard dev, seed 0; an earlier FAH-L video model supplies hand masks and scored hands). Each row substitutes a ground-truth component into the reference camera; a diagnostic, not a development progression, so no best marks.

hands	camera	runs	med. / p90	top-10	common
FAH-L	DROID-SLAM	326	900 / 900	0.55	203.00 / 46.13
FAH-L	FAH-camera	326	900 / 900	0.24	141.31 / 38.67
FAH-L	offline	326	900 / 900	0.30	124.32 / 35.36
HaWoR [ 11 ]	DROID-SLAM	677	137 / 900	0.45	210.46 / 53.59
WiLoR [ 7 ]	DROID-SLAM	461	343 / 900	0.45	—

Appendix

Table 20: World-space tracks on EgoStandard. Run lengths are in frames; common-set W/WA errors are in mm. Protocol: Appendix B .

		HOT3D v1					EgoStandard
		chunks		whole			chunks		whole
hands	camera	W	WA	W	WA	RTE	W	WA	W	WA	RTE
WiLoR [ 7 ]	DROID-SLAM	0.19	0.05	0.25	0.07	0.02	4.26	0.23	31.75	1.00	0.06
HaWoR [ 11 ]	DROID-SLAM	0.21	0.05	0.26	0.07	0.02	4.11	0.25	28.90	1.21	0.08
FAH-L video	DROID-SLAM	0.21	0.07	0.41	0.12	0.02	4.79	0.23	41.97	1.08	0.06
FAH-L video	DROID-SLAM, our masks	0.34	0.09	0.40	0.12	0.04	1.66	0.41	18.97	1.42	0.02

Appendix

Table 21: Standard deviations of world-space errors over three runs. Means are in Tab. 3 . Runs vary camera seeds, SLAM realizations or both; the extra mask-control row uses our hand masks in HaWoR’s camera pipeline. Units match the main table.

method	recall	PA	PA-p	PVE	accel	jitter	root
WiLoR † [ 7 ]	1.00	4.87	5.14	4.66	5.63	4.52	52.05
FAH-L §	0.99	5.40	6.49	5.13	6.04	4.95	25.92
FAH-L video §‡	0.99	5.29	5.91	5.10	3.93	2.25	23.87

Appendix

Table 22: DexYCB (s0 test, 160 clips, 9.3k hands; mm, accel and jitter in mm/frame 2 ). † trained on DexYCB; § with DexYCB training crops; ‡ refiner also trained on DexYCB training tracks.

method	rr	MRRPE	recall	prec.
WiLoR [ 7 ]	28.10	44.38	0.99	0.98
FAH-L	17.05	32.97	1.00	0.98
FAH-L video	17.15	33.12	1.00	0.98

Appendix

Table 23: Additional EgoStandard camera-frame metrics. Root-relative MPJPE and relative hand position (mm), plus detection coverage, complement the primary results in Tab. 1 .

model	PA	2D	3D fit	root	MRRPE	PVE	fps
WiLoR	6.72	0.69	0.77	43.89	48.57	6.49	4.7
MHR-ref	4.85	0.83	0.85	11.59	14.65	5.70	31
dense ViT-B	4.38	0.84	0.82	8.67	11.67	5.59	53
last milestone + fit	4.41	0.83	0.82	9.10	12.48	4.08	85

Appendix

Table 24: Points versus parameters. We compare WiLoR’s MANO output, MHR-ref, the dense ViT-B model after reprojection consistency, and the final development model with MANO fitting. HOT3D errors are in mm; bench2d scores are [email protected] for direct 2D and fitted projections. Dense PVE uses the raw surface.

mesh	HOT3D	EgoStandard
WiLoR (MANO)	6.49	7.52
dense surface	5.66	9.08
learned MANO-parameter head	6.79	12.05
closed form alone	6.51	—
closed form + 3 GN steps	4.49	7.42
learned fitter, no input noise	4.03	8.40

Appendix

Table 25: Mesh fitting. PA-MPVPE (mm) on development image predictions. The learned parameter head shares the network of Tab. 5 ; GN: Gauss–Newton; fitter input noise: 3 mm unless noted. Bold row: default fitting stage.

Figure 5: SAM 3D Body hand branch versus FAH-L on bench2d v2 (HOT3D, COCO, HInt; ninth: synthetic low light). Scores: per-hand PCK@0.1, projected HOT3D GT/human labels elsewhere, averaged for two hands. Each stratum shows the first correctly sided hand found by both in fixed random order; outdoor/blur additionally require $\geq$ 0.1 FAH gain. FAH overlays use dense surfaces without fitting; the timed pipeline includes the fit. Failures: Fig. 7 ; sources: Appendix D . — Figure 5: SAM 3D Body hand branch versus FAH-L on bench2d v2 (HOT3D, COCO, HInt; ninth: synthetic low light). Scores: per-hand [email protected], projected HOT3D GT/human labels elsewhere, averaged for two hands. Each stratum shows the first correctly sided hand found by both in fixed random order; outdoor/blur additionally require $\geq$ 0.1 FAH gain. FAH overlays use dense surfaces without fitting; the timed pipeline includes the fit. Failures: Fig. 7 ; sources: Appendix D .

points	PA	PA-p	PVE	PVE-p	F@5	root	2D	3D fit	hard	dark	fps
778	4.40	4.99	5.66	6.21	0.80	8.54	0.84	0.83	0.98	0.82	55
400	4.30	4.88	5.40	5.95	0.78	8.64	0.84	0.83	0.98	0.83	53
200	4.27	4.85	5.48	6.03	0.76	8.80	0.84	0.83	0.98	0.83	56

Appendix

Table 26: Number of dense surface points. All models share data, initialization and an 8k-step label-convention fine-tune. Reduced-point models reconstruct 778 vertices with

K=N/2

spectral modes; the 778-point row is unsmoothed. HOT3D errors are in mm, and 2D scores use [email protected]. Speed differences are below run-to-run variation.

	joints	mesh	HOT3D	length
SAM 3D Body hand	$-$ 0.18	$-$ 0.14	$-$ 0.19	0.81
FAH-L, no tip step	$-$ 0.06	$+$ 0.01	0.00	0.99
FAH-L	$-$ 0.05	$+$ 0.02	$+$ 0.02	1.00
WiLoR [ 7 ]	$+$ 0.05	$+$ 0.12	$+$ 0.47	0.97

Appendix

Table 27: Fingertip extent on bench2d v2 matched hands. Signed tip error is measured along the labeled distal segment in segment lengths; negative values indicate shortfall. Mesh extent uses the furthest tip-region vertex. Length is the median mesh-to-labeled finger-length ratio. Ideal values are zero error and unit length.

method	h+a	h	fps
	VISOR
FAH-L	0.54	0.78	54
FAH detector	0.53	0.77	253
SAM 2.1-B [ 12 ] , boxes	0.55	0.77	23
SAM 2.1-L, boxes	0.56	0.78	15
SAM 3 [ 13 ] , boxes	0.56	0.81	8.4

Appendix

Table 28: VISOR validation segmentation. We report mIoU for hand-and-arm (h+a) and wrist-cut hand (h) masks, and inference speed. o Ground-truth box oracle. VISOR-HOS is trained on VISOR.

model	hand	hand	F1	AP	hand	F1	AP	hand	F1
	HOT3D	DexYCB	HOI4D (no Chair)			ARCTIC val			MOW
HACO [ 29 ]	0.60	0.66	—	—	—	—	—	—	0.52
FAH-S	0.74	0.69	0.62	0.61	0.54	0.74	0.79	0.69	0.58
FAH-B	0.76	0.71	0.63	0.62	0.56	0.76	0.80	0.70	0.57
FAH-L	0.76	0.71	0.64	0.64	0.56	0.76	0.81	0.72	0.56
FAH-H	0.77	0.72	0.63	0.62	0.55	0.77	0.81	0.72	0.53

Appendix

Table 29: Additional contact metrics. HOT3D and DexYCB use mean per-hand vertex F1; HOI4D and ARCTIC also report pooled F1/AP, and MOW uses per-image F1. Primary HOT3D/DexYCB results are in Tab. 4 . HACO is excluded from best-value marks.

Figure 6: Selected FAH-L contact examples. Panels 1–5 and 10–12 are from training-disjoint HOT3D; 6–9 are from COCO. Each shows fitted-mesh contact, a palm view and ground truth where available. Gray to violet indicates increasing probability, with a calibrated threshold of 0.5. Failures include false contact above a vase (10), missed occluded fingertip contact (11) and missed stick contact without a ground-truth object pose (12). Credits are in Appendix D .

data	F1	AP	F1	AP	F1	F1	AP	F1	AP
	HOT3D		DexYCB		MOW	HOI4D		ARCTIC
original	0.80	0.85	0.76	0.81	0.51	0.52	0.52	0.68	0.70
+ ARCTIC	0.80	0.85	0.76	0.82	0.53	0.54	0.53	0.76	0.81
+ ARCTIC, HOI4D	0.80	0.85	0.76	0.82	0.54	0.64	0.63	0.76	0.81

Appendix

Table 30: Contact data ablation. Two-seed means for FAH-L; vertex F1/AP and MOW per-image F1 (HACO protocol). All recipes include HOT3D, DexYCB and MOW. Added ARCTIC/HOI4D data make those evaluations in-domain; HOI4D excludes Chair and holds out the test subject.

Figure 7: FAH-H failures , with the five outputs of Fig. 1 . Piano: flattened left mesh/missed key contact; guitar: tangled fretting-hand joints; soldering: coarse meshes; phone: parts spill onto the screen and label occluded fingers; dough: missed right-hand contact; knife: coarse mesh. Sources: Wikimedia Commons and, bottom row, EPIC-KITCHENS-100 via HInt; public test splits, full credits in Appendix D .

Hand-Object Contact Estimation

See all 8 papers

HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion
Aug 20, 2026Muhammad Asad Ali, Nadia Robertini, Didier StrickerCross-View Attention 3D Hand Pose Estimation
Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany
REACH: Hand Pose Estimation from Room Corners
May 21, 2026Shu Nakamura, Ryo Kawahara, Genki Kinoshita +43D Hand Pose Estimation 3D Hand Reconstruction
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
Sep 21, 2026Kaiwen Ren, Yiran Jiang, Yongjing Ye +13D Hand Pose Estimation
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China

Aug 20, 2026J/K move · Enter open · S save

HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion

Muhammad Asad Ali, Nadia Robertini, Didier Stricker

Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany

Cross-View Attention 3D Hand Pose Estimation 3D Hand Reconstruction

Open paper AskPDF arXiv

In this work, we present HandMvNet, one of the first real-time method designed to estimate 3D hand motion and shape from multi-view camera images. Unlike previous monocular approaches, which suffer from scale-depth ambiguities, our method ensures consistent and accurate absolute hand poses and shapes. This is achieved through a multi-view attention-fusion mechanism that effectively integrates features from multiple viewpoints. In contrast to previous multi-view methods, our approach eliminates the need for camera parameters as input to learn 3D geometry. HandMvNet also achieves a substantial reduction in inference time while delivering competitive results compared to the state-of-the-art methods, making it suitable for real-time applications. Evaluated on publicly available datasets, HandMvNet qualitatively and quantitatively outperforms previous methods under identical settings. Code is available at github.com/pyxploiter/handmvnet.

Abstract

Hand-Object Contact Estimation

Explore similar work

HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion

REACH: Hand Pose Estimation from Room Corners

Estimating Accurate Hand Pose in Camera Space with Vision Transformer