HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion

Muhammad Asad Ali; Nadia Robertini; Didier Stricker

cs.CVAug 20, 2026

HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion

Authors: Muhammad Asad Ali, Nadia Robertini, Didier Stricker

Organizations: Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany

Abstract

In this work, we present HandMvNet, one of the first real-time method designed to estimate 3D hand motion and shape from multi-view camera images. Unlike previous monocular approaches, which suffer from scale-depth ambiguities, our method ensures consistent and accurate absolute hand poses and shapes. This is achieved through a multi-view attention-fusion mechanism that effectively integrates features from multiple viewpoints. In contrast to previous multi-view methods, our approach eliminates the need for camera parameters as input to learn 3D geometry. HandMvNet also achieves a substantial reduction in inference time while delivering competitive results compared to the state-of-the-art methods, making it suitable for real-time applications. Evaluated on publicly available datasets, HandMvNet qualitatively and quantitatively outperforms previous methods under identical settings. Code is available at github.com/pyxploiter/handmvnet.

Figures & tables

Figure 1: Comparison of error vs. inference speed across different methods. Our approach outperforms other methods in both inference speed and accuracy.

Figure 2: HandMvNet’ architecture consists of three stages: (a) Sampling joint-aligned features using predicted 2D joints from each image (b) Fusing multi-view sampled features, (c) Regressing 3D hand joints and vertices.

Panel of Figure 3 — Figure 3: Modules of HandMvNet’s architecture: (a) Point Feature Sampler. (b) Multi-view Feature Fusion. (c) Attention Module. (d) Joint & Mesh Decoder

	#	Methods	MPJPE ${}_{rel}\downarrow$	PA ${}_{J}\downarrow$	AUC ${}_{J@20}\uparrow$	MPVPE ${}_{rel}\downarrow$	PA ${}_{V}\downarrow$	AUC ${}_{V@20}\uparrow$
DexYCB-MV	1	MvP	9.47	4.26	0.69	12.18	8.14	0.53
	2	PE-Mesh-TR	8.87	4.76	0.64	8.67	4.70	0.64
	3	FTL-Mesh-TR	9.81	5.51	0.59	9.80	5.75	0.59
	4	POEM	7.30	3.93	0.68	7.21	4.00	0.70
	5	Multi-view Fit.	8.77	5.19	0.65	8.71	5.29	0.65
	6	HandMvNet (ours)	6.73	4.08	0.67	7.19	4.52	0.65

Table 1: Quantitative results (mm) on the test sets of DexYCB-MV, HO3D-MV, and MVHand. denotes the methods that require camera parameters. The best and second-best results are highlighted in bold and underlined respectively.

Table 2: Ablation Studies.

Panel of Figure 4 — Figure 4: Qualitative results on the test set of DexYCB-MV dataset.

Panel of Figure 5 — Figure 5: Qualitative results on the test set of HO3D-MV dataset.

Panel of Figure 6 — Figure 6: Qualitative results on the test set of MVHand dataset.

Figure 7: Inference Speed (FPS) comparison across methods with different camera views. HandMvNet achieves the highest FPS across all configurations.

Cross-View Attention

See all 9 papers

REACH: Hand Pose Estimation from Room Corners
May 21, 2026Shu Nakamura, Ryo Kawahara, Genki Kinoshita +43D Hand Pose Estimation 3D Hand Reconstruction
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
Sep 21, 2026Kaiwen Ren, Yiran Jiang, Yongjing Ye +13D Hand Pose Estimation
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China
EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
May 12, 2026Christen Millerdurai, Shaoxiang Wang, Yaxu Xie +33D Hand Pose Estimation Monocular 3D Reconstruction
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, Germany · Max Planck Institute for Informatics (MPII), Saarbrücken, Germany

May 21, 2026J/K move · Enter open · S save

REACH: Hand Pose Estimation from Room Corners

Shu Nakamura, Ryo Kawahara, Genki Kinoshita, Ryosuke Hirai+3

Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology

3D Hand Pose Estimation 3D Hand Reconstruction

Open paper AskPDF arXiv

We introduce a novel 3D hand pose estimator that can accurately recover the shape and pose of people's hands in a room from afar, typically from fixed cameras at room corners, in extremely low-resolution and frequently occluded views. Our key idea is to fully leverage hand-body coordination, its temporal progression, and multiview observations. We achieve this with a novel Transformer-based model, in which hand and body configurations are modeled through correlations between their visual features expressed as per-view tokens, and their temporal coordination is exploited in an autoregressive manner. We introduce a novel dataset, which we refer to as REACH, Room-Environment dataset Annotated with Chest cameras for Hand pose estimation, to train and test our method. REACH is a first-of-its-kind large-scale hand pose dataset that captures accurate hand movements of 50 participants across a wide variety of daily activities. In order to avoid interfering with natural movements while annotating the hands with accurate shape and pose, we leverage concealed chest cameras. Through extensive experiments, including comparative studies with existing methods, we show that our model, REACH-Net, achieves highly accurate 3D hand pose estimation from afar. These results broaden the horizon of 3D hand pose estimation, especially towards "in-the-wild" continuous human behavior analysis.

Abstract

Cross-View Attention

Explore similar work

REACH: Hand Pose Estimation from Room Corners

Estimating Accurate Hand Pose in Camera Space with Vision Transformer

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera