Fast and accurate 3D hand reconstruction is essential for real-time applications in VR/AR, human-computer interaction, robotics, and healthcare. Most state-of-the-art methods rely on heavy models, limiting their use on resource-constrained devices like headsets, smartphones, and embedded systems. In this paper, we investigate how the use of lightweight neural networks, combined with Knowledge Distillation, can accelerate complex 3D hand reconstruction models by making them faster and lighter, while maintaining comparable reconstruction accuracy. While our approach is suited for various hand reconstruction frameworks, we focus primarily on boosting the HaMeR model, currently the leading method in terms of reconstruction accuracy. We replace its original ViT-H backbone with lighter alternatives, including MobileNet, MobileViT, ConvNeXt, and ResNet, and evaluate three knowledge distillation strategies: output-level, feature-level, and a hybrid of both. Our experiments show that using lightweight backbones that are only 35% the size of the original achieves 1.5x faster inference speed while preserving similar performance quality with only a minimal accuracy difference of 0.4mm. More specifically, we show how output-level distillation notably improves student performance, while feature-level distillation proves more effective for higher-capacity students. Overall, the findings pave the way for efficient real-world applications on low-power devices. The code and models are publicly available under https://github.com/hunainahmedj/Fast-HaMeR.
Figures & tables
Figure 1: Real-time monocular 3D hand mesh reconstruction using lightweight student networks. We present a Knowledge Distillation approach for training lightweight networks that accelerate 3D hand reconstruction without compromising quality. Our best-performing network achieves real-time inference with only a modest drop in reconstruction quality. The Figure showcases qualitative results across challenging scenarios, including mutual occlusion, complex hand poses, and interactions with various objects.
Figure 2: High-level overview of the teacher-student architecture, only relevant distillation losses are used depending on the KD level. ϕ(FT) refers to a 1x1 convolution to project the dimensions of the teacher’s features to match those of the student’s for feature-level distillation. The teacher network is used only during training; at inference time, only the trained student network is used.
Figure 3: Trade-off between model accuracy (PA-MPJPE), speed (FPS), and parameter size (circle size). The Figure shows that the most accurate model is HaMeR; however, it shows that other alternatives give close performance with fewer resources and better runtime. ConvNeXt-L with feature distillation falls right behind HaMeR in our experiments with 1.48x FPS boost and 64% reduction in size.
Backbone
Total Params (M)
FPS ↑
GFLOPs
MobileNet-L
42.0
48
0.2
MobileViT-S
42.0
42
0.5
ResNet-50
69.3
50
4.1
ResNet-101
88.3
36
7.8
ConvNeXt-L
240
40
34.4
ViT-H
671
27
167.3
Table 1: The number of networks’ parameters in millions, along with the runtime of different backbone configurations in Frames-per-Second (FPS). All configurations were tested on a consumer-grade RTX 4060 Ti GPU. We also report the number of floating point operations (GFLOPs) as stated in the torchvision library.
Table 2: Performance evaluation after training different backbones without KD to set baselines. The experiments are sorted in ascending order of the total parameter size of the network.
Figure 4: Qualitative results on images from the internet, the scenes represent hands interacting with the environment. We compare our results of the best configuration (ConvNext-L with feature-level distillation) with HaMeR both in 2D and 3D.
Table 3: Output-level Distillation: Evaluating different backbones trained with KD loss applied on the outputs of the student and teacher models. λKD refers to the weight of the KD loss.
Table 4: Feature-level Distillation: Evaluating different backbones trained with KD loss applied on the feature maps produced by the backbones of student and teacher models. λKD refers to the weight of the total KD loss, while γFD refers to a scalar used to scale only the feature-distillation loss.
Table 5: Combined Distillation: Evaluating different backbones trained with KD loss applied on the outputs and the feature maps produced by the backbones of student and teacher models. λKD refers to the weight of the total KD loss, while γFD refers to a scalar used to scale only the feature-distillation loss.
Table 6: Comparison with the state-of-the-art using our best network configuration. The selected model contains a ConvNeXt-L [ Liu et al., 2022 ] as the backbone and was trained using feature-level distillation with λKD=0.8,γFD=12 .
While recent feed-forward 3D reconstruction models accelerate 3D reconstruction by jointly inferring dense geometry and camera poses in a single pass, their reliance on dense attention imposes a quadratic complexity, creating a prohibitive computational bottleneck that severely limits inference speed. To resolve this, we introduce Speed3R, an end-to-end trainable model inspired by the core principle of Structure-from-Motion: that a sparse set of keypoints is sufficient for robust pose estimation. Speed3R features a dual-branch attention mechanism where a compression branch creates a coarse contextual prior to guide a selection branch, which performs fine-grained attention only on the most informative image tokens. This strategy mimics the efficiency of traditional keypoint matching, achieving a remarkable 12.4x inference speedup on 1000-view sequences, while introducing a minimal, controlled trade-off in geometric accuracy. Validated on standard benchmarks with both VGGT and π3 backbones, our method delivers high-quality reconstructions at a fraction of computational cost, paving the way for efficient large-scale scene modeling.
Recovering high-fidelity 3D hand geometry from images is a critical task in computer vision, holding significant value for domains such as robotics, animation and VR/AR. Crucially, scalable applications demand both accuracy and deployment flexibility, requiring the ability to leverage massive amounts of unstructured image data from the internet or enable deployment on consumer-grade RGB cameras without complex calibration. However, current methods face a dilemma. While single-view approaches are easy to deploy, they suffer from depth ambiguity and occlusion. Conversely, multi-view systems resolve these uncertainties but typically demand fixed, calibrated setups, limiting their real-world utility. To bridge this gap, we draw inspiration from 3D foundation models that learn explicit geometry directly from visual data. By reformulating hand reconstruction from arbitrary views as a visual-geometry grounded task, we propose a feed-forward architecture that, for the first time in literature, jointly infers 3D hand meshes and camera poses from uncalibrated views. Extensive evaluations show that our approach outperforms state-of-the-art benchmarks and demonstrates strong generalization to uncalibrated, in-the-wild scenarios. Here is the link of our project page: https://lym29.github.io/HGGT/.
Yumeng Liu, Xiao-Xiao Long, Marc Habermann +5
University of Science and Technology of China · Nanjing University · Max-Planck-Institut für Informatik +3
In this work, we present HandMvNet, one of the first real-time method designed to estimate 3D hand motion and shape from multi-view camera images. Unlike previous monocular approaches, which suffer from scale-depth ambiguities, our method ensures consistent and accurate absolute hand poses and shapes. This is achieved through a multi-view attention-fusion mechanism that effectively integrates features from multiple viewpoints. In contrast to previous multi-view methods, our approach eliminates the need for camera parameters as input to learn 3D geometry. HandMvNet also achieves a substantial reduction in inference time while delivering competitive results compared to the state-of-the-art methods, making it suitable for real-time applications. Evaluated on publicly available datasets, HandMvNet qualitatively and quantitatively outperforms previous methods under identical settings. Code is available at github.com/pyxploiter/handmvnet.
Muhammad Asad Ali, Nadia Robertini, Didier Stricker
Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany