Reconstructing high-fidelity 3D hands from egocentric monocular videos remains a challenge due to the limitations in capturing high-resolution geometry, hand-object interactions, and complex objects on hands. Additionally, existing methods often incur high computational costs, making them impractical for real-time applications. In this work, we propose Mesh-inellipse Aligned deformable Surfel Splatting (MASS) to address these challenges by leveraging a deformable 2D Gaussian Surfel representation. We introduce the mesh-aligned Steiner Inellipse and fractal densification for mesh-to-surfel conversion that initiates high-resolution 2D Gaussian surfels from coarse parametric hand meshes, providing surface representation with photorealistic rendering potential. Second, we propose Gaussian Surfel Deformation, which enables efficient modeling of hand deformations and personalized features by predicting residual updates to surfel attributes and introducing an opacity mask to refine geometry and texture without adaptive density control. In addition, we propose a two-stage training strategy and a novel binding loss to improve the optimization robustness and reconstruction quality. Extensive experiments on the ARCTIC dataset, the Hand Appearance dataset, and the Interhand2.6M dataset demonstrate that our model achieves superior reconstruction performance compared to state-of-the-art methods.
Figures & tables
Figure 1: Hand reconstruction on real-world phone-shot videos. Extracted frames from two videos are shown in the first row. The second row images depict renderings of reconstructed hands.
Figure 2: Overall pipeline of our method. Starting with an egocentric monocular RGB video {Ik}k=1n , our pipeline first generates a parametric hand mesh T using a pre-trained model. The Mesh-to-Surfel Conversion module then transforms the coarse mesh T into a high-resolution 2D Gaussian Surfel representation, initializing surfel attributes (centroid px , scale s , and rotation r ). Next, the Gaussian Surfel Deformation module refines these attributes by predicting residual updates ( δpx , δs , δr ) and generating an opacity mask Mα to model complex hand deformations. Finally, the refined surfels are rendered following the 2D Gaussian Splatting pipeline, producing high-fidelity hand geometry and texture reconstructions. The optimization losses include image-based reconstruction loss Lc , geometry constraints Ld , and Ln , and weak geometry supervision Lsil from the SAM-HQ segmentation result.
Test-set
Methods
LPIPS ↓
PSNR ↑
MS-SSIM ↑
waffle
HARP
0.0345
28.74
0.9728
Ours
0.0328
34.33
0.9783
capsule
HARP
0.0359
29.37
0.9779
Ours
0.0381
31.22
0.9816
phone
HARP
0.0317
30.18
0.9767
Ours
0.2740
32.50
0.9807
Table 1: Quantitative Evaluation on the ARCTIC dataset [ 11 ] .
Figure 3: Varied Interacting objects and subjects of the quantitative evaluation on the ARCTIC dataset [ 4 ] . The Y-axis represents the best PSNR evaluation result on each test set.
Figure 4: (a) represents the architecture of the deformation module; (b) demonstrates some intermediate rendering results. Depth Distortion Map shows the inconsistency of the intersecting surfels. Surfel Map indicates the center of surfels with white small disks.
Figure 5: Comparison of rendering results with HARP on the ARCTIC dataset. Test results are shown for the sequences ’s01 laptop use’ and ’s02 waffle use’. Our method reconstructs the markers on the hand more clearly.”
Figure 6: Qualitative evaluation of the ablation study on the ARCTIC dataset [ 4 ] . (a) method without Fractal Densification; (b) method without Detached Gaussian surfels; (c) full method; (d) referring to ground truth. (e) control setting with lower loop subdivision
Methods
PSNR ↑
LPIPS ↓
SSIM ↑
HARP
21.50
0.107
0.878
3D-PSHR
23.55
0.095
0.893
Ours
26.32
0.091
0.921
Table 2: Quantitative evaluation on the Hand Appearance dataset [ 11 ] .
Methods
Views
val/Capture0
LPIPS ↓
PSNR ↑
SSIM ↑
SelfRecon
139
0.149
25.78
0.869
HTML
139
0.186
23.41
0.851
S2 HAND
139
0.146
25.94
0.877
AMVUR
139
0.132
27.43
0.885
HumanNeRF
139
0.119
27.80
0.882
Table 3: Quantitive evaluation on Interhand 2.6M. Note that our method selected 40,20,10 training views rather than 139 views from the HandAvatar evaluation setting.
Methods
PSNR ↑
LPIPS ↓
MS-SSIM ↑
Surfels
full model
32.47
0.0223
0.984
4 × N
model w/o Lb
27.0
0.0612
0.953
4 × N
w/o FD
30.60
0.0253
0.979
N
w/o DG
31.20
0.0267
0.977
4 × N
LS ↓
32.20
0.0168
0.986
N
Table 4: Quantitative evaluation of ablation study on “laptop” test set, N is 49216 surfels.
Figure 7: Visual comparison of the evaluation results of our method and the previous method on the hand appearance dataset [ 11 ] . Our rendering results have smoother boundaries and more detailed textures.
Figure 8: Visual result of our method on the Interhand 2.6M dataset [ 17 ] .
Figure 9: Rendering result of the experiment on Hand Appearance. For poses such as a closed fist, reference images show subtle shadows between fingers, but our model fails to reconstruct this effect.
HandAvatar
HARP
3D-PSHR
Ours
FPS
0.3
26
-
58
Table 5: Real-time performance in inference on the Interhand2.6M.
PSNR ↑
LPIPS ↓
MS-SSIM ↑
Noise Std
31.77
0.0242
0.982
0
31.58
0.0244
0.982
0.05
31.42
0.0244
0.981
0.1
31.06
0.0246
0.980
0.2
Table 6: Quantitative evaluation of ablation study on “laptop” test set, noise level is indicated by standard deviation.
Figure 10: Qualitative evaluation of the noise robustness ablation study on ARCTIC dataset.
Figure 11: Rendering and reconstruction on the in-the-wild video.
Figure 12: Rendering and reconstruction on the in-the-wild video with accessories.
Single-image 3D hand avatar reconstruction is fundamentally ill-posed and particularly challenging due to limited visual evidence under severe self-occlusion and the complex pose-dependent deformation of highly articulated hands. Existing methods predominantly rely on implicit NeRF-style representations, whose volumetric fitting is computationally expensive and often struggles to preserve fine-grained hand details. In this work, we present OASIS, a tailored 3D Gaussian Splatting framework for single-image hand avatar reconstruction. To faithfully encode sparse image-specific appearance cues in single-view reconstruction, we construct geometry-aligned visual evidence tokens by explicitly aligning input image observations with 3D hand geometry and context-adaptively tokenizing the resulting visual evidence. Since severe self-occlusion makes the reliability of image evidence inherently visibility-dependent, we introduce a visibility-conditioned point-image attention to reliably transfer visual evidence to geometric tokens, yielding occlusion-aware Gaussian features for faithful and robust reconstruction. To further capture non-rigid deformation of articulated hands, we introduce a Feature-on-Mesh representation to enable Gaussian deformation to be guided by local surface stretching. Under this framework, we adopt a one-shot adaptation scheme that learns a shared hand prior from multi-identity training data and then fits it to a target image for target-specific reconstruction. Extensive experiments show that OASIS outperforms existing baselines in both visual fidelity and efficiency across challenging poses and in-the-wild scenarios, and further demonstrates strong versatility in downstream applications such as text-to-avatar generation and texture editing.
Zhisheng Han, Shiyao Wu, Jiayan Qiu +6
University of Leicester Leicester, UK · University of Exeter Exeter, UK · University of Birmingham Birminghan, UK +1
Dynamic 3D hand reconstruction from egocentric videos is essential for next-generation computing platforms such as AR/VR and AI glasses. Despite its importance, most prior works focus either on multi-view 3D hand reconstruction or on 4D human body reconstruction. Egocentric 4D hand reconstruction remains difficult due to rapid hand and camera motion, hand-object and inter-hand interactions, and inherent ambiguity from single-view observations. To address these challenges, we introduce Hand-4DGS, a feed-forward framework for dynamic 4D hand reconstruction from egocentric videos. Our approach incorporates a mesh-guided representation for structural priors and temporal convolutions to model dynamic motion. We evaluate our framework on H2O and ARCTIC, two egocentric hand-object interaction datasets, and show improvements over baselines. Our model can efficiently adapt to unseen videos from datasets that are not included in training. In addition, image supervision through differentiable Gaussian rasterization provides an additional signal for appearance optimization and pose adjustment during training and test-time optimization, without ground-truth 3D hand pose annotations.
Jeongmin Bae, Seoha Kim, Marc Pollefeys +3
Yonsei University · Electronics and Telecommunications Research Institute · ETH Zurich +2
Despite recent progress in 3D hand reconstruction from monocular videos, most existing methods rely on data captured in well-controlled environments and therefore degrade in real-world settings with severe perturbations, such as hand-object interactions, extreme poses, illumination changes, and motion blur. To tackle these issues, we introduce WildGHand, an optimization-based framework that enables self-adaptive 3D Gaussian splatting on in-the-wild videos and produces high-fidelity hand avatars. WildGHand incorporates two key components: (i) a dynamic perturbation disentanglement module that explicitly represents perturbations as time-varying biases on 3D Gaussian attributes during optimization, and (ii) a perturbation-aware optimization strategy that generates per-frame anisotropic weighted masks to guide optimization. Together, these components allow the framework to identify and suppress perturbations across both spatial and temporal dimensions. We further curate a dataset of monocular hand videos captured under diverse perturbations to benchmark in-the-wild hand avatar reconstruction. Extensive experiments on this dataset and two public datasets demonstrate that WildGHand achieves state-of-the-art performance and substantially improves over its base model across multiple metrics (e.g., up to a 15.8% relative gain in PSNR and a 23.1% relative reduction in LPIPS). Our implementation and dataset are available at https://github.com/XuanHuang0/WildGHand.
Hanhui Li, Xuan Huang, Wanquan Liu +5
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, P.R. China, 518107. · Lenovo Research Group, Shenzhen, P.R. China, 518038.