WildGHand: Learning Anti-Perturbation Gaussian Hand Avatars from Monocular In-the-Wild Videos

Hanhui Li; Xuan Huang; Wanquan Liu; Yuhao Cheng; Long Chen; Yiqiang Yan; Xiaodan Liang; Chenqiang Gao

cs.CVFeb 24, 2026

WildGHand: Learning Anti-Perturbation Gaussian Hand Avatars from Monocular In-the-Wild Videos

Authors: Hanhui Li, Xuan Huang, Wanquan Liu, Yuhao Cheng, Long Chen, Yiqiang Yan, Xiaodan Liang, Chenqiang Gao

Organizations: School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, P.R. China, 518107. · Lenovo Research Group, Shenzhen, P.R. China, 518038.

Abstract

Despite recent progress in 3D hand reconstruction from monocular videos, most existing methods rely on data captured in well-controlled environments and therefore degrade in real-world settings with severe perturbations, such as hand-object interactions, extreme poses, illumination changes, and motion blur. To tackle these issues, we introduce WildGHand, an optimization-based framework that enables self-adaptive 3D Gaussian splatting on in-the-wild videos and produces high-fidelity hand avatars. WildGHand incorporates two key components: (i) a dynamic perturbation disentanglement module that explicitly represents perturbations as time-varying biases on 3D Gaussian attributes during optimization, and (ii) a perturbation-aware optimization strategy that generates per-frame anisotropic weighted masks to guide optimization. Together, these components allow the framework to identify and suppress perturbations across both spatial and temporal dimensions. We further curate a dataset of monocular hand videos captured under diverse perturbations to benchmark in-the-wild hand avatar reconstruction. Extensive experiments on this dataset and two public datasets demonstrate that WildGHand achieves state-of-the-art performance and substantially improves over its base model across multiple metrics (e.g., up to a $15.8\%$ relative gain in PSNR and a $23.1\%$ relative reduction in LPIPS). Our implementation and dataset are available at https://github.com/XuanHuang0/WildGHand.

Figures & tables

Fig. 1: We present WildGHand, a novel Gaussian splatting framework for generating realistic hand avatars from short monocular videos exhibiting challenging perturbations, including hand-object interactions, complex poses, illumination variations, and motion blur.

Dataset	Env.	Clean Testset	Single Hand	Int. Hand	Hand-Object Int.	Complex Poses	Illum. Var.	Motion Blur
InterHand2.6M [ 7 ]	Lab	✓	✓	✓
HARP [ 20 ]	Wild	✓	✓
UHM [ 8 ]	Wild	✓	✓
DexYCB [ 18 ]	Lab		✓		✓	✓
EpicKitchen [ 48 ]	Wild			✓	✓	✓	✓	✓
HInt [ 21 ]	Wild		✓	✓	✓	✓		✓

TABLE I: Detailed comparison between the proposed HWP dataset and existing hand video datasets.

Fig. 2: The proposed WildGHand framework. Given a monocular video affected by perturbations, WildGHand introduces two key components to achieve the robust estimation of 3D Gaussians, including a lightweight dynamic perturbation disentanglement (DPD) module and a perturbation-aware optimization (PAO) strategy. The DPD module represents potential perturbations by biases of Gaussian attributes, which are optimized guided by the weighted masks predicted by the PAO strategy. During inference, the optimized biases are removed to render perturbation-free images.

Fig. 3: Illustration of the proposed PAO and DPD modules. Left: Our perturbation-aware optimization (PAO) strategy segments the hand regions and leverages reconstruction error to generate weighted masks to guide the optimization of 3DGS. Right: The temporal weights estimated by our dynamic perturbation disentanglement (DPD) module that reflect the strengths of perturbations. Partial perturbations (e.g., occlusions) tends to have smaller weights (labeled in green), while holistic perturbations (e.g., motion blur) have larger weights (labeled in red).

Capture	Hand	Action	Capture Device	$\#$ Frames	Perturbations
Capture	Hand	Action	Capture Device	$\#$ Frames	Interaction	Complex Poses	Illumination Changes	Motion Blur
0	Single	Grabbing blocks	Mobile phone	901	✓	✓
1	Single	Multiple gestures	Mobile phone	742		✓		✓
2	Single	Multiple gestures	Mobile phone	828			✓
3	Single	Spinning a pen	Mobile phone	1086	✓			✓
4	Interacting	Spinning a pen	Webcam	1662	✓			✓

TABLE II: Detailed description of the proposed HWP dataset, covering hand type, primary action, recording device, number of frames, and the presence of perturbations.

Fig. 4: Visual examples of our HWP dataset, which covers diverse challenging scenes like motion blur, hand-object interactions, illumination variations, and complex poses (from top left to bottom right).

Method	WildGHand (Single)			WildGHand (Inter.)			InterHand2.6M (Inter.)			AnchorCrafter (Inter.)
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
UHM [ 8 ]	20.24	0.867	0.203	–	–	–	–	–	–	–	–	–
Handy [ 63 ]	22.99	0.874	0.127	24.10	0.864	0.141	24.34	0.867	0.198	25.88	0.912	0.074
InterGaussianHand [ 10 ]	23.06	0.858	0.147	24.69	0.851	0.148	26.70	0.861	0.173	24.28	0.912	0.098
WildGHand (Ours)	26.24	0.916	0.116	26.29	0.893	0.122	28.25	0.894	0.151	27.68	0.948	0.072

TABLE III: Comparison with state-of-the-art methods on the three dataset, including WildGHand, InterHand2.6M and AnchorCrafter. “Single” and “Inter.” denote single-hand and interacting-hand scenarios, respectively.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Handy [ 63 ]	21.46	0.809	0.191
InterGaussianHand [ 10 ]	22.95	0.820	0.157
WildGHand (Ours)	27.39	0.920	0.080

TABLE IV: Comparison on online videos.

Fig. 5: Qualitative comparisons between our proposed WildGHand model with state-of-the-art methods on interacting-hand videos (top) and single-hand videos (bottom).

Fig. 6: Visual example of the ablation study on the proposed modules. Left: Weight masks learned by the proposed PAO strategy, light hand regions indicate higher weights. Right: Results of the variants of the proposed method. The full model obtains the best results consistently on various videos.

Method	Hand-object Interactions			Complex Poses			Illumination Variations			Motion Blur
Method	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
Handy	24.67	0.872	0.138	24.14	0.863	0.138	20.65	0.814	0.156	24.33	0.872	0.139
InterGaussianHand	25.11	0.858	0.146	24.51	0.848	0.142	22.18	0.805	0.158	24.71	0.860	0.145
WildGHand (ours)	26.87	0.902	0.118	25.29	0.881	0.124	22.81	0.839	0.148	26.68	0.901	0.116

TABLE V: Comparison with state-of-the-art methods on the interacting-hand videos of the HWP dataset, evaluated across four scenarios: hand-object interactions, complex poses, illumination variations, and motion blur.

DPD	PAO	Single Hand			Interacting Hands
DPD	PAO	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
		22.65	0.862	0.151	24.86	0.867	0.123
✓		23.00	0.869	0.138	25.01	0.868	0.122
✓	✓	26.24	0.916	0.116	26.29	0.893	0.122

TABLE VI: Ablation study on the proposed modules.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
WildGHand (ours)	26.44	0.871	0.179
WildGHand w/ binary mask	25.36	0.844	0.184

TABLE VII: Comparison between PAO and SAM-based binary mask.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
WildGHand (ours)	26.44	0.871	0.179
w/o Heads (iii) & (iv)	26.20	0.864	0.177

TABLE VIII: Ablation on prediction heads.

Capture	Disturbance	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Avg. T. Wt. $(\times 10^{-3})$ $\uparrow$
4	Low	29.59	0.931	0.115	0.234
5	Medium	28.00	0.915	0.116	1.581
6	High	27.83	0.907	0.123	7.572

TABLE IX: Average temporal weight vs. perturbation level. Higher perturbation correlates with larger weights and degraded quality. Weights are scaled by

10^{-3}

for readability.

Method	PSNR↑	SSIM↑	LPIPS↓
InterGaussianHand	26.2518 $\pm$ 0.0770	0.8723 $\pm$ 0.0009	0.1582 $\pm$ 0.0012
WildGHand (ours)	29.5859 $\pm$ 0.1028	0.9313 $\pm$ 0.0015	0.1146 $\pm$ 0.0029

TABLE X: Mean and standard deviation of the performance of InterGaussianHand and WildGHand.

3D Hand Reconstruction

See all 36 papers

Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions
Apr 14, 2026Ayce Idil Aytekin, Xu Chen, Zhengyang Shen +4Hand-Object Interaction Reconstruction 3D Reconstruction
MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA +1
GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting
Mar 19, 2026Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini +4Hand-Object Interaction Reconstruction Hand-Object Interaction
RPTU · DFKI-AV Kaiserslautern · NUST-SEECS Pakistan +1
OASIS: Occlusion-aware Single-image Hand Avatar Reconstruction via 3D Gaussian Splatting
Jul 31, 2026Zhisheng Han, Shiyao Wu, Jiayan Qiu +63D Gaussian Splatting 3D Reconstruction
University of Leicester Leicester, UK · University of Exeter Exeter, UK · University of Birmingham Birminghan, UK +1

Apr 14, 2026J/K move · Enter open · S save

Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions

Ayce Idil Aytekin, Xu Chen, Zhengyang Shen, Thabo Beeler+3

MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA+1

Hand-Object Interaction Reconstruction 3D Reconstruction Monocular 3D Reconstruction 3D Hand Pose Estimation

Open paper AskPDF arXiv

We present Grasp in Gaussians (GraG), a fast and robust method for reconstructing dynamic 3D hand-object interactions from a single monocular video. Unlike recent approaches that optimize heavy neural representations, our method focuses on tracking the hand and the object efficiently, once initialized from pretrained large models. Our key insight is that, given strong pretrained object and hand priors, accurate and temporally stable hand-object motion can be recovered using a compact Sum-of-Gaussians (SoG) representation, revived from classical tracking literature and integrated with generative Gaussian-based initializations. We initialize object pose and geometry using a video-adapted SAM3D pipeline, then convert the resulting dense Gaussian representation into a lightweight SoG via subsampling. This compact representation enables efficient and fast tracking while preserving geometric fidelity, with the pretrained priors and simple geometric/contact losses providing accurate hand-object placement. For the hand, we adopt a complementary strategy: starting from off-the-shelf monocular hand pose initialization, we refine hand motion using simple yet effective 2D joint, silhouette, depth, and contact losses, avoiding per-frame refinement of a detailed 3D hand appearance model while maintaining stable articulation. Extensive experiments on public benchmarks demonstrate that GraG reconstructs temporally coherent hand-object interactions on long sequences 4.4x-38.9x faster than prior work while preserving temporally coherent hand-object motion.

Abstract

3D Hand Reconstruction

Explore similar work

Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions

GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

OASIS: Occlusion-aware Single-image Hand Avatar Reconstruction via 3D Gaussian Splatting