cs.CVNov 17, 2026

HandRSI: Recursive Self-Improvement for Hand Pose Estimation

Authors: Zhekai Wang, Luqing Luo, Wenhao Yao, Junjie Lu, Huanan Zhu, Feifei Gao, Yuhua Jiang

Organizations: LivSyn Robotics · South China University of Technology · University of Technology Sydney · University of Science and Technology Beijing · 

Abstract

Egocentric 3D hand pose estimators still miss hands at the image border, misplace occluded hands and jitter. Such failures are usually fixed by hand-written inference refinements and by fine-tuning, two interdependent steps that are tuned separately. We present HandRSI, a recursive self-improvement framework for a hand pose system that consists of a video estimator and its executable inference harness. In each round, coding agents refine the harness, and a change is accepted only if it improves a composite of all ten benchmark metrics on held-out data without significantly regressing any of them. The accepted harness then guides the estimator through selective distillation, which supervises hand presence and temporal consistency on frames without ground truth, while ground truth supervises pose. Since the updated estimator may change which refinements remain useful, the harness is re-validated on it. Starting from the released ACE-Ego-Hand model, five rounds reduce MPJPE-p by 35% on HOT3D and 11% on ARCTIC and roughly halve jitter. On HOT3D, HandRSI's MPJPE-p is 16% lower than that of ACE-Ego-Hand fine-tuned on the same ground truth. Under a unified protocol, it outperforms nine re-run published methods on all ten metrics of both benchmarks. Hypervolume selection on the accuracy-speed front yields Lite, Medium and Heavy systems running at 20.7 to 41.1 FPS. This report gives the full method, every stage of every round and the complete protocol. Code, evaluator, harnesses and weights are public.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Estimating Accurate Hand Pose in Camera Space with Vision Transformer

    Sep 21, 2026Kaiwen Ren, Yiran Jiang, Yongjing Ye +13D Hand Pose Estimation

  2. VERA-Hand: A Verified Multidimensional Benchmark for 2D Hand Pose Estimation in Third-Person Dynamic Gesture Videos

    Oct 23, 2026Gibran Benitez-Garcia, Hiroki TakahashiHuman Pose EstimationObject Detection

  3. ScaleHP: Estimating Hand Pose in Metric Space

    Jun 24, 2026Ruitao Jing, Xingyu Chen, Hongyang Li +33D Hand Pose EstimationHuman Pose Estimation