Organizations: LivSyn Robotics · South China University of Technology · University of Technology Sydney · University of Science and Technology Beijing · Tsinghua University
Egocentric 3D hand pose estimators still miss hands at the image border, misplace occluded hands and jitter. Such failures are usually fixed by hand-written inference refinements and by fine-tuning, two interdependent steps that are tuned separately. We present HandRSI, a recursive self-improvement framework for a hand pose system that consists of a video estimator and its executable inference harness. In each round, coding agents refine the harness, and a change is accepted only if it improves a composite of all ten benchmark metrics on held-out data without significantly regressing any of them. The accepted harness then guides the estimator through selective distillation, which supervises hand presence and temporal consistency on frames without ground truth, while ground truth supervises pose. Since the updated estimator may change which refinements remain useful, the harness is re-validated on it. Starting from the released ACE-Ego-Hand model, five rounds reduce MPJPE-p by 35% on HOT3D and 11% on ARCTIC and roughly halve jitter. On HOT3D, HandRSI's MPJPE-p is 16% lower than that of ACE-Ego-Hand fine-tuned on the same ground truth. Under a unified protocol, it outperforms nine re-run published methods on all ten metrics of both benchmarks. Hypervolume selection on the accuracy-speed front yields Lite, Medium and Heavy systems running at 20.7 to 41.1 FPS. This report gives the full method, every stage of every round and the complete protocol. Code, evaluator, harnesses and weights are public.
Figures & tables
Figure 2: Overview of one HandRSI round. Loop A evolves the inference harness under the gate GA ; Loop B distills the refined harness into the estimator and re-validates it on the new network; the round gate G1% accepts the new system on val; Loop C turns the experimental archive into guidance for the next round.
Setting
Value
Per-metric clip of rm
±5%
Detection floor ( 1−am )
0.05
Regression
rm<−2% with CI 90<0 , or rm<−5%
Bootstrap
paired [ 16 ] , over sequences, 90%
Loop A pre-check (dev, uncharged)
Δ>0 , LΔ>0
Loop A gate GA (val)
G0.3% , 1,000 resamples
Table 1: Gate settings. Δ and the regression check use HOT3D; ARCTIC is scored and reported.
Figure 3: Re-validation after a model update (schematic). With θk+1 fixed, paired dev ablations on the same clips reassess every module of HkI,⋆ as useful, redundant or harmful; the weak modules are removed jointly, and the pruned chain is kept only if it passes the benchmark’s rule, otherwise HkI,⋆ is kept. Modules A–C illustrate one possible outcome.
Figure 4: Top: 3D PCK on the sealed tests (wrist-aligned; missed hands count as errors); legend: area under the curve up to 50 mm, HOT3D / ARCTIC; shaded: HandRSI’s margin over ACE-Ego-Hand . Bottom: temporal stability on one HOT3D test clip, image position of the index fingertip and per-frame joint acceleration: ACE-Ego-Hand’s acceleration peaks every four frames, HandRSI follows the ground truth without these steps (clip jitter 4.83 → 1.70 mm/frame 2 ).
Detection
3D pose
Placement
Temporal
Method
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
HOT3D (27 test recordings, 1,162 segments)
WiLoR [ 62 ]
0.861
0.911
0.949
26.82
7.44
41.67
61.48
22.75
0.082
16.22
HaWoR [ 92 ]
0.835
0.895
0.942
24.94
6.92
40.12
55.05
22.20
0.072
11.86
EgoForce [ 50 ]
0.781
0.858
0.917
38.52
7.43
53.01
88.39
33.99
0.105
18.34
ACE-Ego-Hand, K-free [ 45 ]
0.922
0.952
0.961
17.34
6.38
13.61
21.20
12.21
0.045
2.59
Table 3: Comparison with existing methods on the sealed test sets, all re-run by us on the same segments, intrinsics and evaluator; the strongest baselines are shown (all ten re-run configurations in Tab. 5 , none of the omitted ones is best in any column). Ours: the network alone, the three deployment tiers and the full system, with end-to-end FPS on one RTX PRO 5000 (Heavy and the full system also run WiLoR); ACE-Ego-Hand (K-given) is our starting point θ0 ; controls: θ0 fine-tuned on our GT ( C11 ) and the final harness on θ0 ( C52 ). MPJPE-p, PA-p, MPJPE +OOS in mm; EPE 2D -p in px; GO-p in degrees; CT-p in m; Jitter in mm/frame 2 ; best in bold.
Figure 5: Comparison with the strongest baselines on the sealed tests (same crop per row; white outline: ground truth; tag: 2D error in that frame). The baselines miss occluded or close hands and misplace interacting ones; HandRSI recovers and aligns them.
train
dev
val
test (sealed)
HOT3D
81 seq. + 62 w/o GT
15 seq.
13 seq.
27 seq., 1,162 seg.
ARCTIC
6 subjects + s03 w/o GT
1 subject
1 subject
s05, 291 seg.
HOI4D
–
–
–
166 rec., 498 seg.
Table 4: Splits (sequences; segments of 81 frames).
Method
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
HOT3D (27 test recordings, 1,162 segments)
HaMeR [ 59 ]
0.735
0.890
0.895
64.51
10.07
73.03
64.86
47.78
0.095
19.47
Hamba [ 13 ]
0.510
0.739
0.768
72.79
9.83
79.84
169.64
64.16
0.150
14.73
WiLoR [ 62 ]
0.861
0.911
0.949
26.82
7.44
41.67
61.48
22.75
0.082
16.22
WildHands [ 63 ]
0.710
0.868
0.873
51.59
12.13
56.88
102.30
49.61
0.140
23.29
InterWild [ 51 ]
0.729
0.885
0.891
72.16
11.10
80.78
69.45
54.14
0.110
50.93
Table 5: Every re-run method on the HOT3D and ARCTIC tests (Tab. 3 shows the strongest ones). Units as in Tab. 3 . a OmniHands’ HOT3D jitter comes from 0.17% of hand-frames where its PnP placement diverges.
Figure 6: Two HOT3D test clips with the network alone: two frames (cropped to the hand that ACE-Ego-Hand gets most wrong) and the clip’s hands in camera space with wrist trails (ground truth in gray). Below: the clip’s MPJPE-p and EPE 2D -p. ACE-Ego-Hand misses the hand at the image border (left) and misplaces the occluded hand (right); the HandRSI network recovers both.
Figure 7: Recursive improvement on the sealed test sets: the system Sk , its network θk , and the round’s harness on the original network ( Ck2 ); round 0 is ACE-Ego-Hand; val gains in Tab. 6 .
Round
subm. (acc.)
modules
network
val Δ (%)
1
70 (43)
18
θ1
+ 3.37 ( + 2.46) a
2
36 (6)
24
θ2
+ 3.54 ( + 2.86)
3
26 (12)
32
θ2
+ 3.17 ( + 2.74)
4
17 (9)
37
θ2
+ 2.06 ( + 1.84)
5
7 (3)
39
θ2
+ 0.77 ( + 0.47) b
Table 6: The five rounds : Loop A submissions (accepted by GA ), modules in the accepted harness, network, and the round gate’s val gain over the previous system (CI 90 lower bound). From round 3 on, no student passed GB , so the network stays θ2 . a Accepted with FAcc past the 5% cap ( rm=−5.6% , not significant), as the network trades missed hands for false positives (Sec. 4.5 ); b below the round gate, kept by the harness-only rule.
Figure 8: One HOT3D test clip after every round. Top: two frames of the right hand (same crop per column); bottom: its wrist-aligned joint error in each of the 81 frames (red: missed; ticks: the frames above) and the clip’s MPJPE-p. ACE-Ego-Hand misses the hand in 22 frames and S1 in 8; from round 2 on it is always found, and later rounds lower its error on the detected frames (14.4 → 12.5 mm).
Figure 9: Accuracy–speed tiers. (a) Selection on val: timed sub-chains of the final harness (gray), the front, and the three tiers that maximize the hypervolume (Eq. 13 ) for r , shaded by the best tier per time budget; sub-chains are scored after the loop, so one can exceed the full chain. (b, c) Reduction of jitter and MPJPE-p relative to θ2 alone along the val front on the sealed tests.
Figure 10: The harness over the rounds : modules of each accepted HkI by function (from each module’s own description); above the bars, modules added / removed relative to the previous round.
Figure 11: Every Loop A gate decision on val (the 98 of 156 submissions that passed the dev pre-check, five rounds): composite gain against the agent’s incumbent and the worst single-metric change. Dashed: τ ; dotted: the regression thresholds ( rm<−2% when significant, rm<−5% always). Two rejections beyond −10% are drawn at the border.
Round
subm.
gated
acc.
Claude
GPT
tokens (M)
1
70
53
43
14/21
29/32
1.67
2
36
13
6
4/9
2/4
2.04
3
26
14
12
3/5
9/9
2.02
4
17
12
9
3/4
6/8
1.77
5
7
6
3
2/4
1/2
1.19
all
156
98
73
26/43
47/55
8.68
Table 7: Loop A per round : submissions, of which gated on val (after the dev pre-check) and accepted by GA ; accepted/gated per agent family; output tokens of all agents.
Variant
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
ACE-Ego-Hand θ0 , no harness
0.970
0.986
0.990
12.97
5.90
12.84
9.66
6.90
0.024
2.82
GT-only fine-tuning of θ0 ( C11 )
0.982
0.994
0.994
9.91
5.00
10.76
5.56
5.71
0.014
2.77
GT + distillation, round 1 ( θ1 )
0.981
0.994
0.994
9.89
5.00
10.75
5.77
5.80
0.014
2.63
GT-only fine-tuning of θ1 ( C21 )
0.983
0.994
0.994
9.39
4.76
10.33
5.32
5.71
0.013
2.73
GT + distillation, round 2 ( θ2 )
0.982
0.994
0.994
9.37
4.71
10.35
5.31
5.80
0.013
2.51
H1I on θ0 ( C12 )
0.976
0.993
0.992
11.13
5.09
11.73
7.96
5.92
0.021
1.75
Table 8: System-level ablation on HOT3D test, built from the controls scored in every round: network only, harness (or a baseline) only, and both.
Loop B recipe
FAcc
MPJPE-p
EPE 2D -p
Jitter
Δ (%)
GT only
0.954
10.24
5.37
2.73
–
+ teacher articulation
0.965
10.60
5.75
2.76
− 1.62
+ articulation, temp. 0.3
0.942
10.38
5.53
2.33
− 0.70
+ presence, 2D, temp. 0.3
0.944
10.40
5.58
2.32
− 1.01
+ presence, temp. 0.3 ‡
0.954
10.19
5.39
2.32
+ 1.39
temp. 0.1
0.961
10.24
5.33
2.54
+ 1.95
Table 9: What to distill (round 1, HOT3D dev): Loop B recipes from the same GT-fine-tuned network and budget; Δ against GT only. ‡ Indented rows build on this row.
Table 10: Harness modules kept by re-validation for each Loop B recipe.
System
Modules
FPS ↑
harness ms/frame ↓
val Δ (%) ↑
ACE-Ego-Hand ( θ0 )
–
42.7
0
–
θ2 alone
0
42.5
0
+ 0.00
HandRSI-Lite
20
41.1
0.9
+ 3.90
HandRSI-Medium
28
34.0
6.0
+ 4.06
HandRSI-Heavy
29
20.7
25.0
+ 4.93
S5 (loop output)
39
16.2
38.5
+ 4.84
Table 11: Deployment tiers chosen on val (Fig. 9 ); test accuracy in Tab. 3 . Harness time per frame on one RTX PRO 5000, after the network.
R
Loop C change
Δ ( LΔ )
MPJPE
EPE 2D
GO
Jitter
Recall
2
–
+ 1.66 ( + 0.80)
+ 6.4
+ 10.6
− 0.8
+ 4.0
+ 2.2
3
λR 0.5
− 0.84 ( − 2.12)
− 1.5
− 2.6
− 3.3
+ 2.1
− 1.7
4
articulation on HOT3D
− 1.09 ( − 2.55)
− 1.0
− 3.3
− 4.8
+ 1.9
− 2.2
5
λKD 1, lr 5⋅10−5
− 2.35 ( − 3.21)
− 2.1
− 6.8
− 8.0
+ 1.1
− 6.5
Table 12: Loop B’s best student checkpoint per round on val (network alone vs. θk−1 ): change set by Loop C, Δ with its CI 90 lower bound, and rm (%) of five columns. All four checkpoints of rounds 3–5 have the same signs.
Method
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
HOT3D test
ACE-Ego-Hand θ0 , no harness
0.970
0.986
0.990
12.97
5.90
12.84
9.66
6.90
0.024
2.82
θ0 + Savitzky–Golay
0.970
0.986
0.990
12.94
5.89
12.82
9.60
6.86
0.024
2.44
θ0 + 2D alignment + Savitzky–Golay
0.863
0.997
0.958
11.63
5.86
12.82
8.70
5.26
0.022
2.41
H5I on θ0 ( C52 )
0.972
0.986
0.991
10.38
5.21
10.34
7.56
6.56
0.024
1.44
HandRSI network θ2 , no harness
0.982
0.994
0.994
9.37
4.71
10.35
5.31
5.80
0.013
2.51
Table 13: Classical post-processing against the agent-written harness on the sealed tests, on the untouched network θ0 and on θ2 . Savitzky–Golay (window 7, order 2) is the best of 30 filter settings on HOT3D val; 2D alignment refits the hand’s image-plane translation to the network’s own 2D keypoints by least squares before smoothing (the best alignment variant on val). WiLoR ensemble: WiLoR’s finger rotations blended into the network’s with one global weight (0.25, selected on val), then Savitzky–Golay. Units as in Tab. 3 ; best in bold.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
System
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
θ0
0.970
0.986
0.990
12.97
5.90
12.84
9.66
6.90
0.024
2.82
Round 1
Ck1
0.982
0.994
0.994
9.91
5.00
10.76
5.56
5.71
0.014
2.77
teacher Tk−1
0.984
0.995
0.995
9.82
4.96
10.75
5.01
5.58
0.013
1.74
θ1
0.981
0.994
0.994
9.89
5.00
10.75
5.77
5.80
0.014
2.63
S1
0.977
0.998
0.993
9.39
4.83
10.78
5.38
5.20
0.012
1.65
Appendix
Table 14: HOT3D test, every stage of every round. Per round k : the GT-only control Ck1 , the teacher Tk−1 , the network θk , the system Sk and the round’s harness on the original network ( Ck2 ). Units as in Tab. 3 .
System
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
θ0
0.986
0.995
0.996
15.58
7.43
16.54
9.36
12.45
0.024
2.58
Round 1
Ck1
0.988
0.996
0.997
15.51
7.36
16.49
9.28
12.40
0.023
2.57
teacher Tk−1
0.990
0.997
0.997
15.31
6.84
16.41
8.82
12.19
0.022
1.80
θ1
0.987
0.996
0.996
15.40
7.36
16.38
9.20
12.50
0.022
2.44
S1
0.987
1.000
0.997
14.84
6.69
16.18
8.31
11.92
0.019
1.72
Appendix
Table 15: ARCTIC s05 (test), every stage of every round (as Tab. 14 ).
System
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
θ0
0.994
0.994
0.994
23.45
11.02
22.93
15.60
19.39
0.058
2.10
Round 1
Ck1
0.994
0.994
0.994
23.83
11.01
23.29
15.71
19.35
0.062
2.07
θ1
0.993
0.994
0.994
23.71
11.08
23.13
16.18
19.43
0.062
1.98
S1
0.990
0.990
0.990
23.51
10.36
22.54
20.02
19.98
0.064
1.26
Ck2
0.991
0.991
0.992
23.22
10.25
22.39
18.28
19.80
0.060
1.31
Appendix
Table 16: HOI4D (zero-shot), every stage of every round (as Tab. 14 ).
Method
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
HOI4D (zero-shot, 498 segments)
HaMeR [ 59 ]
0.930
0.926
0.961
45.10
11.66
45.12
75.83
34.14
0.170
20.95
Hamba [ 13 ]
0.920
0.913
0.954
46.25
11.24
46.22
90.45
35.73
0.167
18.77
WiLoR [ 62 ]
0.978
0.976
0.988
34.10
10.37
34.11
30.59
25.36
0.105
16.44
WildHands [ 63 ]
0.930
0.925
0.961
44.16
13.96
44.17
81.98
46.01
0.152
18.57
InterWild [ 51 ]
0.929
0.925
0.960
53.83
12.98
53.84
77.67
41.82
0.173
48.22
Appendix
Table 17: Zero-shot HOI4D test , every re-run method (units as in Tab. 3 ).
System
FAcc ↑
Recall ↑
F1 ↑
MPJPE-p ↓
PA-p ↓
MPJPE +OOS↓
EPE 2D -p ↓
GO-p ↓
CT-p ↓
Jitter ↓
ACE-Ego-Hand ( θ0 )
0.959
0.986
0.987
12.97
5.90
12.84
9.66
6.90
0.024
2.82
θ2 alone
0.930
0.994
0.974
9.37
4.71
10.35
5.31
5.80
0.013
2.51
HandRSI-Lite
0.930
0.996
0.974
9.06
4.69
10.31
4.76
5.44
0.012
1.47
HandRSI-Medium
0.930
0.996
0.974
9.04
4.64
10.30
4.79
5.46
0.013
1.33
HandRSI-Heavy
0.932
0.996
0.974
8.41
4.35
9.71
4.16
5.17
0.013
1.35
S5
0.932
0.996
0.975
8.37
4.37
9.70
4.07
5.16
0.013
1.34
Appendix
Table 18: HOT3D test under the strict detection rule (predictions on unannotated hand slots count as false positives).
Figure 12: All 124 searched configurations on val against the summed per-module cost (an estimate that overstates WiLoR about 2.5 × ; Fig. 9 uses measured time).
Figure 13: Decoded sequence length : network alone on HOT3D test against the length decoded in one pass (dashed: 81-frame segments).
Figure 14: Extended comparison on the sealed HOT3D test. Per example: the frame with each method’s meshes (white outline: ground truth; tag: mean 2D joint error of the visible hands, “missed” if one of them is not output) and the same hands in 3D from one third-person view above a floor, identical in every column. Frames: the two with the largest gap between the baselines’ and HandRSI’s 2D error, then the two largest among frames where every method outputs every visible hand; one per recording.
Figure 15: Extended comparison on the sealed ARCTIC test (held-out subject s05), as in Fig. 14 .
Figure 16: Extended comparison on HOI4D , zero-shot for every method, as in Fig. 14 .
Figure 17: Three more HOT3D test clips after every round (as Fig. 8 ), the three with the largest total drop among the clips whose MPJPE-p falls in every round. Per clip: two frames of the hand that improves most over the clip (same crop in every column), that hand’s wrist-aligned joint error in each of the 81 frames (red: missed; ticks: the two frames shown), and the clip’s MPJPE-p, which also counts the other hand.
Graduate School of Informatics and Engineering, The University of Electro-Communications, Tokyo, Japan · Artificial Intelligence eXploration Research Center (AIX), The University of Electro-Communications, Tokyo, Japan · Meta-Networking Research Center (MEET), The University of Electro-Communications, Tokyo, Japan