Reconstructing hand-object interactions is an important task, and achieving it with a single camera makes large-scale capture practical but challenging due to heavy occlusions. We introduce HOForce to track hands and objects, and further recover interaction forces from single-view RGB-D video and known object properties. For two-hand manipulation, we jointly model fingertip forces of both hands and the contact forces between objects when two objects are presented. The reconstructed forces also improve motion tracking by giving pose constraints in the occluded regions. HOForce recovers hand and object motion more accurately on DexYCB and TACO and produces less hand-object penetration than interaction baselines on DexYCB. On our dataset of fingertip forces recorded with tactile sensors, HOForce reduces normal-force MAE by 24.5% relative to the strongest baseline. The recovered forces reproduce object motion in MuJoCo and transfer to other engines after adaptation.
Figures & tables
Figure 2: Method overview. Stage 1 jointly refines hand and object motion with nonpenetration constraints. The finger bone axis guides the correction at thin walls, while the object contact term penalizes overlap between the brush and bowl. Stage 2 fits fingertip forces by comparing simulated and reconstructed object motion. The remaining motion error guides adjustments to contact locations and hand poses, and the forces are then re-estimated at the updated contacts.
Figure 3: Resolving finger penetration at thin walls. In both panels, the bone axis runs from parent joint a to its child b . (a) The thumb axis crosses the wall at q . The entry-face normal guides the thumb back toward the side it entered from. (b) Only the middle-finger capsule crosses the wall. The object SDF gradient at b identifies the exit direction.
Figure 4: Hand–object reconstruction on DexYCB. Each column shows one example, with rows showing the input RGB, ground truth, our final result, AGILE, and HOTrack. Each reconstruction is rendered from two viewpoints shared by all methods. Red boxes highlight finger penetration, object-pose discrepancies, and mismatched grasp configurations in the baselines.
Objects
ADD ↑
ADD-S ↑
T↓
R↓
A↓
PD OO↓
IV OO↓
FP
81.65
92.54
11.46
15.77
3.09
0.33
0.14
ICG
77.43
89.04
15.84
20.13
3.60
0.85
0.93
Point2Pose
84.57
92.62
11.30
12.28
5.98
0.38
0.25
Ours
86.31
95.52
7.37
11.75
1.20
0.12
0.02
Ours (replay)
76.52
90.02
14.28
17.97
1.83
0.03
0.00
Table 1: Object tracking and hand reconstruction on TACO.
Object tracking
ADD ↑
ADD-S ↑
T↓
R↓
A↓
FP
73.47
79.28
62.37
21.30
3.79
ICG
72.57
83.16
41.84
25.89
3.62
Point2Pose
74.07
80.10
55.12
22.82
7.30
HOTrack ∗
79.83
92.76
13.31
22.19
4.80
AGILE ∗
54.59
69.20
81.43
26.55
–
Ours
84.87
93.25
12.50
15.73
1.41
Table 2: Object tracking and hand reconstruction on DexYCB.
Figure 5: Bimanual two-object reconstruction on TACO. Each column shows one interaction from the input view (RGB and mesh overlay) and an alternate view.
Method
MAE (N) ↓
RMSE (N) ↓
EgoTac [ 46 ]
1.473
2.236
EgoPHI [ 16 ]
1.396
2.132
HOPE [ 17 ]
1.326
2.063
Ours
1.002
1.523
Table 3: Measured hand–object force accuracy. Sequence-balanced normal-force errors on 32 sequences. Best in bold; second best underlined.
Figure 6: Force reconstruction under changing loads. Arrows show estimated normal forces in frames A–G (shaded). Curves compare measured and estimated thumb forces over 931 intervals (31 s). See the supplement for all three fingers.
Engine
Forces
RMSE (mm) ↓
Insertion ↑
MuJoCo
Source
–
20/21
PyBullet
Direct
3563.90
0/21
Adapted
5.12
19/21
Isaac Sim
Direct
1432.69
0/21
Adapted
13.37
19/21
Table 4: Cross-engine transfer. Median translation RMSE against MuJoCo over 21 sequences, combining both objects per sequence, and valid insertion counts.
Figure 7: DexYCB grasp ablations. Paired views highlight changes in hand placement (orange), penetration (red), and fingertip contact (purple).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Residual or block
Coefficient
Hand depth / object visual block
1/0.3
Hand rotation prior: wrist / finger
106/103
Wrist translation: first / second order
150/150
Wrist / finger rotation: first order
2×107/2×107
Object translation: first / second order
30/100
Object rotation: first / second order
103/105
Appendix
Table S1: Joint reconstruction weights. Paired values follow the order in the left column. Translation and depth are in mm, rotation in radians, and silhouette distance in pixels.
Figure S1 : Additional DexYCB grasps. Rows show the input RGB, ground truth, our final reconstruction, AGILE, and HOTrack. The two renderings in each cell use the same viewpoints across methods.
Figure S2 : Additional bimanual interactions on TACO. Each column shows the input RGB, the reconstruction overlaid on the input, and an alternate view.
Term or setting
Symbol
Value
Main Eq. (6): dynamics / contact
λdyn,λcontact
106,1
Main Eq. (6): observation / prior / temporal
λobs,λprior,λtemp
1,1,1
Reconstructed object trajectory (Sec. A .2)
Object observation metric: translation / rotation
kp,kR
9×106,22500
Contact coupling / object step scale
λint,ηo
0.1,0.5
Contact selection and force fitting
Appendix
Table S2: Force optimization and physical parameters. Symbols are defined in the text and equations above. Coefficients multiply costs, rather than residuals. All geometric residuals below use meters; angular residuals use radians. Reconstruction parameters inherited from Sec. A are listed separately from the force/contact updates.
Figure S3 : Force reconstruction for all three measured fingers. Rows compare each method with the measured normal force; columns show the thumb, middle, and index fingers. All panels use identical axes and the same 931 frame intervals over the complete 31-second sequence. Row labels report MAE across all three fingers.
Figure S4 : Object models used in the measured-force sequences.
Prediction
Mean 3D error
Mean tangential error
3D RMSE
Normal only
1.4858
0.8887
2.1016
Full vector
1.2216
0.5248
1.6892
Appendix
Table S3: Effect of tangential force estimates on all 32 sequences. Both rows use HOForce predictions. Normal only retains the predicted normal component and sets the tangential component to zero. All errors are in N; lower is better.
Figure S5 : Final states of all 21 PyBullet replays after force adaptation. The rod is inserted in 19 sequences. The two failures are marked (F). Labels identify the recording and segment; each view is framed to show both objects.
Pair
Stage 1 variant
PD (mm) ↓
IV (cm 3 ) ↓
Object–object
w/o LOO
0.31
0.15
Full
0.12
0.02
Hand–object
w/o LHO
3.81
0.90
Bidirectional SDF
3.38
0.64
Full
1.73
0.20
Appendix
Table S4 : Penetration on 228 TACO sequences. The hand variants remove the hand–object loss or replace its correction with a bidirectional signed-distance penalty.
Pair
Reconstruction
PD (mm) ↓
IV (cm 3 ) ↓
Rod–base
Both losses off
2.60
7.21
Reference tracks
0.06
0.00
Hand–object
Both losses off
5.50
0.71
Reference tracks
0.15
0.00
Appendix
Table S5 : Penetration on 21 recorded rod–hole insertion segments. Both mean reference IVs round to 0.00 cm 3 .