Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves. However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. VOKE contains two core perception modules and a downstream application study. First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain.
Figures & tables
Fig. 1: Motivation and overview of VOKE. Top: VOKE augments frame-wise object pose estimation with explicit linear and angular velocity and acceleration. Bottom: During real-world deployment, VOKE converts live robot RGB(D) observations into kinematics estimates for dexterous in-hand manipulation.
Fig. 2: Overview of VOKE. Given a sequence of RGB(D) frames, a base pose estimator shares its image encoder with the Pluggable Kinematics Module (PKM) and produces per-frame pose estimates. PKM aggregates the last k frame features and decodes the motion vector Vt=(vt,ωt,at,βt) . The Kinematics-guided Pose Refinement Module (KPRM) constructs a motion-prior reference pose from Pt−1,net and Vt , then refines the per-frame pose using Vt and the pose history Pt−k+1:t,net . The refined pose and motion vector form the complete kinematic state Kt=(Pt,fin,Vt) consumed by the Kinematics-Augmented Reinforcement Learning (KARL) policy together with robot proprioception.
Backbone
Method
Eo↓
Ec↓
ADD-S ↓
ADD-0.1D(s) ↑
CDo↓
FSo@5↑
FSo@10↑
RPE ↓
mean
median
Trans.
Rot.
gSDF
DexYCB orig.
1.95
2.89
1.40
52.5
3.76
1.58
0.439
0.707
2.50
4.90
DexYCB +VOKE
1.80
2.75
1.35
53.3
3.50
1.43
0.456
0.724
0.89
4.47
OakInk orig.
2.24
3.15
1.55
48.9
6.51
1.52
0.414
0.677
2.65
5.04
OakInk +VOKE
2.12
2.99
1.48
50.2
6.07
1.46
0.428
0.686
1.06
4.64
HFL-Net
DexYCB orig.
3.98
5.57
3.19
30.2
–
–
–
–
3.67
4.45
TABLE I: Main object-pose estimation results and temporal stability of the pose predictions on DexYCB and OakInk. Values absent from the original publications are recomputed from our inference outputs where possible; “–” denotes metrics unavailable from those outputs. Here, Eo , Ec , and ADD-S are in cm; ADD-0.1D(s) is in percent; CDo is in cm 2 ; and translational and rotational RPE are in cm and degrees, respectively.
Dataset
Method
MEE ↓
NRMSE ↓
v
ω
a
β
v
ω
a
β
DexYCB
Pose Diff (SG)
0.147
2.022
6.176
32.997
0.832
1.472
4.928
2.367
VOKE
0.087
0.965
1.307
19.832
0.259
0.468
0.871
0.984
OakInk
Pose Diff (SG)
0.140
1.858
6.064
27.507
0.932
1.225
6.128
2.398
VOKE
0.065
0.553
0.845
12.225
0.317
0.456
0.778
0.898
Simulation
Pose Diff (SG)
0.598
12.552
16.142
169.133
1.550
1.790
5.005
4.127
TABLE II: Object velocity and acceleration estimation on DexYCB, OakInk, and the dexterous in-hand manipulation simulation dataset. Here, v , ω , a , and β denote linear velocity, angular velocity, linear acceleration, and angular acceleration, respectively. Their MEE units are m/s , rad/s , m/s2 , and rad/s2 , respectively; NRMSE is dimensionless.
Fig. 3: Visualization of VOKE on DexYCB and OakInk datasets. The three left columns show the last three input frames; the four right columns show estimates of linear and angular velocity and acceleration. Green arrows denote the reference motion quantities derived from ground-truth poses, red arrows denote VOKE estimates, and light-blue boxes visualize the 3D bounding boxes of the refined poses.
Method
CDo↓
FSo@5↑
ADD-S ↓
Eo↓
mean
median
DexYCB
gSDF
3.76
1.58
0.439
1.40
1.95
VOKE w/o motion & ref. pose
3.70
1.59
0.438
1.41
1.94
VOKE w/o ref. pose
3.66
1.54
0.442
1.38
1.90
VOKE (full)
3.50
1.43
0.456
1.35
1.80
TABLE III: Ablation study of pose refinement components on the DexYCB and OakInk datasets.
Metric
k
v
ω
a
β
MEE ↓
1
0.671
3.720
13.693
90.610
2
0.150
1.774
2.817
62.089
3
0.087
0.965
1.307
19.832
5
0.090
0.971
1.210
18.718
NRMSE ↓
1
1.717
1.923
7.198
5.205
2
0.516
0.958
3.950
2.993
TABLE IV: Ablation on the temporal window size k : velocity and acceleration estimation accuracy on DexYCB. Here, v , ω , a , and β denote linear velocity, angular velocity, linear acceleration, and angular acceleration, respectively. Their MEE units are m/s , rad/s , m/s2 , and rad/s2 , respectively; NRMSE is dimensionless.
k
CDo↓
FSo@5↑
ADD-S ↓
Eo↓
mean
median
1
3.72
1.58
0.440
1.40
1.95
2
3.55
1.50
0.447
1.37
1.89
3
3.50
1.43
0.456
1.35
1.80
5
3.50
1.43
0.458
1.35
1.81
TABLE V: Ablation on the temporal window size k : refined pose quality on DexYCB (gSDF + VOKE).
Fig. 4: Online VOKE predictions during real-world deployment of the simulation-trained kinematics-augmented policy. The three left columns show consecutive input frames, and the two right columns visualize the predicted linear and angular velocity and acceleration. Blue arrows denote estimated velocity and red arrows denote estimated acceleration.
Source
Object-state input
Success rate ↑
Rot. err. ↓
Sim. GT
History Pt−2:tgt
97.20
5.25
Complete Ktgt=(Ptgt,Vtgt)
98.10
3.69
VOKE pred.
History P^t−2:t
68.20
12.75
Complete K^t=(P^t,V^t)
78.90
8.90
TABLE VI: Closed-loop dexterous in-hand cube reorientation evaluation with oracle and visually estimated object-state inputs. Success rate is reported in percent, and final rotation error in degrees. Each condition contains 1,000 paired held-out episodes (five training seeds and 200 episodes per seed). Results are averaged across the five training seeds.