cs.RONov 12, 2026

VOKE: Visual Object Kinematics Estimator for Dexterous In-Hand Manipulation

Authors: Shengjia Zhu, Wenqiang Xu, Lin Shao

Organizations: Shanghai Jiao Tong University. · National University of Singapore. · Fudan University.

Abstract

Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves. However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. VOKE contains two core perception modules and a downstream application study. First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain.

Figures & tables

Explore similar work

CardsList
  1. GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation

    Nov 1, 2026Arjun Bhardwaj, Mayank Mittal, Maximum Wilder-Smith +2Dexterous ManipulationGoal-Conditioned Robotic Manipulation

  2. DexOAK: Object-Aware Kinematic Retargeting

    Oct 19, 2026Federica Spinola, Robert Katzschmann, Cordelia SchmidMotion RetargetingHuman-to-Robot Motion Imitation

  3. V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos

    Jun 15, 2026Kaihan Chen, Yanming Shao, Haifeng Ji +2Cross-Embodiment Robot LearningRobotic Manipulation