VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets

Yubin Jeon; Uiseong Shin; Hwanchul La; Jaeseong Kang; Hyelim Choi; Yongseok Lee

cs.ROOct 4, 2026

VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets

Authors: Yubin Jeon, Uiseong Shin, Hwanchul La, Jaeseong Kang, Hyelim Choi, Yongseok Lee

Organizations: DGIST, Daegu 42988, Republic of Korea · Seoul National University, Republic of Korea

Abstract

Learning dexterous manipulation benefits from human demonstration datasets that capture diverse and natural hand-object interactions. In particular, contact points and forces provide supervision on where and how strongly to interact, which cannot be fully captured by motion trajectories alone. However, methods for jointly capturing hand and object motion, contact points, and forces remain limited. Moreover, severe occlusion from hand-object interaction challenges accurate tracking of both hands and objects. To address these limitations, we present a Visual-Inertial-CONtact based hand-object tracking (VICON) framework. It holistically captures both hand and object motion along with contact information during manipulation, even under severe occlusion. First, we adopt a visual-inertial glove and an RGB-D camera for accurate hand tracking, and redesign the glove to incorporate contact sensing. Specifically, force-sensitive resistors (FSRs) are placed on the glove based on human grasp frequency to synchronously record contact states and calibrated normal forces. Second, without requiring pre-existing CAD models, we estimate object poses using RGB-D images and a mesh reconstructed from a monocular video. We propose factor-graph-based object trajectory estimation that fuses object-pose estimates weighted by visibility under hand-object occlusion, FSR measurements, and a hand-motion prior. Across 40 motion-capture sessions with five objects, VICON achieves a 2.5% failed-frame rate compared with 50.9-64.6% for the baselines, with median errors of 3.9 mm and 3.0 degrees under occlusion. Using VICON, we construct a dataset containing synchronized hand-object motion, contact points, and normal forces, and will publicly release an expanded dataset covering 10 object categories at https://github.com/VICON-dataset/dataset.

Figures & tables

Fig. 1: VICON records hand motion, object pose, contact points, and forces using a visual-inertial glove [ 1 ] with integrated FSRs and a single RGB-D camera. The glove tracks the hand skeleton, and our offline refinement maintains robust and accurate object 6-DoF tracking even under severe occlusion. Inset: object-frame contact points, with arrows colored by calibrated normal force magnitude.

Dataset	Hand	Object	Contact	Force/	Setup
	pose	pose	format	tactile
EgoDex [ 2 ]	$\checkmark$	–	–	–	Vision Pro
ContactPose [ 6 ]	$\checkmark$	$\checkmark$	Object surface map	–	3 RGB-D, thermal, mocap
ARCTIC [ 7 ]	$\checkmark$	$\checkmark$	Object surface map	–	8+1 RGB, mocap
OpenTouch [ 9 ]	$\checkmark$	–	Hand map	Pressure	Aria glasses, tracking and tactile gloves
TactiDex [ 10 ]	$\checkmark$	$\checkmark$	Hand map	Pressure	Mocap, tracking and tactile gloves

TABLE I: Comparison of our dataset with existing interaction datasets

Panel of Figure 3 — Fig. 2: Multi-layer VICON-glove. (a) Placement of the 14 IMUs in the inner glove. (b) Outer glove with the visual markers. (c) Hand contact-frequency map [ 17 ] , averaged over the mechanic and housemaid tasks (values in %), used to choose the FSR sites. (d) Palmar view of the eight FSRs.

Fig. 3: VICON on a recorded session. The VICON-glove provides hand tracking (poses $\mathbf{H}_{1}$ – $\mathbf{H}_{5}$ , increments $\Delta H$ ) and contact (FSR); the RGB-D camera feeds FoundationPose, whose candidates enter as $f_{\mathrm{vis}}$ (filled: accepted; hollow: rejected under occlusion). Between object poses $X_{k}$ , contact selects $f_{\mathrm{stat}}$ (visible, after release; red crosses: rejected hand increments), $f_{\mathrm{pivot}}$ (2-point grasp), or $f_{\mathrm{hand}}$ (3-point grasp). Re-measurement feeds the refined poses back to FoundationPose.

$v_{d}$	0.10	0.15	0.30	0.50	0.70	0.90	1.0
$\sigma_{p}$ (mm)	6	5	3	3	2	1.2	0.8
$\sigma_{r}$ (deg)	180	180	8	3	2	1	0.8

TABLE II: Nominal visual-measurement uncertainty as a function of visibility

Panel of Figure 6 — Fig. 4: Experimental setup. (a) Motion-capture room; the inset shows one of the motion-capture cameras. (b) The five objects: block, mouse, mug, mustard bottle, and phone.

Fig. 5: Object-pose errors over the 40 motion-capture sessions (top: translation; bottom: symmetry-aware rotation). (a) Median error of each tracker binned by the visibility

v_{d}^{\mathrm{GT}}

; shading marks the occluded range. (b) PCF on the 24,285 occluded frames as a function of the error threshold; dotted lines mark the 20 mm and 15 ∘ thresholds used for

S_{T20}

and

S_{R15}

, and the numbers give the AUC. Medians are used in (a) because diverging baseline errors would dominate a mean.

Fig. 6: Qualitative results with increasing hand occlusion from top to bottom. Columns: object, 3DGS mesh, and the input image overlaid with the ground-truth pose (green) and the pose of FoundationPose, ICG+, and VICON. Last column: rotated view of the tracked hand, visualized as a MANO [ 27 ] mesh, and the VICON object pose with the contact points and force arrows of the recognized FSRs, colored by the calibrated normal force.

	Failed	Success rate				Median error
		Visible		Occluded		Visible		Occluded
Method	frames	$S_{T20}$	$S_{R15}$	$S_{T20}$	$S_{R15}$	$T$	$R$	$T$	$R$
	(%)	(%)	(%)	(%)	(%)	(mm)	(deg)	(mm)	(deg)
FoundationPose	50.9	63.6	62.5	48.8	39.6	4.0	2.7	21.8	33.9
MegaPose	62.6	79.3	64.5	26.2	15.3	4.4	5.9	45.7	82.9
ICG+	64.6	52.3	57.9	25.6	27.3	6.9	9.5	61.1	34.3

TABLE III: Comparison with baseline trackers

	Failed	Success rate				Median error
		Visible		Occluded		Visible		Occluded
Variant	frames	$S_{T20}$	$S_{R15}$	$S_{T20}$	$S_{R15}$	$T$	$R$	$T$	$R$
	(%)	(%)	(%)	(%)	(%)	(mm)	(deg)	(mm)	(deg)
w/o vis. weighting	28.9	93.7	84.8	74.1	63.3	3.0	1.9	5.9	6.3
w/o hand motion	18.1	99.9	97.5	73.7	78.4	1.0	1.8	5.2	5.2
w/o contact timing	23.9	99.7	92.9	85.4	65.8	1.4	1.9	7.6	9.4

TABLE IV: Component ablations

Explore similar work

ComPose: When to Trust Hands for Object Pose Tracking
May 22, 2026Jisu Shin, Junoh Lee, JunGyu Lee +5Visual Object Tracking Object Pose Estimation
GIST · Yonsei Univ. · DGIST
KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking
Jul 16, 2026Chanyoung Ahn, Jaesung Lee, Sungwoo Park +1Object Pose Estimation Robotic Manipulation
Center for Humanoid Research, KIST, Seoul, 02792 South Korea · Korea University, Seoul, 02841 South Korea
NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation
Jun 23, 2026Soham Patil, Avirup Das, Sourabh Bhosale +1Vision-Based Tactile Sensing Dexterous Manipulation
Robotics Research Center (RRC), International Institute of Information Technology (IIIT), Hyderabad, India · Department of Computer Science, The University of Manchester

May 22, 2026, cs.CVJ/K move · Enter open · S save

ComPose: When to Trust Hands for Object Pose Tracking

Jisu Shin, Junoh Lee, JunGyu Lee, Inhwan Bae+4

GIST · Yonsei Univ. · DGIST

Visual Object Tracking Object Pose Estimation

Open paper AskPDF arXiv

Reconstructing the motion of objects from videos is a key component for embodied AI and robot manipulation. While diverse approaches to object pose tracking have been studied, they rely heavily on strong external priors, such as depth data or 3D templates, and remain highly vulnerable to severe occlusions by hand grasps despite the use of explicit masks. In this work, we present ComPose, a 6DoF object tracking framework designed for hand-aware object pose estimation from RGB video. Rather than treating the hand purely as an occluder, our method harmonizes hand motions as a \textit{complementary cue} for object tracking. In detail, we recover a variety of object motions over time by combining object and hand cues from foundation models within a unified tracking pipeline. Here, ComPose adaptively selects informative hand joints, combines object- and hand-derived cues for motion estimation, and refines the resulting object motion using visible geometric evidence and a learned correction. We further enforce the temporal consistency over both rotation and translation, yielding stable 3D object trajectories over time without any external smoothing. Extensive experiments show that our method is accurate, efficient, and robust under severe hand occlusion and geometric ambiguity. In addition, the resulting trajectories can also effectively transfer to downstream robot manipulation by enabling robots to reconstruct human actions from online videos.

Abstract

Explore similar work

ComPose: When to Trust Hands for Object Pose Tracking

KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking

NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation