TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects

Fengguan Li; Yifan Ma; Chen Qian; Wentao Rao; Weiwei Shang

cs.ROMar 14, 2026

TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects

Authors: Fengguan Li, Yifan Ma, Chen Qian, Wentao Rao, Weiwei Shang

Organizations: University of Science and Technology of China

Abstract

Dexterous manipulation enables complex tasks but suffers from self-occlusion, severe depth noise, and depth information loss when manipulating transparent objects. To solve this problem, this paper proposes TransDex, a 3D visuo-tactile fusion motor policy based on point cloud reconstruction pre-training. Specifically, we first propose a self-supervised point cloud reconstruction pre-training approach based on Transformer. This method accurately recovers the 3D structure of objects from interactive point clouds of dexterous hands, even when random noise and large-scale masking are added. Building on this, TransDex is constructed in which perceptual encoding adopts a fine-grained hierarchical scheme and multi-round attention mechanisms adaptively fuse features of the robotic arm and dexterous hand to enable differentiated motion prediction. Results from transparent object manipulation experiments conducted on a real robotic system demonstrate that TransDex outperforms existing baseline methods. Further analysis validates the generalization capabilities of TransDex and the effectiveness of its individual components.

Figures & tables

Fig. 1: The Overall Framework of the Pre-Training-Based Visual-Tactile Fusion Motor Policy, TransDex. Perceptual information enters the encoder through a fine-grained, hierarchical manner. The hand-object interaction point cloud utilizes a pre-trained encoder, followed by an attention fusion module and two policy heads to achieve feature integration and differentiated action prediction.

Fig. 2: Pre-Training Framework. First, noise is added to and masked from the original hand-object interaction point cloud to generate input data. Then, features are extracted using a pre-encoder and a Transformer encoder. Finally, the generated query points and a Transformer decoder are employed to accomplish the point cloud reconstruction task for the object.

Fig. 3: Robotic System Setup: 1 a 16-DOF dexterous hand, with array tactile sensors equipped on the fingertips and finger pads; 2 a 7-DOF robotic arm; 3 depth cameras; 4 experimental items; 5 a data glove; 6 a motion capture camera.

Fig. 4: Visualization of Policy’s rollout on Three Transparent Object Manipulation Tasks , including pouring, shaking, and rotating. The unseen objects used in the test, as well as the complex backgrounds and lighting conditions, are shown at the bottom.

Method	Pouring	Shaking	Rotating	Avg
3D-ViTac	0%	0%	0%	0%
DP3	10%	0%	50%	20%
3DTacDex-P	70%	50%	20%	47%
TransDex(Ours)	100%	80%	70%	83%

TABLE I: Success Rate of TransDex and Baseline Policies

Fig. 5: Visualization of Point Cloud Reconstruction Performance in Pre-Training Tasks and Real-World Transfer Reconstruction Outcomes.

Method	Pouring	Shaking	Rotating	Avg
Ours w/o Diff. Pred.	0%	0%	0%	0%
Ours w/o Tac	0%	0%	20%	7%
Ours w/o Pre-train	30%	30%	10%	23%
TransDex(Ours)	100%	80%	70%	83%

TABLE II: Success Rate of Ablation Study

Settings	Pouring	Shaking	Rotating	Avg
Complex Backgrounds	90%	80%	70%	80%
Complex Lighting	80%	80%	60%	73%
Unseen Objects	100%	90%	50%	80%

TABLE III: Success Rate of Generalization Study

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion
Sep 30, 2026Ziheng Xu, Yueyuan Chen, Xinyuan He +6Visuo-Tactile Representation Learning Generalization in Robotic Manipulation
Shanghai Jiao Tong University · Tongji University · The Chinese University of Hong Kong, Shenzhen
Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
Jun 9, 2026Beomjun Kim, Seong Hyeon Park, Seunghoon Sim +3Dexterous Manipulation Cross-Embodiment Robot Learning
KAIST
VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment
Sep 25, 2026Julien Poffet, Matthew Strong, Ankush Dhawan +5Robotic Manipulation Robot Imitation Learning
Department of Mechanical and Process Engineering (D-MAVT), ETH Zurich, Zurich, Switzerland. Work done as a visiting student at Stanford University. · Department of Computer Science, Stanford University, Stanford CA, USA. · Department of Mechanical Engineering, Stanford University, Stanford CA, USA. +2

Sep 30, 2026J/K move · Enter open · S save

OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion

Ziheng Xu, Yueyuan Chen, Xinyuan He, Guoxing Liu+5

Shanghai Jiao Tong University · Tongji University · The Chinese University of Hong Kong, Shenzhen

Visuo-Tactile Representation Learning Generalization in Robotic Manipulation Robotic Manipulation

Open paper AskPDF arXiv

Reliable dexterous manipulation requires continuous estimation of object geometry and hand-object contact throughout interaction. With egocentric sensing, however, the manipulating hand frequently occludes task-relevant object surfaces and contact regions, reducing the visual evidence available for state estimation and thereby making robust closed-loop control and generalization to unseen object geometries particularly challenging. To address this, we present OccluDex, a hierarchical 3D visuo-tactile representation learning framework that integrates global geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction. OccluDex adopts multi-scale masked autoencoding to progressively encode partial 3D geometry and fuses tactile contact tokens with high-level geometric features through cross-modal attention. The encoder is pretrained from synchronized human visuo-tactile demonstrations and transferred as a frozen perceptual backbone for downstream reinforcement learning. We evaluate OccluDex on a faucet rotation task, requiring one full clockwise handle revolution, and a tabletop object reorientation task, requiring a 180-degree tabletop object reorientation without toppling. In simulation experiments, OccluDex demonstrated 12.6% higher accuracy for unseen objects and 8.3% higher accuracy for previously seen objects than the strongest state-of-the-art baseline models. Physical experiments were further performed with a Shadow Hand to demonstrate successful zero-shot sim-to-real generalization on unseen physical objects. This results could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes vision.

Abstract

Robotic Manipulation

Explore similar work

OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment