OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion

Ziheng Xu; Yueyuan Chen; Xinyuan He; Guoxing Liu; Yuanshuo Tan; Huiming Pan; Bin He; Shuo Jiang

cs.ROSep 30, 2026

OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion

Authors: Ziheng Xu, Yueyuan Chen, Xinyuan He, Guoxing Liu, Yuanshuo Tan, Huiming Pan, Bin He, Shuo Jiang, +1 more

Organizations: Shanghai Jiao Tong University · Tongji University · The Chinese University of Hong Kong, Shenzhen

Abstract

Reliable dexterous manipulation requires continuous estimation of object geometry and hand-object contact throughout interaction. With egocentric sensing, however, the manipulating hand frequently occludes task-relevant object surfaces and contact regions, reducing the visual evidence available for state estimation and thereby making robust closed-loop control and generalization to unseen object geometries particularly challenging. To address this, we present OccluDex, a hierarchical 3D visuo-tactile representation learning framework that integrates global geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction. OccluDex adopts multi-scale masked autoencoding to progressively encode partial 3D geometry and fuses tactile contact tokens with high-level geometric features through cross-modal attention. The encoder is pretrained from synchronized human visuo-tactile demonstrations and transferred as a frozen perceptual backbone for downstream reinforcement learning. We evaluate OccluDex on a faucet rotation task, requiring one full clockwise handle revolution, and a tabletop object reorientation task, requiring a 180-degree tabletop object reorientation without toppling. In simulation experiments, OccluDex demonstrated 12.6% higher accuracy for unseen objects and 8.3% higher accuracy for previously seen objects than the strongest state-of-the-art baseline models. Physical experiments were further performed with a Shadow Hand to demonstrate successful zero-shot sim-to-real generalization on unseen physical objects. This results could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes vision.

Figures & tables

Fig. 1: Simulation and real-world evaluation of OccluDex. In simulation (left), Faucet Rotation uses three faucet geometries for seen evaluation and two unseen scales for generalization testing, while Tabletop Object Reorientation uses ten seen and six unseen YCB objects. In real-world experiments (right), two previously unseen physical objects are evaluated for each task.

Fig. 2: Overview of the proposed 3D visuo–tactile representation learning framework. (a) Multi-scale grouping and masking. (b) A hierarchical encoder progressively abstracts 3D geometry and performs high-level cross-modal interaction. (c) The decoder reconstructs masked geometric and tactile observations through hierarchical token propagation. (d) The pretrained encoder is frozen and transferred to downstream PPO-based dexterous manipulation.

Fig. 3: Hierarchical multi-scale grouping and progressive masking. Multi-scale geometric regions are constructed through recursive sampling and neighborhood grouping, while the visibility pattern is propagated from the highest semantic scale to finer scales through hierarchical correspondences, preserving structural consistency across the geometric hierarchy.

Parameter	Value
Tactile threshold	40
Point-cloud mask ratio	0.8
Tactile mask ratio	0.5
Group sizes	$[16,\,8,\,8]$
Number of groups	$[512,\,256,\,64]$
Encoder dimensions	$[96,\,192,\,384]$

TABLE I: Main hyperparameters used for OccluDex pre-training.

Parameter	Tabletop Object Reorientation	Faucet Rotation
Parallel environments	200	150
Episode length	600	500
Camera position	$(0,\,0.25,\,0.5)$	$(0.25,\,0,\,0.5)$
Camera target	$(0,\,-0.1,\,-0.2)$	$(-0.1,\,0,\,-0.2)$

TABLE II: Environment configurations for the downstream manipulation tasks.

Fig. 4: Training curves on Faucet Rotation and Tabletop Object Reorientation. The top row shows success rate, and the bottom row shows mean reward per step. Solid lines denote the mean over three random seeds, and shaded regions indicate the corresponding standard deviation.

Task	Method	Seen			Unseen
Task	Method	Success (%)	Return	Ep. Len.	Success (%)	Return	Ep. Len.
Faucet Rotation	Base-only	$83.8\pm 23.9$	$842.8\pm 168.9$	$292.0\pm 120.3$	$80.6\pm 14.9$	$810.4\pm 129.3$	$318.6\pm 106.5$
	VTDexManip	$86.3\pm 20.5$	$743.3\pm 110.1$	$238.3\pm 96.6$	$63.0\pm 25.6$	$714.9\pm 56.6$	$339.5\pm 85.9$
	VTT-3D	$69.7\pm 29.5$	$896.5\pm 133.6$	$340.5\pm 112.9$	$48.8\pm 26.9$	$813.1\pm 52.7$	$404.8\pm 77.1$
	OccluDex	$\mathbf{99.9\pm 0.2}$	$643.2\pm 39.3$	$\mathbf{157.9\pm 16.9}$	$\mathbf{86.6\pm 2.9}$	$636.3\pm 25.9$	$\mathbf{229.8\pm 7.1}$
Tabletop Object Reorientation	Base-only	$65.1\pm 5.7$	$573.2\pm 118.9$	$77.1\pm 12.4$	$62.7\pm 1.6$	$554.9\pm 154.3$	$84.2\pm 23.6$

TABLE III: Evaluation results on Faucet Rotation and Reorient Object on Table. Values are mean

\pm

standard deviation over three random seeds.

Task	Variant	Seen (%)	Unseen (%)
Faucet Rotation	OccluDex (full)	$\mathbf{100.0}$	$\mathbf{83.7}$
	w/o point cloud	$75.7$	$77.0$
	w/o tactile	$54.3$	$35.7$
	From scratch (w/o hierarchy)	$98.7$	$76.8$
Tabletop Object Reorientation	OccluDex (full)	$\mathbf{77.3}$	$\mathbf{81.3}$
	w/o point cloud	$67.7$	$76.7$

TABLE IV: Ablation study. Values denote success rates (%).

Fig. 5: Real-world experimental setup. (a) Distribution of the tactile sensors on the Shadow Dexterous Hand. The sensors are attached to the fingertips, finger links, and palm to provide contact observations during manipulation. (b) Overview of the real-world manipulation workspace, including the robotic arm, the tactile-instrumented Shadow Hand, the overhead RealSense D435i camera, and the corresponding egocentric camera observation used for point-cloud perception.

Explore similar work

Jun 17, 2026cs.RO

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

Establishing a universal benchmark for tactile representation learning in robotic manipulation remains challenging due to the diversity of tactile sensor designs, data formats, and robot embodiments. Rather than seeking to establish such, we explore a scalable and promising direction for future development: egocentric vision paired with full-hand tactile data. To this end, we introduce \textbf{HT-Bench}, a large-scale multi-task benchmark for dexterous full-hand tactile sensing, comprising 10M RGB frames and 7.8M tactile frames collected across 226 tasks. HT-Bench evaluates tactile representations from three key perspectives: whether they encode meaningful contact geometry, whether they can align tactile observations with visual information, and whether they generalize to unseen tasks. To assess these capabilities, HT-Bench includes four tasks: fine-grained tactile similarity retrieval, masked tactile inpainting, vision-to-tactile synthesis, and multimodal tactile frame prediction. We further propose \textbf{HandTouch}, a vector-quantized vision--tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training. Across HT-Bench, HandTouch consistently outperforms representative tactile encoder baselines, improving Recall@5 on fine-grained tactile similarity retrieval from 74.65% to 85.23%, reducing RMSE on masked tactile inpainting from 0.022 to 0.010, and increasing OOD cIoU on vision-to-tactile synthesis from 0.628 to 0.705. These results demonstrate the effectiveness of HandTouch and suggest that large-scale egocentric full-hand tactile data provides a scalable basis for evaluating and advancing tactile representation learning in dexterous manipulation.

Yuzhe Huang, Jiaping Wu, Jiaming Jiang +6

Beihang University · Rimbot · BUPT +3

Tactile Finger Egocentric Vision

Ask PDF

Jun 7, 2026cs.RO

RGB-S: Image-Aligned Tactile Saliency for Robust Dexterous Manipulation

Effective visuo-tactile integration is critical for robotic dexterous manipulation, especially when visual observations are unreliable or occluded. However, robustly aligning sparse, heterogeneous tactile measurements with dense visual representations remains a fundamental challenge. Most existing approaches require policies to learn cross-modal correspondences implicitly from limited demonstrations, without leveraging geometric priors. As a result, they are often data-inefficient and generalize poorly when visual observations are degraded. To address this limitation, we propose a framework that explicitly grounds physical contacts in the image domain. Using robot forward kinematics and camera calibration, we project tactile sensor locations directly onto the RGB image plane. We then render force-modulated Gaussian saliency maps to model spatial uncertainty arising from kinematic and calibration errors. By integrating these 2D spatial anchors through a zero-initialized conditioning architecture, our method injects physical contact priors into standard visual backbones while preserving pre-trained visual representations. We evaluate our method on six dexterous manipulation tasks in both simulation and the real world under severe visual occlusions. Real-world experiments show that explicit RGB-S grounding in the image domain improves real-world occluded manipulation success rates by

26.7

percentage points over the strongest implicit visuo-tactile baseline, suggesting its improved spatial reasoning and robustness to occlusion. Project page: touch-as-saliency.github.io

Shengcheng Luo, Kefei Wu, Xiaoying Zhou +3

ShanghaiTech University · Beijing Institute for General Artificial Intelligence

Tactile Contact-Rich Dexterous Manipulation Saliency Geometric Priors

Ask PDF

Jun 12, 2025cs.CV

Occlusion-Aware 3D Hand-Object Pose Estimation with Masked AutoEncoders

Hand-object pose estimation from monocular RGB images remains a significant challenge mainly due to the severe occlusions inherent in hand-object interactions. Existing methods do not sufficiently explore global structural perception and reasoning, which limits their effectiveness in handling occluded hand-object interactions. To address this challenge, we propose an occlusion-aware hand-object pose estimation method based on masked autoencoders, termed as HOMAE. Specifically, we propose a target-focused masking strategy that imposes structured occlusion on regions of hand-object interaction, encouraging the model to learn context-aware features and reason about the occluded structures. We further integrate multi-scale features extracted from the decoder to predict a signed distance field (SDF), capturing both global context and fine-grained geometry. To enhance geometric perception, we combine the implicit SDF with an explicit point cloud derived from the SDF, leveraging the complementary strengths of both representations. This fusion enables more robust handling of occluded regions by combining the global context from the SDF with the precise local geometry provided by the point cloud. Extensive experiments on challenging DexYCB and HO3Dv2 benchmarks demonstrate that HOMAE achieves state-of-the-art performance in hand-object pose estimation. We will release our code and model.

Hui Yang, Wei Sun, Jian Liu +3

National Engineering Research Center for Robot Visual Perception and Control Technology, School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, China · School of Architecture and Art, Central South University, Changsha, 410082, China · Department of Computer Science and Software Engineering, The University of Western Australia, WA 6009, Australia

3D Hand 3D Object Detection Autoencoder Architectures House

Ask PDF