In contact-rich dexterous manipulation, occlusion by the fingers and interacting objects obscures local geometry, preventing visual observations from fully capturing contact locations. Visually similar scenes may also differ subtly in the relative positions of interacting objects, leading to different contact states. We propose 3DTouch, a tactile-geometric conditioning approach for vision-language-action (VLA) models. 3DTouch constructs a shared 3D visuo-tactile representation by expressing RGB-D scene points and dense five-finger tactile measurements in the robot-base frame, preserving each taxel's spatial location and force vector. Hand-object and object-object relations derived from hand landmarks and pose-aligned reconstructed surfaces explicitly describe the relative arrangement of interacting entities. Together, these representations condition the VLA action expert on spatially localized contact forces and relative surface geometry, providing explicit cues for contact adjustment and object alignment. We evaluate 3DTouch on five real-world contact-rich dexterous manipulation tasks using a UR7e robot arm equipped with an XHand dexterous hand. 3DTouch improves the average success rate by 42 percentage points over the corresponding VLA baseline and by 20 percentage points over the strongest evaluated alternative. Ablation studies reveal the complementary benefits of scene geometry and spatial tactile feedback, with further gains from shared spatial grounding and explicit geometric relations.
Figures & tables
Figure 1: Overview of 3DTouch. Compared with the 2D-only VLA shown on the left, 3DTouch augments a pretrained vision-language-action policy with scene geometry, spatially grounded tactile feedback, and explicit geometric relations. Scene points and tactile measurements are represented in a shared robot-base coordinate frame, while hand–object and object–object relations describe task-relevant geometric configurations. These complementary representations jointly condition action generation, with the bottom panels illustrating five real-world dexterous manipulation tasks and their success-rate comparisons against baselines.
Figure 2: Pipeline of 3DTouch. (a) 3D Tactile Grounding (3DTG) back-projects RGB-D observations into scene point clouds and transforms taxel positions and force vectors into a shared robot-base frame using forward kinematics (FK). Separate encoders produce scene tokens and finger-level tactile tokens. (b) Geometric Relation Modeling (GRM) aligns reference object meshes using estimated poses and encodes relative displacements and distances between hand landmarks and nearby object surface points, and between representative regions of interacting objects, into hand–object (HO) and object–object (OO) tokens. The scene, tactile, and relation tokens jointly condition the action expert, while the original vision-language pathway and flow-matching objective are preserved.
Figure 3: Pose Estimator. SAM2 generates object segmentation masks, which RGBTrack uses together with RGB-D observations and reconstructed reference meshes to obtain offline 6D pose annotations. These annotations supervise a lightweight pose estimator that predicts object poses directly from RGB-D observations and sampled reference surface points.
Figure 4: Overview of five contact-rich manipulation tasks performed on the UR7e robot equipped with an XHand dexterous hand. Each column shows the start and end states of a task, with red dashed arrows indicating the intended manipulation motions.
Category
Method
Funnel Insertion
Plug Assembly
Button Press
Sponge Wiping
Pipette Press
Avg. ↑
Vision-based Policy
π0 [ 2 ]
0.0
0.0
45.0
60.0
10.0
23.0
π0.5 [ 1 ]
15.0
50.0
75.0
60.0
20.0
44.0
Point-cloud Policy
Simple DP3 + tactile [ 31 ]
10.0
0.0
10.0
20.0
10.0
10.0
PointACT [ 4 ]
0.0
20.0
15.0
20.0
10.0
13.0
Tactile-augmented Policy
π0 + tactile
0.0
15.0
45.0
70.0
25.0
31.0
π0.5 + tactile
20.0
55.0
30.0
70.0
40.0
43.0
Table 1: Success rates (%) on real-world contact-rich dexterous manipulation tasks. Each method is evaluated over 20 real-robot rollouts per task. Baselines are grouped into point-cloud policy (w or w/o tactile feedback), vision-based policy, and tactile-augmented policy.
Method
Funnel Insertion
Plug Assembly
Trans.
Rot.
Both
Trans.
Rot.
Both
π0.5 + tactile
10.0
0.0
0.0
55.0
50.0
30.0
3DTouch
30.0
30.0
30.0
55.0
60.0
50.0
Table 2: Robustness to target pose variations on Funnel Insertion and Plug Assembly. Trans., Rot., and Both denote translation, rotation, and their combination, respectively. Success rates (%) are reported over 20 real-robot trials per task and setting.
Variant
Input Configuration
Task Success Rate (%)
Tactile
Point Cloud
Interaction Geometry
Funnel Insertion
Plug Assembly
Button Press
Sponge Wiping
Pipette Press
Avg. ↑
3DTouch (Full)
Global
✓
✓
35.0
60.0
80.0
90.0
60.0
65.0
w/o Geometric Relations
Global
✓
×
15.0
60.0
55.0
80.0
55.0
53.0 ( −12.0 )
Local Tactile
Local
✓
×
15.0
40.0
25.0
90.0
40.0
42.0 ( −23.0 )
Point Cloud Only
×
✓
×
10.0
50.0
50.0
80.0
40.0
46.0 ( −19.0 )
Tactile Only
Global
×
×
10.0
50.0
30.0
60.0
25.0
35.0 ( −30.0 )
Table 3: Ablation study of 3DTouch across five real-world contact-rich dexterous manipulation tasks. We investigate the individual and combined contributions of spatial tactile sensing, scene point clouds, and interaction geometry, as well as tactile coordinate alignment. Global and Local denote robot-base-frame and hand-local tactile representations, respectively. Success rates (%) are reported. Values in red parentheses indicate changes in average success rate (percentage points) relative to the full model.
Figure 5: Qualitative Evaluation of Pose Estimation. Object surface points are transformed into the robot base frame using predicted poses (top) or reference poses from the offline SAM2–RGBTrack pipeline (bottom), and reprojected onto RGB images. Similar overlays demonstrate that our estimator produces object alignments consistent with the offline pose annotations.
Figure 6: Visualization of Geometric Relations. Hand–object (HO) and object–object (OO) relations constructed using object poses predicted by our pose estimator (top) and reference poses obtained from the offline SAM2–RGBTrack pipeline (bottom). The similar spatial configurations illustrate the consistency of the resulting interaction geometry.