Dexterous manipulation with world action models (WAMs) relies on hand configuration and contact information, yet joint-state and whole-hand tactile observations may be partially or entirely missing. RGB images provide scene context and hand-object geometry but are insufficient to reliably recover these missing modalities. We further observe a correlation between hand state and tactile information. Motivated by this observation, we introduce BAT-WAM, a framework that bridges visual, joint-state, and tactile representations for WAMs. BAT-WAM learns separate codebooks and decoders for the tactile and action streams. An RGB-conditioned continuous latent bridge connects the two streams for bidirectional learning, while visual information serves as continuous context. Under the HTBench evaluation protocol, BAT-WAM reduces the root mean squared error (RMSE) of tactile reconstruction by 34.73% under fully masked conditions and 38.84% under partially masked conditions, relative to an image-conditioned HandTouch checkpoint. For future tactile prediction conditioned on target-time RGB and historical actions, it reduces mean RMSE by 85.56% relative to OpenWAM with HandTouch. Real-robot experiments further show that BAT-WAM consistently outperforms related WAM baselines in dexterous manipulation. Generalization experiments on unseen objects and scenes also demonstrate substantial improvements over baseline models. These findings support joint modeling of visual context, hand state, and touch as a representation approach for multimodal WAMs.
Figures & tables
Figure 1: Overview. BAT-WAM bridges state and contact information with RGB observation.
Figure 2: Pipeline of BAT-WAM.Overview of the BAT-WAM pipeline. A shared bridge Transformer integrates separately tokenized touch and joint states with continuous RGB features. Masked cross-modal reconstruction and temporal prediction train the representation interface, whose outputs, together with language instructions, condition a WAM backbone for action generation.
Available inputs
Model
Full touch
Image
Action
F-RMSE ↓
F-cIoU ↑
HandTouch
–
✓
–
0.0221
0.8316
BAT-WAM
–
✓
–
0.0163
0.8828
BAT-WAM
–
–
✓
0.0159
0.8836
BAT-WAM
–
✓
✓
0.0145
0.8933
Table 1: Full-mask touch completion.
Available inputs
Model
Partial touch
Image
Action
H-RMSE ↓
H-cIoU ↑
mean fill
✓
–
–
0.0955
0.3565
zero fill
✓
–
–
0.1221
0.0000
HTBench
✓
✓
–
0.0224
0.8211
BAT-WAM
✓
✓
–
0.0146
0.8925
BAT-WAM
✓
–
✓
0.0143
0.8934
Table 2: Local-mask touch completion.
Figure 3: Under HTBench, BAT-WAM is the best on both displayed metrics for the Action+Image condition.
Available inputs
Model
Touch
Image
Action
RMSE ↓
cIoU ↑
HandTouch
✓
✓
–
0.0211
0.8486
BAT-WAM
✓
✓
✓
0.0157
0.8901
Table 3: Future tactile prediction.
Method
Pick & Place
Fine Manipulation
Pull & Push
Long Horizon
Average
Pick
Pick
Pick
Stack
Open
Close
Wipe
Blue Cube
White Flange
Black Flange
Blue Cube
Drawer
Drawer
Whiteboard
OpenWAM- α
80.0
55.0
60.0
70.0
85.0
100.0
15.0
66.4
OpenWAM- α + HandTouch
75.0
65.0
65.0
75.0
85.0
95.0
25.0
69.3
BAT-WAM
90.0
75.0
70.0
85.0
95.0
95.0
45.0
79.3
Table 4: Real-world success rates (%) across different tasks.
Method
Color
Size
Shape
Layout
Pose
Unseen Avg.
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
OpenWAM- α
77.5
67.5
80.0
20.0
55.0
10.0
70.0
57.5
65.0
37.5
38.5
OpenWAM- α + HandTouch
75.0
70.0
80.0
27.5
60.0
15.0
67.5
60.0
70.0
37.5
42.0
BAT-WAM
87.5
80.0
87.5
37.5
72.5
27.5
85.0
80.0
77.5
47.5
54.5
Table 5: Zero-shot success rates (%) across different tasks.
Figure 4: Real-world experiment environment and representative task examples. The left column shows two views of the shared tabletop setup.