In contact-rich manipulation, contact conditions can change before a visual policy updates its action plan. Recomputing the plan through the visual policy adds latency, while tactile feedback alone does not specify how the ongoing motion should be revised for the task. We introduce T-WLA, a world-language-action architecture that uses shared visual memory to guide both action generation and tactile correction. At each visual update, the visual branch integrates the current image, task instruction, and retrieved history into a visual memory that conditions base-action generation. The same memory is cached for a lightweight tactile editor, which combines it with the active action chunk and incoming touch to predict residual corrections to unexecuted actions without rerunning the visual branch. Future-image and future-pressure prediction provide training-only supervision, encouraging the visual memory and tactile features to capture interaction dynamics relevant to action generation and correction. We evaluate T-WLA on tactile-conditioned hand-motion prediction using EgoTouch and Tachin, and on four real-world dexterous manipulation tasks. T-WLA reduces finger-joint MPJPE by 30% relative to the strongest reported baseline and achieves an average success rate of approximately 50.0% across real-robot manipulation tasks. Videos and code are available at https://anonymous-submission-20.github.io/T-WLA.github.io/.
Figures & tables
Figure 1: Tactile World–Language–Action (T-WLA). The visual branch uses current and past visual information to produce an action chunk and visual memory. Between visual updates, the tactile editor uses this cached visual memory and recent touch to correct the action chunk. During training, predicting future images and pressure helps the model learn features useful for control.
Figure 2: Shared memory connects slow visual planning with fast tactile correction. Each visual update generates a base action chunk and visual memory. Between updates, the tactile editor reads this cached pair and revises unexecuted actions without rerunning the visual branch.
Figure 3
Figure 3: The memory-to-editor interface. (a) Retrieved history is combined with the current observation and instruction. (b) Observed tactile history is encoded relative to a fixed reference. (c) Touch modulates action features conditioned on the cached visual memory, base chunk, and elapsed time. Module details are in Appendices B.1 – B.3 .
EgoTouch
Tachin
Method
Home
Workb.
Office
Retail
Outdoor
Avg.
Overall
Tactile Freq.
Wrist translation error (cm, ↓ )
VITRA + tactile
4.54
3.71
2.84
3.98
5.34
4.08
1.48
58.2
SmolVLA + tactile
4.55
3.74
2.88
3.98
5.33
4.10
1.50
69.0
MemoryVLA + tactile
4.65
3.78
2.96
4.08
5.41
4.18
1.55
15.0
T-Rex
4.63
3.76
2.95
4.05
5.41
4.16
1.54
17.16
Table 1: Tactile-conditioned future hand-motion prediction. EgoTouch is reported across five native environments and as a macro-average; Tachin is evaluated on its held-out test split. Metrics use the same prediction horizon and are averaged per sequence before aggregation. Lower prediction error is better. Gray highlights T-WLA. Bold / underline indicate the best/second-best distinct reported errors within each metric and column, including ties. Tactile Freq. denotes the effective tactile-correction rate under our multi-rate protocol. Frequency is reported in Hz and is not ranked.
Figure 4: Qualitative comparison on EgoTouch and Tachin. VITRA, SmolVLA, MemoryVLA, and T-Rex denote the tactile-augmented variants used in Table 1 . For each dataset, we show the source RGB frame at t0 , ground truth, and predictions from each of the models at t0+100ms . RGB overlays depict ground-truth hand poses in green and predictions in orange. Fingertip-detail views show wrist-aligned hands rendered from a +60∘ visualization view, with ground-truth hand mesh in green and predicted fingertips in orange crosses. MPJPE is reported in centimeters ( ↓ : lower is better). T-WLA obtains the lowest MPJPE on both displayed examples.
Controller
Egg Transfer
Single-Sheet Pickup
Paper-Cup Nesting
Paper-Bag Dragging
Avg.
SmolVLA
0
28
0
39
16.75
SmolVLA + Tactile
0
30
1
37
17.0
T-ReX
4
3
22
63
23.0
T-WLA w/o tactile
18
72
17
82
47.25
T-WLA
21
78
17
84
50.0
Table 2: Closed-loop success rates (%, ↑ ) on four contact-rich robot tasks after adaptation with 30 teleoperated demonstrations per task. Avg. denotes the unweighted mean across tasks. Bold / underline indicate the best/second-best distinct results in each column, including ties. Gray highlights tactile-enabled T-WLA.
Figure 5: Representative real-robot trials for cup stack and egg transfer. Each column shows the start, contact, and goal frames of one episode; the left column of each task is a successful T-WLA trial and the right column is a failed baseline trial.
EgoTouch
Tachin
Variant
Trans. (cm)
Rot. ( ∘ )
Joint (cm)
Trans. (cm)
Rot. ( ∘ )
Joint (cm)
Without tactile input
4.010
6.200
0.430
1.450
4.400
0.500
Without recurrent memory
4.070
6.280
0.370
1.690
4.560
0.490
Shorter memory ( Kshort<K )
3.980
6.150
0.360
1.480
4.310
0.460
Without world prediction (T+V)
3.963
6.113
0.353
1.413
4.283
0.463
Without world prediction (T)
3.962
6.112
0.352
1.412
4.263
0.462
Table 3: Ablation of tactile input, memory, and predictive supervision. World-prediction ablations remove auxiliary objectives during editor training, not backbone pretraining. Lower is better for every metric. Bold / underline indicate the best/second-best distinct results in each column.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Left hand
Right hand
Channels per hand
Wrist translation increment
[0,3)
[69,72)
3
Wrist Euler rotation increment
[3,6)
[72,75)
3
Palm-local joint coordinates
[6,69)
[75,138)
63
Padding
[138,192)
—
Appendix
Table 4: Hand-motion action layout. Indices are zero-based, half-open intervals. Joint channels store 21 three-dimensional local positions per hand.
GigaHands
OakInk2
TACO
Method
Cook.
Office
Craft.
Entert.
Housew.
Avg.
Overall
Overall
Freq.
Wrist translation error (cm, ↓ )
VITRA
1.16
1.25
1.11
1.44
1.37
1.27
0.53
0.97
19.4
SmolVLA
0.84
0.81
0.73
1.01
0.98
0.87
0.50
0.72
23.0
MemoryVLA
1.32
1.26
1.23
1.35
1.35
1.30
1.05
1.20
8.8
T-WLA
0.63
0.60
0.56
0.69
0.74
0.65
0.44
0.75
39.7
Appendix
Table 5: Visual-only future hand-motion prediction. GigaHands uses its five official activity families; OakInk2 and TACO use their complete held-out test sets. The GigaHands average gives equal weight to the five families. Lower prediction error is better. Gray highlights T-WLA. Bold / underline indicate the best/second-best distinct reported errors within each metric and column, including ties. Frequency is reported in Hz and is not ranked. Dashes indicate unavailable results and are excluded from ranking.
Figure 6: Robot platform and data glove
Figure 7: Additional camera-frame qualitative comparisons on EgoTouch. Each row shows the source RGB observation at t0 , followed by the ground-truth hand pose and predictions from VITRA, SmolVLA, MemoryVLA, T-Rex, and T-WLA at t0+100ms . Ground truth is shown in green and model predictions in orange. Values below each prediction report the corresponding wrist-aligned finger-joint MPJPE in centimeters.
Figure 8: Additional wrist-aligned fingertip comparisons on EgoTouch. The same six examples from Fig. 7 are shown after wrist alignment to emphasize finger articulation independently of global wrist motion. Within each example, all hands share the same alignment, scale, and +60∘ visualization rotation. The ground-truth hand mesh is shown in green, with ground-truth fingertips marked by green circles and predicted fingertips marked by orange crosses. Values below each prediction report wrist-aligned MPJPE over the 20 non-root hand joints at t0+100ms . These values are per-example diagnostics rather than dataset-level test averages.
Figure 9: Additional camera-frame qualitative comparisons on Tachin. Each row shows one source RGB observation at t0 , followed by the ground-truth hand pose and predictions from VITRA, SmolVLA, MemoryVLA, T-Rex, and T-WLA at t0+100ms . Ground-truth hand poses are shown in green and model predictions in orange. This camera-frame visualization preserves both global wrist motion and finger articulation. The values below each prediction report the corresponding wrist-aligned finger-joint MPJPE in centimeters.
Figure 10: Additional wrist-aligned fingertip comparisons on Tachin. The same six examples from Fig. 9 are shown after wrist alignment to emphasize finger articulation independently of global wrist motion. Within each example, all hands share the same alignment, scale, and +60∘ visualization rotation. The ground-truth hand mesh is shown in green, with ground-truth fingertips marked by green circles and predicted fingertips marked by orange crosses. Values below each prediction report wrist-aligned MPJPE over the 20 non-root hand joints at t0+100ms . These values are per-example diagnostics rather than dataset-level test averages.
Figure 11: Additional real-robot qualitative examples. Selected frames from paper-bag dragging and single-sheet pickup. Both tasks involve grasping thin, deformable edges, requiring sustained contact during dragging or a delicate pinch during lifting.
Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.
Shuai Tian, Yupeng Zheng, Yuhang Zheng +7
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · TARS Robotics +2
Tactile-enhanced vision-language-action (VLA) policies have been introduced for contact-rich manipulation, where critical interaction states are often hidden from vision. Future tactile prediction is a promising way to use touch because it turns tactile outcomes into supervision for action-induced contact dynamics. Yet VLA policies contain representations with different roles, from perceptual encoding to motor prediction, making it unclear where this supervision should be applied. We study this as a representation-alignment problem. Through a linear probe analysis, we find that future tactile states are most predictable from intermediate action-expert features, rather than from vision-language features or final action states. Motivated by this observation, we introduce a lightweight Latent Tactile Predictor (LTP), which predicts compact future tactile embeddings from the identified intermediate representation. By avoiding direct prediction of noisy raw tactile signals, LTP provides an action-outcome grounding signal that aligns intermediate action representations with future contact consequences. Experiments on real-world contact-rich manipulation tasks show that representation-aligned tactile grounding outperforms less aligned or multi-interface tactile prediction, highlighting the importance of where tactile supervision is applied.
Ruilin Chen, Jingkai Jia, Tong Yang +8
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · TeleAl, China Telecom +2
Dexterous manipulation in everyday environments requires both anticipation and reaction: a robot must predict how contact should evolve while rapidly correcting local errors caused by slip, misalignment, unstable grasping, or force mismatch. Vision and language provide semantic and geometric guidance, but they cannot reliably reveal hidden contact states such as force, slip, and contact stability. Although tactile sensing exposes these physical cues, most existing policies treat touch as a low-frequency observation stream within a monolithic action model, coupling slow task reasoning, action generation, and fast contact feedback in a single loop. We introduce TouchWorld, a predictive-and-reactive tactile foundation model for dexterous manipulation. TouchWorld uses a hierarchical policy that separates vision-language subtask planning, tactile world-model prediction, visuo-tactile goal-conditioned action generation, and high-frequency tactile residual refinement. A High-Level Planning Layer produces executable subtasks and predicts tactile subgoals; a Visuo-Tactile Goal-Conditioned Policy generates nominal action chunks; and a Tactile-Conditioned Refinement Policy performs online residual correction using recent tactile and proprioceptive feedback. By using touch as both a predictive contact reference and a fast feedback signal, TouchWorld preserves the semantic generalization of vision-language-action policies while improving local contact adaptation. Across six long-horizon and contact-rich dexterous manipulation tasks, TouchWorld achieves 65.0% success in the clean setting and 53.7% success under human perturbations, outperforming the strongest baseline by 15.7 and 18.5 percentage points, respectively.
Jianyi Zhou, Feiyang Hong, Yunhao Li +9
Harbin Institute of Technology, Shenzhen · PHANES AI