cs.ROOct 20, 2026
SaveT-WLA: Bridging Slow Vision and Fast Touch
Organizations: Massachusetts Institute of Technology · Harvard University · University of California, Davis · University of California, Berkeley
Abstract
In contact-rich manipulation, contact conditions can change before a visual policy updates its action plan. Recomputing the plan through the visual policy adds latency, while tactile feedback alone does not specify how the ongoing motion should be revised for the task. We introduce T-WLA, a world-language-action architecture that uses shared visual memory to guide both action generation and tactile correction. At each visual update, the visual branch integrates the current image, task instruction, and retrieved history into a visual memory that conditions base-action generation. The same memory is cached for a lightweight tactile editor, which combines it with the active action chunk and incoming touch to predict residual corrections to unexecuted actions without rerunning the visual branch. Future-image and future-pressure prediction provide training-only supervision, encouraging the visual memory and tactile features to capture interaction dynamics relevant to action generation and correction. We evaluate T-WLA on tactile-conditioned hand-motion prediction using EgoTouch and Tachin, and on four real-world dexterous manipulation tasks. T-WLA reduces finger-joint MPJPE by 30% relative to the strongest reported baseline and achieves an average success rate of approximately 50.0% across real-robot manipulation tasks. Videos and code are available at https://anonymous-submission-20.github.io/T-WLA.github.io/.
Figures & tables
Figure 1: Tactile World–Language–Action (T-WLA). The visual branch uses current and past visual information to produce an action chunk and visual memory. Between visual updates, the tactile editor uses this cached visual memory and recent touch to correct the action chunk. During training, predicting future images and pressure helps the model learn features useful for control.
Figure 2: Shared memory connects slow visual planning with fast tactile correction. Each visual update generates a base action chunk and visual memory. Between updates, the tactile editor reads this cached pair and revises unexecuted actions without rerunning the visual branch.
Figure 3: The memory-to-editor interface. (a) Retrieved history is combined with the current observation and instruction. (b) Observed tactile history is encoded relative to a fixed reference. (c) Touch modulates action features conditioned on the cached visual memory, base chunk, and elapsed time. Module details are in Appendices B.1 – B.3 .
| EgoTouch | Tachin | |||||||
| Method | Home | Workb. | Office | Retail | Outdoor | Avg. | Overall | Tactile Freq. |
| Wrist translation error (cm, ) | ||||||||
| VITRA + tactile | 4.54 | 3.71 | 2.84 | 3.98 | 5.34 | 4.08 | 1.48 | 58.2 |
| SmolVLA + tactile | 4.55 | 3.74 | 2.88 | 3.98 | 5.33 | 4.10 | 1.50 | 69.0 |
| MemoryVLA + tactile | 4.65 | 3.78 | 2.96 | 4.08 | 5.41 | 4.18 | 1.55 | 15.0 |
| T-Rex | 4.63 | 3.76 | 2.95 | 4.05 | 5.41 | 4.16 | 1.54 | 17.16 |
Table 1: Tactile-conditioned future hand-motion prediction. EgoTouch is reported across five native environments and as a macro-average; Tachin is evaluated on its held-out test split. Metrics use the same prediction horizon and are averaged per sequence before aggregation. Lower prediction error is better. Gray highlights T-WLA. Bold / underline indicate the best/second-best distinct reported errors within each metric and column, including ties. Tactile Freq. denotes the effective tactile-correction rate under our multi-rate protocol. Frequency is reported in Hz and is not ranked.
Figure 4: Qualitative comparison on EgoTouch and Tachin. VITRA, SmolVLA, MemoryVLA, and T-Rex denote the tactile-augmented variants used in Table 1 . For each dataset, we show the source RGB frame at , ground truth, and predictions from each of the models at . RGB overlays depict ground-truth hand poses in green and predictions in orange. Fingertip-detail views show wrist-aligned hands rendered from a visualization view, with ground-truth hand mesh in green and predicted fingertips in orange crosses. MPJPE is reported in centimeters ( : lower is better). T-WLA obtains the lowest MPJPE on both displayed examples.
| Controller | Egg Transfer | Single-Sheet Pickup | Paper-Cup Nesting | Paper-Bag Dragging | Avg. |
|---|---|---|---|---|---|
| SmolVLA | 0 | 28 | 0 | 39 | 16.75 |
| SmolVLA + Tactile | 0 | 30 | 1 | 37 | 17.0 |
| T-ReX | 4 | 3 | 22 | 63 | 23.0 |
| T-WLA w/o tactile | 18 | 72 | 17 | 82 | 47.25 |
| T-WLA | 21 | 78 | 17 | 84 | 50.0 |
Table 2: Closed-loop success rates (%, ) on four contact-rich robot tasks after adaptation with 30 teleoperated demonstrations per task. Avg. denotes the unweighted mean across tasks. Bold / underline indicate the best/second-best distinct results in each column, including ties. Gray highlights tactile-enabled T-WLA.
Figure 5: Representative real-robot trials for cup stack and egg transfer. Each column shows the start, contact, and goal frames of one episode; the left column of each task is a successful T-WLA trial and the right column is a failed baseline trial.
| EgoTouch | Tachin | |||||
|---|---|---|---|---|---|---|
| Variant | Trans. (cm) | Rot. ( ∘ ) | Joint (cm) | Trans. (cm) | Rot. ( ∘ ) | Joint (cm) |
| Without tactile input | 4.010 | 6.200 | 0.430 | 1.450 | 4.400 | 0.500 |
| Without recurrent memory | 4.070 | 6.280 | 0.370 | 1.690 | 4.560 | 0.490 |
| Shorter memory ( ) | 3.980 | 6.150 | 0.360 | 1.480 | 4.310 | 0.460 |
| Without world prediction (T+V) | 3.963 | 6.113 | 0.353 | 1.413 | 4.283 | 0.463 |
| Without world prediction (T) | 3.962 | 6.112 | 0.352 | 1.412 | 4.263 | 0.462 |
Table 3: Ablation of tactile input, memory, and predictive supervision. World-prediction ablations remove auxiliary objectives during editor training, not backbone pretraining. Lower is better for every metric. Bold / underline indicate the best/second-best distinct results in each column.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Left hand | Right hand | Channels per hand |
|---|---|---|---|
| Wrist translation increment | 3 | ||
| Wrist Euler rotation increment | 3 | ||
| Palm-local joint coordinates | 63 | ||
| Padding | — | ||
Table 4: Hand-motion action layout. Indices are zero-based, half-open intervals. Joint channels store 21 three-dimensional local positions per hand.
| GigaHands | OakInk2 | TACO | |||||||
| Method | Cook. | Office | Craft. | Entert. | Housew. | Avg. | Overall | Overall | Freq. |
| Wrist translation error (cm, ) | |||||||||
| VITRA | 1.16 | 1.25 | 1.11 | 1.44 | 1.37 | 1.27 | 0.53 | 0.97 | 19.4 |
| SmolVLA | 0.84 | 0.81 | 0.73 | 1.01 | 0.98 | 0.87 | 0.50 | 0.72 | 23.0 |
| MemoryVLA | 1.32 | 1.26 | 1.23 | 1.35 | 1.35 | 1.30 | 1.05 | 1.20 | 8.8 |
| T-WLA | 0.63 | 0.60 | 0.56 | 0.69 | 0.74 | 0.65 | 0.44 | 0.75 | 39.7 |
Table 5: Visual-only future hand-motion prediction. GigaHands uses its five official activity families; OakInk2 and TACO use their complete held-out test sets. The GigaHands average gives equal weight to the five families. Lower prediction error is better. Gray highlights T-WLA. Bold / underline indicate the best/second-best distinct reported errors within each metric and column, including ties. Frequency is reported in Hz and is not ranked. Dashes indicate unavailable results and are excluded from ranking.
Figure 6: Robot platform and data glove
Figure 7: Additional camera-frame qualitative comparisons on EgoTouch. Each row shows the source RGB observation at , followed by the ground-truth hand pose and predictions from VITRA, SmolVLA, MemoryVLA, T-Rex, and T-WLA at . Ground truth is shown in green and model predictions in orange. Values below each prediction report the corresponding wrist-aligned finger-joint MPJPE in centimeters.
Figure 8: Additional wrist-aligned fingertip comparisons on EgoTouch. The same six examples from Fig. 7 are shown after wrist alignment to emphasize finger articulation independently of global wrist motion. Within each example, all hands share the same alignment, scale, and visualization rotation. The ground-truth hand mesh is shown in green, with ground-truth fingertips marked by green circles and predicted fingertips marked by orange crosses. Values below each prediction report wrist-aligned MPJPE over the 20 non-root hand joints at . These values are per-example diagnostics rather than dataset-level test averages.
Figure 9: Additional camera-frame qualitative comparisons on Tachin. Each row shows one source RGB observation at , followed by the ground-truth hand pose and predictions from VITRA, SmolVLA, MemoryVLA, T-Rex, and T-WLA at . Ground-truth hand poses are shown in green and model predictions in orange. This camera-frame visualization preserves both global wrist motion and finger articulation. The values below each prediction report the corresponding wrist-aligned finger-joint MPJPE in centimeters.
Figure 10: Additional wrist-aligned fingertip comparisons on Tachin. The same six examples from Fig. 9 are shown after wrist alignment to emphasize finger articulation independently of global wrist motion. Within each example, all hands share the same alignment, scale, and visualization rotation. The ground-truth hand mesh is shown in green, with ground-truth fingertips marked by green circles and predicted fingertips marked by orange crosses. Values below each prediction report wrist-aligned MPJPE over the 20 non-root hand joints at . These values are per-example diagnostics rather than dataset-level test averages.
Figure 11: Additional real-robot qualitative examples. Selected frames from paper-bag dragging and single-sheet pickup. Both tasks involve grasping thin, deformable edges, requiring sustained contact during dragging or a delicate pinch during lifting.