Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90 percent on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.
Figures & tables
Figure 1: VTAM Overview. A pretrained video backbone jointly models multi-view visual and tactile latents via alternating intra-view and cross-view attention. The resulting multimodal representation is injected into a conditional action diffusion head to predict action, virtual force, and proprioceptive state.
Figure 2: Experiment setup and data acquisition. We collect demonstrations through manual teleoperation using a visuo–tactile sensing setup for contact-rich manipulation tasks such as chip pick-and-place.
Model
Chip
Peel
Wipe
Genie Envisioner
0%
0%
2.5%
π0.5 (Vision)
10%
0%
0%
π0.5 + Tactile
5%
0%
0%
VTAM (Ours)
90%
85%
95%
Table 1: Overall performance comparison.
Figure 3: Qualitative comparison between VTAM and baseline methods on real-world manipulation tasks. Top: Chip pick-and-place . Vision-only baselines fail to determine whether the chip has been successfully grasped and proceed to the placement stage even when the grasp fails. Middle: Cucumber peeling into strips . Baselines tend to follow a vision-driven trajectory that approaches the center of the cucumber but fail to maintain consistent contact with the surface, indicating poor force regulation and lack of contact awareness. Bottom: Whiteboard wiping under varying heights and tilt angles . Baselines exhibit unstable wiping behaviors, often applying either insufficient or excessively large forces, particularly on tilted surfaces. In contrast, VTAM maintains stable contact and appropriate force regulation across all tasks, enabling robust manipulation behaviors. Red boxes highlight representative failure cases of baselines.
Figure 4: Prediction visualization of the backbone video model. From top to bottom: Camera-1 view, Camera-2 view, Tactile stream prediction. Ground-truth (top rows) and VTAM predictions (bottom rows).
Model Variant
Tactile Integration
Success Rate
Vision-only (No Tactile)
None
0%
Late-Fusion Tactile
Downstream Only
0%
No Virtual-Force Reg.
Joint Latent
10%
VTAM (Ours)
Hierarchical World Model
90%
Table 2: Ablation study on the Chip Pick-and-Place task (10 trials per variant).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Qualitative peeling results. VTAM achieves an 85% success rate (17/20 trials), producing peel strips longer than 10 cm in successful runs.
Figure 6: Cucumber peeling video prediction (rear camera view). Top row: ground-truth; Bottom row: model predictions. The predicted frames closely match the ground-truth observations.
Figure 7: Cucumber peeling video prediction (front camera view). The model maintains consistency with the ground truth across the manipulation sequence.
Figure 8: Cucumber peeling tactile prediction. Tactile frames are predicted accurately. Yellow arrows visualize estimated forces for interpretation only.
Figure 9: Whiteboard wiping video prediction (rear camera view). Top row: ground-truth; Bottom row: model predictions. Predictions match the ground truth across the wiping motion.
Figure 10: Whiteboard wiping video prediction (front camera view). The model successfully reproduces the visual dynamics of the wiping interaction.
Figure 11: Whiteboard wiping tactile prediction. The predicted tactile frames capture the interaction between the wiper and the board surface.
Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.
Shuai Tian, Yupeng Zheng, Yuhang Zheng +7
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · TARS Robotics +2
We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into vision-language-action (VLA) models. While advances in VLAs have introduced robot policies that are both generalizable and semantically grounded, these models mainly rely on vision-based perception. Vision alone, however, cannot capture the complex interaction dynamics that occur during contact-rich manipulation, including contact forces, surface friction, compliance, and shear. While recent attempts to integrate tactile signals into VLA models often increase complexity through token concatenation or large-scale pretraining, the heavy computational demands of behaviour models necessitate lightweight fusion strategies. To address these challenges, TacFiLM outlines a post-training finetuning approach that conditions intermediate visual features on pretrained tactile representations using feature-wise linear modulation (FiLM). Experimental results on insertion and drawer opening tasks demonstrate consistent improvements in success rate, direct task performance, completion time, and force stability across both in-distribution and out-of-distribution tasks. Together, these results support our method as an effective approach to integrating tactile signals into VLA models, improving contact-rich manipulation behaviours. Project page: https://charliem7.github.io/projects/TacFilm/
Charlotte Morissette, Amin Abyaneh, Wei-Di Chang +5
McGill University, CAN · Mila - Québec AI Institute, CAN · NVIDIA, USA
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling. Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline. Ablations attribute the gain to modeling contact evolution as part of the predicted world state rather than tactile conditioning alone: removing tactile world modeling reduces the four-task mean from 74.7 to 26.6 while keeping the same tactile features and action expert. After four hours of tactile-encoder adaptation with a frozen pretrained vision VAE, our continual vision-to-touch learning extends the pretrained video model to touch using roughly 100 demonstrations per task without tactile midtraining, while retaining visual prediction quality within 0.5 dB of vision-only counterparts. The compressor retains 89.4% of pre-fusion contact recall while enabling 2.26x faster training and 1.29x faster inference. Together, these results show that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.
Haoran Yuan, Zekai Wang, Boning Shao +4
University of Illinois Urbana-Champaign · University of California, Berkeley · Northwestern University