OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Yuhang Zheng; Songen Gu; Yupeng Zheng; Weize Li; Yujie Zang; Shuai Tian; Xiang Li; Ce Hao

cs.ROMar 19, 2026

OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Authors: Yuhang Zheng, Songen Gu, Yupeng Zheng, Weize Li, Yujie Zang, Shuai Tian, Xiang Li, Ce Hao, +6 more

Organizations: National University of Singapore · Fudan University · CASIA · TARS Robotics · Tsinghua University · Zhongguancun Academy · Beihang University

Abstract

Contact-rich manipulation tasks, such as wiping and assembly, require accurate perception of contact forces, friction changes, and state transitions that cannot be reliably inferred from vision alone. Despite growing interest in visuo-tactile manipulation, progress is constrained by two persistent limitations: existing datasets are small in scale and narrow in task coverage, and current methods treat tactile signals as passive observations rather than using them to model contact dynamics or enable closed-loop control explicitly. In this paper, we present OmniViTac, a large-scale visuo-tactile-action dataset comprising $21{,}000+$ trajectories across $86$ tasks and $100+$ objects, organized into six physics-grounded interaction patterns. Building on this dataset, we propose OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules: a self-supervised tactile encoder, a two-stream visuo-tactile world model for predicting short-horizon contact evolution, a contact-aware fusion policy for action generation, and a 60Hz reflexive controller that corrects deviations between predicted and observed tactile signals in a closed loop. Real-robot experiments across all six interaction categories show that OmniVTA outperforms existing methods and generalizes well to unseen objects and geometric configurations, confirming the value of combining predictive contact modeling with high-frequency tactile feedback for contact-rich manipulation. All data, models, and code will be made publicly available on the project website at https://mrsecant.github.io/OmniVTA.

Figures & tables

Fig. 2: Overview of OmniViTac dataset. Left: The Cross-Embodiment Data Collection Platform features UNFactory 7-DoF xArm and the TacUMI manipulation interface, both supporting identical end-effectors and diverse tactile sensors (Xense, GelSight Mini, Tac3D, Daimon). Middle: The dataset covers 6 visuo-tactile manipulation patterns, instantiated across 5 semantic scenarios. Top-Right: A scale comparison demonstrates that OmniViTac (21,879 trajectories) significantly exceeds existing visuo-tactile manipulation datasets in tactile-rich data volume. Bottom-Right: The High-quality Data Pipeline ensures reliability through timestamp alignment, visualization, and human-in-the-loop verification.

Fig. 3: Example visualization of the $6$ visuo-tactile manipulation patterns in OmniViTac. The dataset captures diverse, contact-rich behaviors across various scenarios, including Assembly, Cutting, Adjustment, Peeling, Grasping, and Wiping. Each panel illustrates the global third-person workspace view overlaid with the end-effector’s trajectory, alongside synchronized, high-frequency tactile maps (bottom) that continuously record the complex tool-object contact dynamics during task execution.

Dataset	#Task	#Traj.	#Object	Vision	Tactile			Action	Collection
Dataset	#Task	#Traj.	#Object	Vision	Sensors	Force Dim.	Frequency	Action	Collection
ObjectFolder2.0 [ 20 ]	3	–	1,000	Rendered	GelSight	3	–	✗	Simulated
AnyTouch [ 18 ]	1	–	124	–	GelSight Mini/DIGIT/DuraGel/Tac3D	3	30	✗	Handheld
VTV150k [ 48 ]	–	–	100	–	GelSight Mini/DIGIT/Tac3D	–	–	✗	Handheld
Octopi-1.5 [ 59 ]	3	–	–	GoPro	GelSight Mini/TAC-02	–	50 $\sim$ 1000	✓	TMI
See, Hear, and Feel [ 30 ]	2	60	4	RealSense	GelSight	–	30	✓	Teleop

TABLE I: Comparison of OmniViTac with existing vision-tactile-action datasets. Note: * indicates that the statistics count only the visuo-tactile trajectories in the corresponding dataset. “–” denotes not applicable.

Fig. 4: Comprehensive statistical analysis of the OmniViTac. (a) Pattern-level contact area distribution, highlighting the distinct dichotomy between precision-dominant tasks (concentrated in the $0$ - $10\%$ range) and surface-dominant tasks (peaking at $70$ - $90\%$ ). (b) Force intensity distribution, showcasing the wide spectrum of force magnitudes required across different manipulation modes. (c) Hierarchical distribution illustrating the rich diversity of $86$ instantiated tasks. (d) Effective contact ratio with variance across patterns, demonstrating the temporal dependency on tactile feedback (e.g., Adjustment exhibits the highest continuous engagement). (e) Total trajectory counts per category, demonstrating the massive scale and balanced composition of the benchmark. (f) t-SNE projection of the high-dimensional tactile signals, revealing physically intuitive and semantically separable latent clusters that strictly align with the underlying contact mechanics of each interaction pattern.

Fig. 5: System Overview . OmniVTA is a hierarchical slow–fast policy for contact-rich manipulation. The slow policy contains a visuo-tactile world model and an adaptive fusion policy to generate long-horizon action chunks from multi-modal inputs. The fast policy outputs high-frequency refinements at $60$ Hz using tactile feedback. Final actions are a weighted summation of slow-planned and fast-refined outputs, enabling both long-horizon planning and reactive control for robust manipulation.

Fig. 6: Overview of TactileVAE. The model encodes 3D marker displacements into spatio-temporal features, and reconstructs the deformation via an implicit decoder that represents the deformation as a continuous field.

Fig. 7: Overview of the Slow Policy. (a): The proposed Visuo-Tactile World Model takes noisy tactile and visual latent sequences as input and jointly generates both modalities via a two-stream spatio-temporal diffusion transformer. (b): The Adaptive Visuo-Tactile Fusion Policy employs a Latent Tactile Differential Encoder to effectively encode the observed and predicted tactile and a gating mechanism to adaptively balance tactile and visual information for robust action planning.

Fig. 8: Overview of the Reflexive Latent Tactile Controller (RLTC). It takes single-frame tactile feedback, chunk-aligned predicted tactile latents, and robot/gripper delta states as input to produce high-frequency (60 Hz) refined single-step corrective actions.

Task	Object Used for Training	Object Used for Manipulation
Wipe	vases in 4 different colors and shapes, plate, and whiteboard	vases in 4 different colors and shapes
Peel	cucumber, Chinese yam, radish, carrot, and white daikon	cucumber, Chinese yam, radish
Cut	cucumber, Chinese yam, carrot, pepper, and banana	cucumber, Chinese yam, pepper, banana
Assembly	3 colors of USB stick and 3 types of charging adapters	red USB stick and silver USB stick
Grasp	blueberry, strawberry, grape, cherry, and cherry tomato	blueberry, strawberry, grape, and cherry tomato
Adjustment	test tube, pencil, cuboid, cylinder, and triangular prism	cuboid, cylinder

TABLE II: Objects used for training and manipulation across different robotic tasks. Each object is associated with 150 manipulation trajectories in the dataset for training world models and manipulation policies.

Component	Parameters
TactileVAE	1.96 M
Visuo–Tactile World Model	271.57 M
Adaptive Visuo–Tactile Fusion Policy	326.89 M
RLTC	0.10 M
Total	600.52 M

TABLE III: Model sizes of the main OmniVTA components.

Fig. 9: Objects used in the manipulation tasks.

Fig. 10: Real-robot manipulation results. We visualize the manipulation process across six task categories.

Task	Wipe			Peel			Cut			Assembly			Grasp	Adjustment
Method / Eval. setting	O	G	P	O	G	P	O	G	P	O	G	P	O	O	G
DP	0.12	0.05	0	0.06	0	0	0.28	0.10	0	0.10	0	0.05	0.20	0	0
DP+tactile	0.36	0.28	0	0.32	0.20	0.08	0.33	0.15	0.13	0.30	0.10	0.10	0.48	0.25	0.15
KineDex	0.40	0.25	0	0.24	0.13	0.05	0.38	0.30	0.20	0.30	0.15	0.15	0.65	0.30	0.20
ForceMimic	0.33	0.20	0	0.27	0.18	0	0.50	0.25	0.05	0.35	0.15	0.10	0.60	0.10	0
RDP	0.50	0.38	0.42	0.48	0.36	0.45	0.65	0.50	0.43	0.60	0.50	0.35	0.88	0.50	0.50

TABLE IV: Performance comparison of OmniVTA and baseline methods across six tasks under three evaluation settings: object diversity (O), generalization (G), and perturbation robustness (P). Results are reported in terms of success rate.

Method	Wipe		Peel		Cut		Assembly		Grasp		Adjustment
	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$
PCA	0.091	0.810	0.085	0.430	0.109	0.400	0.071	0.720	0.036	0.600	0.069	0.560
PointNet-AE	0.059	0.910	0.067	0.850	0.062	0.840	0.058	0.900	0.028	0.750	0.047	0.760
Ours	0.038	0.930	0.033	0.880	0.031	0.940	0.022	0.910	0.011	0.720	0.017	0.850

TABLE V: Performance comparison of different methods across robotic tasks using L2 and cosine similarity metrics. Lower L2 (

\downarrow

) and higher cos (

\uparrow

) indicate better performance.

Method	GelSight-Mini	Tac3D-A1	Xense-QN1
w/o implicit decoder	$0.126$	$0.098$	$0.038$
w/ position embed.	$0.102$	$0.085$	$0.035$
w/o spatial feature map	$0.107$	$0.084$	$0.071$
w/ implicit decoder	0.047	0.058	0.034

TABLE VI: Ablation study on the design of TactileVAE. We evaluate reconstruction performance using the

L_{2}

metric.

Fig. 11: Ablation study of TactileVAE.(a) shows the sensors and objects for testing. (b) represents the three types of force loaded on the sensors. (c) visualizes the t-SNE of the tactile latent feature maps extracted by different models.

Fig. 12: Visuo-tactile generation visualization across six task categories. Red arrows indicate predicted tangential deformation, while blue arrows denote ground truth. Normal deformation in the grasp task is visualized using heatmaps.

	Wipe								Peel								Cut
Method	$L2_{1}$	$L2_{2}$	$L2_{3}$	$L2_{\text{avg}}$	$C_{1}$	$C_{2}$	$C_{3}$	$C_{\text{avg}}$	$L2_{1}$	$L2_{2}$	$L2_{3}$	$L2_{\text{avg}}$	$C_{1}$	$C_{2}$	$C_{3}$	$C_{\text{avg}}$	$L2_{1}$	$L2_{2}$	$L2_{3}$	$L2_{\text{avg}}$	$C_{1}$	$C_{2}$	$C_{3}$	$C_{\text{avg}}$
UVA	0.084	0.091	0.088	0.088	0.68	0.65	0.64	0.66	0.090	0.097	0.102	0.097	0.62	0.60	0.60	0.61	0.075	0.076	0.078	0.077	0.72	0.71	0.71	0.71
exUMI	0.097	0.098	0.103	0.101	0.59	0.58	0.58	0.58	0.093	0.099	0.102	0.097	0.63	0.62	0.62	0.62	0.083	0.086	0.088	0.086	0.73	0.72	0.71	0.72
Kinedex	0.081	0.084	0.085	0.082	0.82	0.80	0.81	0.81	0.064	0.066	0.068	0.066	0.77	0.79	0.80	0.79	0.091	0.100	0.104	0.096	0.74	0.73	0.72	0.73
Forcemimic	0.089	0.091	0.094	0.091	0.70	0.68	0.68	0.68	0.077	0.077	0.078	0.077	0.76	0.76	0.73	0.76	0.089	0.091	0.091	0.090	0.71	0.71	0.72	0.71
Ours	0.052	0.062	0.072	0.059	0.93	0.92	0.91	0.93	0.033	0.037	0.041	0.036	0.88	0.88	0.87	0.87	0.048	0.052	0.057	0.050	0.89	0.89	0.88	0.88

TABLE VII: Prediction performance. We report the

L_{2}

distance and cosine similarity at the 2nd, 4th, and 6th latent prediction steps (denoted as

L2_{1},L2_{2},L2_{3}

and

C_{1},C_{2},C_{3}

), corresponding to the 8th, 16th, and 24th decoded tactile frames, respectively. We also report the average performance over all 24 predicted frames (

L2_{\text{avg}}

C_{\text{avg}}

). Top: Wipe/Peel/Cut. Bottom: Adjustment/Assembly/Grasp. “–” denotes not applicable.

Settings	Action	Dyn. Weight	Joint Gen.	L2 $\downarrow$	Cos $\uparrow$
Unseen Position	3D absolute	✓	✓	0.075	0.72
	3D relative	✓	✓	0.056	0.88
	2D	✓	✓	0.042	0.91
Seen Position	2D	$\times$	$\times$	0.041	0.90
	2D	$\times$	✓	0.038	0.92
	2D	✓	✓	0.035	0.93

TABLE VIII: Performance evaluation under different test cases, action spaces, and dynamic weight settings. Lower

L_{2}

and higher cosines indicate better performance.

Fig. 13: Perturbation and recovery. The first row shows a contact break due to downward perturbation, with tactile signals transitioning to a no-contact state. The second row shows recovery to the contact state within one generation chunk. In tactile visualization, red arrows denote tactile prediction while blue arrows denote ground truth.

Component					Success Rate
Pred. Len.	Used as Contact State	LTD	Gating	Visual Gen.	Wipe	Peel	Avg.
0	$\times$	$\times$	$\times$	$\times$	0.12	0.06	0.09
2	$\times$	$\times$	$\times$	$\times$	0.40	0.26	0.33
4	$\times$	$\times$	$\times$	$\times$	0.45	0.30	0.38
6	$\times$	$\times$	$\times$	$\times$	0.50	0.30	0.40
6	✓	$\times$	$\times$	$\times$	0.46	0.28	0.37

TABLE IX: Ablation study on Adaptive Visuo-Tactile Fusion Policy and the components’ impact on task success rate.

Slow Policy	Slow Policy w/ Visual Gen.	Fast Policy
230 ms	480 ms	3.5 ms

TABLE X: Inference time of each component on RTX 4090D.

Fig. 14: Gating Fusion Mechanism. We visualize the temporal evolution of the predicted contact probability (purple), together with the corresponding visual (blue) and tactile (red) weights over the course of task execution.

Fig. 15: Perturbation experiments. We break contact by abruptly lowering the object and demonstrate how the policy rapidly restores contact via the RLTC.

Fig. 16: Recovery time of OmniVTA w/ RLTC and w/o RLTC.

Fig. 17: Effect of tactile prediction accuracy. Left: At 60% prediction performance, the model fails to estimate future contact probability, resulting in improper modality weighting. Right: Lower prediction performance leads to a significant drop in success rate.

Explore similar work

Jul 24, 2026

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/

Yunao Huang, Shiyu Sang, Suting Ni +6

ShanghaiTech University · InstAdapt

World Models for Robotics Contact-Rich Robotic Manipulation World Models Synthetic Data Generation for Robotics Tactile Sensing for Robotic Manipulation Tactile World Models

Ask PDF

Jul 2, 2026

VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation

Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.

Shuai Tian, Yupeng Zheng, Yuhang Zheng +7

SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · TARS Robotics +2

Contact-Rich Robotic Manipulation World Action Models Embodied World Models Tactile World Models

Ask PDF

Dec 29, 2025

Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

Vision-Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs coupled with wrist-camera local vision and third-person macro vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial Alignment (HSA) loss that aligns tactile tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic agents.

Guo Ye, Zexi Zhang, Xu Zhao +4

Northwestern University, Evanston, IL, USA

Contact-Rich Robotic Manipulation World Models Tactile Sensing for Robotic Manipulation VLMs for Robotics Robotic Perception Tactile World Models

Ask PDF