OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Yuhang Zheng; Songen Gu; Yupeng Zheng; Weize Li; Yujie Zang; Shuai Tian; Xiang Li; Ce Hao

Fig. 2: Overview of OmniViTac dataset. Left: The Cross-Embodiment Data Collection Platform features UNFactory 7-DoF xArm and the TacUMI manipulation interface, both supporting identical end-effectors and diverse tactile sensors (Xense, GelSight Mini, Tac3D, Daimon). Middle: The dataset covers 6 visuo-tactile manipulation patterns, instantiated across 5 semantic scenarios. Top-Right: A scale comparison demonstrates that OmniViTac (21,879 trajectories) significantly exceeds existing visuo-tactile manipulation datasets in tactile-rich data volume. Bottom-Right: The High-quality Data Pipeline ensures reliability through timestamp alignment, visualization, and human-in-the-loop verification.

Fig. 3: Example visualization of the

6

visuo-tactile manipulation patterns in OmniViTac. The dataset captures diverse, contact-rich behaviors across various scenarios, including Assembly, Cutting, Adjustment, Peeling, Grasping, and Wiping. Each panel illustrates the global third-person workspace view overlaid with the end-effector’s trajectory, alongside synchronized, high-frequency tactile maps (bottom) that continuously record the complex tool-object contact dynamics during task execution.

Dataset	#Task	#Traj.	#Object	Vision	Tactile			Action	Collection
Dataset	#Task	#Traj.	#Object	Vision	Sensors	Force Dim.	Frequency	Action	Collection
ObjectFolder2.0 [ 20 ]	3	–	1,000	Rendered	GelSight	3	–	✗	Simulated
AnyTouch [ 18 ]	1	–	124	–	GelSight Mini/DIGIT/DuraGel/Tac3D	3	30	✗	Handheld
VTV150k [ 48 ]	–	–	100	–	GelSight Mini/DIGIT/Tac3D	–	–	✗	Handheld
Octopi-1.5 [ 59 ]	3	–	–	GoPro	GelSight Mini/TAC-02	–	50 $\sim$ 1000	✓	TMI
See, Hear, and Feel [ 30 ]	2	60	4	RealSense	GelSight	–	30	✓	Teleop

TABLE I: Comparison of OmniViTac with existing vision-tactile-action datasets. Note: * indicates that the statistics count only the visuo-tactile trajectories in the corresponding dataset. “–” denotes not applicable.

Fig. 4: Comprehensive statistical analysis of the OmniViTac. (a) Pattern-level contact area distribution, highlighting the distinct dichotomy between precision-dominant tasks (concentrated in the

0

-

10\%

range) and surface-dominant tasks (peaking at

70

-

90\%

). (b) Force intensity distribution, showcasing the wide spectrum of force magnitudes required across different manipulation modes. (c) Hierarchical distribution illustrating the rich diversity of

86

instantiated tasks. (d) Effective contact ratio with variance across patterns, demonstrating the temporal dependency on tactile feedback (e.g., Adjustment exhibits the highest continuous engagement). (e) Total trajectory counts per category, demonstrating the massive scale and balanced composition of the benchmark. (f) t-SNE projection of the high-dimensional tactile signals, revealing physically intuitive and semantically separable latent clusters that strictly align with the underlying contact mechanics of each interaction pattern.

Fig. 5: System Overview . OmniVTA is a hierarchical slow–fast policy for contact-rich manipulation. The slow policy contains a visuo-tactile world model and an adaptive fusion policy to generate long-horizon action chunks from multi-modal inputs. The fast policy outputs high-frequency refinements at

60

Hz using tactile feedback. Final actions are a weighted summation of slow-planned and fast-refined outputs, enabling both long-horizon planning and reactive control for robust manipulation.

Fig. 6: Overview of TactileVAE. The model encodes 3D marker displacements into spatio-temporal features, and reconstructs the deformation via an implicit decoder that represents the deformation as a continuous field.

Fig. 7: Overview of the Slow Policy. (a): The proposed Visuo-Tactile World Model takes noisy tactile and visual latent sequences as input and jointly generates both modalities via a two-stream spatio-temporal diffusion transformer. (b): The Adaptive Visuo-Tactile Fusion Policy employs a Latent Tactile Differential Encoder to effectively encode the observed and predicted tactile and a gating mechanism to adaptively balance tactile and visual information for robust action planning.

Fig. 8: Overview of the Reflexive Latent Tactile Controller (RLTC). It takes single-frame tactile feedback, chunk-aligned predicted tactile latents, and robot/gripper delta states as input to produce high-frequency (60 Hz) refined single-step corrective actions.

Task	Object Used for Training	Object Used for Manipulation
Wipe	vases in 4 different colors and shapes, plate, and whiteboard	vases in 4 different colors and shapes
Peel	cucumber, Chinese yam, radish, carrot, and white daikon	cucumber, Chinese yam, radish
Cut	cucumber, Chinese yam, carrot, pepper, and banana	cucumber, Chinese yam, pepper, banana
Assembly	3 colors of USB stick and 3 types of charging adapters	red USB stick and silver USB stick
Grasp	blueberry, strawberry, grape, cherry, and cherry tomato	blueberry, strawberry, grape, and cherry tomato
Adjustment	test tube, pencil, cuboid, cylinder, and triangular prism	cuboid, cylinder

TABLE II: Objects used for training and manipulation across different robotic tasks. Each object is associated with 150 manipulation trajectories in the dataset for training world models and manipulation policies.

Component	Parameters
TactileVAE	1.96 M
Visuo–Tactile World Model	271.57 M
Adaptive Visuo–Tactile Fusion Policy	326.89 M
RLTC	0.10 M
Total	600.52 M

TABLE III: Model sizes of the main OmniVTA components.

Fig. 9: Objects used in the manipulation tasks.

Fig. 10: Real-robot manipulation results. We visualize the manipulation process across six task categories.

Task	Wipe			Peel			Cut			Assembly			Grasp	Adjustment
Method / Eval. setting	O	G	P	O	G	P	O	G	P	O	G	P	O	O	G
DP	0.12	0.05	0	0.06	0	0	0.28	0.10	0	0.10	0	0.05	0.20	0	0
DP+tactile	0.36	0.28	0	0.32	0.20	0.08	0.33	0.15	0.13	0.30	0.10	0.10	0.48	0.25	0.15
KineDex	0.40	0.25	0	0.24	0.13	0.05	0.38	0.30	0.20	0.30	0.15	0.15	0.65	0.30	0.20
ForceMimic	0.33	0.20	0	0.27	0.18	0	0.50	0.25	0.05	0.35	0.15	0.10	0.60	0.10	0
RDP	0.50	0.38	0.42	0.48	0.36	0.45	0.65	0.50	0.43	0.60	0.50	0.35	0.88	0.50	0.50

TABLE IV: Performance comparison of OmniVTA and baseline methods across six tasks under three evaluation settings: object diversity (O), generalization (G), and perturbation robustness (P). Results are reported in terms of success rate.

Method	Wipe		Peel		Cut		Assembly		Grasp		Adjustment
	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$	L2 $\downarrow$	cos $\uparrow$
PCA	0.091	0.810	0.085	0.430	0.109	0.400	0.071	0.720	0.036	0.600	0.069	0.560
PointNet-AE	0.059	0.910	0.067	0.850	0.062	0.840	0.058	0.900	0.028	0.750	0.047	0.760
Ours	0.038	0.930	0.033	0.880	0.031	0.940	0.022	0.910	0.011	0.720	0.017	0.850

TABLE V: Performance comparison of different methods across robotic tasks using L2 and cosine similarity metrics. Lower L2 (

\downarrow

) and higher cos (

\uparrow

) indicate better performance.

Method	GelSight-Mini	Tac3D-A1	Xense-QN1
w/o implicit decoder	$0.126$	$0.098$	$0.038$
w/ position embed.	$0.102$	$0.085$	$0.035$
w/o spatial feature map	$0.107$	$0.084$	$0.071$
w/ implicit decoder	0.047	0.058	0.034

TABLE VI: Ablation study on the design of TactileVAE. We evaluate reconstruction performance using the

L_{2}

metric.

Fig. 11: Ablation study of TactileVAE.(a) shows the sensors and objects for testing. (b) represents the three types of force loaded on the sensors. (c) visualizes the t-SNE of the tactile latent feature maps extracted by different models.

Fig. 12: Visuo-tactile generation visualization across six task categories. Red arrows indicate predicted tangential deformation, while blue arrows denote ground truth. Normal deformation in the grasp task is visualized using heatmaps.

	Wipe								Peel								Cut
Method	$L2_{1}$	$L2_{2}$	$L2_{3}$	$L2_{\text{avg}}$	$C_{1}$	$C_{2}$	$C_{3}$	$C_{\text{avg}}$	$L2_{1}$	$L2_{2}$	$L2_{3}$	$L2_{\text{avg}}$	$C_{1}$	$C_{2}$	$C_{3}$	$C_{\text{avg}}$	$L2_{1}$	$L2_{2}$	$L2_{3}$	$L2_{\text{avg}}$	$C_{1}$	$C_{2}$	$C_{3}$	$C_{\text{avg}}$
UVA	0.084	0.091	0.088	0.088	0.68	0.65	0.64	0.66	0.090	0.097	0.102	0.097	0.62	0.60	0.60	0.61	0.075	0.076	0.078	0.077	0.72	0.71	0.71	0.71
exUMI	0.097	0.098	0.103	0.101	0.59	0.58	0.58	0.58	0.093	0.099	0.102	0.097	0.63	0.62	0.62	0.62	0.083	0.086	0.088	0.086	0.73	0.72	0.71	0.72
Kinedex	0.081	0.084	0.085	0.082	0.82	0.80	0.81	0.81	0.064	0.066	0.068	0.066	0.77	0.79	0.80	0.79	0.091	0.100	0.104	0.096	0.74	0.73	0.72	0.73
Forcemimic	0.089	0.091	0.094	0.091	0.70	0.68	0.68	0.68	0.077	0.077	0.078	0.077	0.76	0.76	0.73	0.76	0.089	0.091	0.091	0.090	0.71	0.71	0.72	0.71
Ours	0.052	0.062	0.072	0.059	0.93	0.92	0.91	0.93	0.033	0.037	0.041	0.036	0.88	0.88	0.87	0.87	0.048	0.052	0.057	0.050	0.89	0.89	0.88	0.88

TABLE VII: Prediction performance. We report the

L_{2}

distance and cosine similarity at the 2nd, 4th, and 6th latent prediction steps (denoted as

L2_{1},L2_{2},L2_{3}

and

C_{1},C_{2},C_{3}

), corresponding to the 8th, 16th, and 24th decoded tactile frames, respectively. We also report the average performance over all 24 predicted frames (

L2_{\text{avg}}

,

C_{\text{avg}}

). Top: Wipe/Peel/Cut. Bottom: Adjustment/Assembly/Grasp. “–” denotes not applicable.

Settings	Action	Dyn. Weight	Joint Gen.	L2 $\downarrow$	Cos $\uparrow$
Unseen Position	3D absolute	✓	✓	0.075	0.72
	3D relative	✓	✓	0.056	0.88
	2D	✓	✓	0.042	0.91
Seen Position	2D	$\times$	$\times$	0.041	0.90
	2D	$\times$	✓	0.038	0.92
	2D	✓	✓	0.035	0.93

TABLE VIII: Performance evaluation under different test cases, action spaces, and dynamic weight settings. Lower

L_{2}

and higher cosines indicate better performance.

Fig. 13: Perturbation and recovery. The first row shows a contact break due to downward perturbation, with tactile signals transitioning to a no-contact state. The second row shows recovery to the contact state within one generation chunk. In tactile visualization, red arrows denote tactile prediction while blue arrows denote ground truth.

Component					Success Rate
Pred. Len.	Used as Contact State	LTD	Gating	Visual Gen.	Wipe	Peel	Avg.
0	$\times$	$\times$	$\times$	$\times$	0.12	0.06	0.09
2	$\times$	$\times$	$\times$	$\times$	0.40	0.26	0.33
4	$\times$	$\times$	$\times$	$\times$	0.45	0.30	0.38
6	$\times$	$\times$	$\times$	$\times$	0.50	0.30	0.40
6	✓	$\times$	$\times$	$\times$	0.46	0.28	0.37

TABLE IX: Ablation study on Adaptive Visuo-Tactile Fusion Policy and the components’ impact on task success rate.

Slow Policy	Slow Policy w/ Visual Gen.	Fast Policy
230 ms	480 ms	3.5 ms

TABLE X: Inference time of each component on RTX 4090D.

Fig. 14: Gating Fusion Mechanism. We visualize the temporal evolution of the predicted contact probability (purple), together with the corresponding visual (blue) and tactile (red) weights over the course of task execution.

Fig. 15: Perturbation experiments. We break contact by abruptly lowering the object and demonstrate how the policy rapidly restores contact via the RLTC.

Fig. 16: Recovery time of OmniVTA w/ RLTC and w/o RLTC.

Fig. 17: Effect of tactile prediction accuracy. Left: At 60% prediction performance, the model fails to estimate future contact probability, resulting in improper modality weighting. Right: Lower prediction performance leads to a significant drop in success rate.

OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Abstract

Figures & tables

Explore similar work

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation

Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation