OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework

Yifan Wu; Qin Li; Nan Min; Guojin Zhong; Haoyu Zhao; Zhiyuan Li; Houze Xu; Shengqi Xu

cs.ROOct 7, 2026

OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework

Authors: Yifan Wu, Qin Li, Nan Min, Guojin Zhong, Haoyu Zhao, Zhiyuan Li, Houze Xu, Shengqi Xu, +9 more

Organizations: Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Hefei University of Technology · National University of Singapore · China Unicom · Neote AI

Abstract

Tactile feedback provides embodied agents with physical information beyond visual observations, enabling more reliable interaction with the real world. However, despite the rapid progress of vision-tactile-language-action (VTLA) policies, there remains a lack of unified benchmarks for evaluating tactile-enabled robot manipulation across simulation and the real world. To address this gap, we introduce OpenViTac, a visuo-tactile manipulation benchmark for evaluating robot policies across simulation and the real world. OpenViTac organizes contact-rich manipulation into four tactile-relevant capability dimensions and provides paired simulation-real-world settings for consistent evaluation of VLA, WAM, and VTLA policies. Building upon this benchmark, we investigate how different tactile representations and integration strategies affect the performance of pretrained VLA models. Correspondingly, we introduce OpenVTLA, a tactile augmentation framework that combines the best-performing representation and integration strategy. Furthermore, we leverage the paired benchmark setting to study sim-real co-training and analyze factors affecting cross-domain policy learning. Together, OpenViTac provides a unified platform for evaluating and advancing visuo-tactile robot manipulation.

Figures & tables

Figure 1 : Overview of OpenViTac. OpenViTac establishes a unified benchmark for visuo-tactile manipulation, spanning four complementary capability dimensions: physical-property perception, fragility-aware interaction, contact-rich manipulation, and precision manipulation. It provides task-corresponding simulation and real-world settings with aligned assets, supports multiple gripper configurations and tactile sensors, and enables consistent evaluation of visuo-tactile policies across both domains.

Benchmark	Gripper Types	Tactile Sensing	Tactile Sensor Types	Property Perception	Fragility-/Force- Aware	Precision Manipulation	Scene Diversity
CALVIN [ 25 ]	1	✓	1	✗	✗	✗	✗
LIBERO [ 20 ]	1	✗	-	✗	✗	✗	✗
ManiFeel [ 22 ]	1	✓	1	✗	✗	✗	✗
Tabero [ 37 ]	1	✓	1	✗	✓	✗	✗
UniVTAC [ 3 ]	1	✓	1	✓	✗	✓	✗
NeoSim [ 31 ]	1	✓	1	✗	✓	✓	✗

Table 1 : Comparison of representative simulated manipulation benchmarks with OpenViTac. We include recent visuo-tactile benchmarks together with representative vision-based benchmarks.

Panel of Figure 2 — Figure 2 : Overview of the OpenViTac design. (a) Capability taxonomy with four visuo-tactile capabilities and their underlying skills. (b) Construction of task-corresponding simulation environments from real-world references at the asset- and task-levels. (c) Support for scene diversification through configurable appearance variations in simulation.

Figure 3 : Design space for adapting pretrained VLAs to touch. We investigate two complementary dimensions: tactile representation and tactile integration .

Models		PP					FA	CR		PR			Avg.
Models		WC	HC	RC	RR	ECS	GC	GA	PD	In-USB	In-B v1	In-B v2	Avg.
VLAs	$\pi_{0.5}$	53	60	64	52	48	39	87	99	14	9	9.3	48.6
	StarVLA (Qwen3)	44	38	42	35	32	10	26	35	4	0	0	24.2
	InternVLA-A1.5	53	76	65	77	47	53	79	98	3	6	9.3	51.5
	XVLA	28	30	56	47	45	42	84	76	2	3	0.7	37.6
	Lingbot-VLA-2	52	58	58	53	52	41	78	92	8	5	2.7	45.4

Table 2 : Comparison of VLA, WAM, and VTLA models on OpenViTac in the simulation environment . Bold denotes the best results and underline denotes the second-best.

Models		PP		FA	CR		PR			Avg.
Models		RC	ECS	GC	GA	PD	In-USB	In-B v1	In-B v2
VLAs	$\pi_{0.5}$	10/20	10/20	16/20	7/20	12/20	3/20	5/20	4/30	41.0
	InternVLA-A1.5	10/20	10/20	17/20	9/20	13/20	3/20	3/20	6/30	43.1
	Lingbot-VLA-2	10/20	10/20	15/20	3/20	10/20	1/20	2/20	5/30	34.0
	Xiaomi-Robotics	10/20	10/20	16/20	5/20	8/20	1/20	3/20	2/30	34.0
WAMs	FastWAM	8/20	9/20	12/20	5/20	10/20	0/20	1/20	2/30	29.0

Table 3 : Comparison of VLA, WAM, and VTLA models on OpenViTac in the real-world environment . Bold denotes the best results and underline denotes the second-best.

Figure 4 : Performance of different tactile representations under the same token-level concatenation strategy.

Method	Representation	Integration	PP	FA	CR	PR	Avg.
OpenVTLA	AnyTouch2	① Concat	90.2	69.0	95.0	15.3	68.7
${\pi_{0.5}}^{\dagger}$	Sparsh	② Modulation	82.2	49.0	73.0	5.0	56.5
${\pi_{0.5}}^{\dagger}$	AnyTouch2	③ Action	70.8	43.0	62.0	4.7	48.6
FTP-1	T3	④ Expert	77.8	60.0	88.5	15.8	61.2
${\pi_{0.5}}^{\dagger}$	AnyTouch2	⑤ Refine	63.0	51.0	81.0	8.7	50.4

Table 4 : Tactile integration comparison. Scores are averaged over four task groups.

{\pi_{0.5}}^{\dagger}

denotes variants obtained by adapting the pretrained

\pi_{0.5}

backbone with different tactile integration strategies. Bold denotes the best results and underline denotes the second-best.

Model	Sim.	Std. Lighting		Lighting Perturb.
Model	Sim.	In-USB	GA	In-USB	GA
$\pi_{0.5}$	0	3/20	7/20	0/20	2/20
	250	5/20	9/20	–	–
	500	8/20	10/20	4/20	8/20
OpenVTLA	0	4/20	10/20	2/20	10/20
	250	6/20	10/20	–	–

Table 5 : Sim–real co-training and robustness. All settings use 100 real demonstrations per task; “Sim.” denotes the number of additional simulated demonstrations.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Abbr.	Task	Capability under evaluation
Physical Property Perception	WC	Weight Classify	Infer object weight through interaction and place the object at the corresponding target.
	HC	Hardness Classify	Infer object hardness through contact and place the object at the corresponding target.
	RC	Roughness Classify	Infer surface roughness through contact and place the object at the corresponding target.
	RR	Roughness-Guided Regrasp	Infer relative roughness and keep or regrasp the appropriate object accordingly.
	ECS	Empty-Can Selection	Distinguish visually similar empty and filled cans from their physical properties and select the instructed one.
Fragility-Aware Manipulation	GC	Grasp Chip	Grasp and place a fragile chip while avoiding damage throughout the manipulation.

Appendix

Table S1: OpenViTac task taxonomy. Simulation and real-world evaluations use the same task semantics and success criteria for tasks instantiated in both domains.

Table S2 : Data collection statistics of OpenViTac. Statistics are reported per tactile sensor configuration for both simulated episodes and real-world demonstrations.

Task	Task Objective	Success Condition
Weight Classify (WC)	Infer whether a block is light or heavy through interaction and place it on the corresponding target.	The block is correctly classified and placed on the corresponding target plate.
Hardness Classify (HC)	Infer whether a block is soft or hard through contact interaction and place it on the corresponding target.	The block is correctly classified and placed on the corresponding target plate.
Roughness Classify (RC)	Infer whether a block is smooth or rough through contact interaction and place it on the corresponding target.	The block is correctly classified and placed on the corresponding target plate.
Empty-Can Selection (ECS)	Identify the empty can among visually similar cans through physical interaction and place it into the basket.	The empty can is correctly identified and successfully placed into the basket.
Roughness-Guided Regrasp (RR)	Determine whether to retain the initially grasped block based on roughness and regrasp when necessary.	The rough block is correctly selected and placed on the target plate.
Grasp Chip (GC)	Grasp and transport a fragile chip while avoiding damage during manipulation.	The chip is successfully grasped, transported, and placed without damage while satisfying physical safety constraints.

Appendix

Table S3 : Task catalog of OpenViTac, including task objectives and high-level success conditions.

Models		Steps	Global batch
VLAs	$\pi_{0.5}$	20k	64
	StarVLA (Qwen3)	20k	64
	InternVLA-A1.5	40k	32
	XVLA	30k	64
	Lingbot-VLA-2	20k	64
	Xiaomi-Robotics	20k	64

Appendix

Table S4 : Training settings for the evaluated policies.

Representation	PP	FA	CR	PR	Avg.
AnyTouch2 (Random Init.)	78.5	34.0	82.0	10.2	56.5
AnyTouch2 (Pretrained)	90.2	69.0	95.0	15.3	68.7

Appendix

Table S5 : Comparison between pretrained AnyTouch2 and its randomly initialized counterpart across the four capability groups.

Temporal Input	PP	FA	CR	PR	Avg.
Repeated Current Frame	79.0	52.0	90.0	5.2	58.4
Tactile History	90.2	69.0	95.0	15.3	68.7

Appendix

Table S6 : Comparison between the original tactile history and repeated copies of the current frame across the four capability groups.

Figure S1 : Visualization of the insert USB task using GelSight, Xense, and NeoteAI tactile sensors.

Figure S2 : Visualization of weight classify task.

Figure S3 : Visualization of grasp chip task.

Figure S4 : Visualization of hardness classify task.

Figure S5 : Visualization of gear assembly task.

Figure S6 : Visualization of roughness classify task.

Figure S7 : Visualization of roughness-guided regrasp task.

Figure S8 : Visualization of empty-can selection task.

Figure S9 : Visualization of pull drawer task.

Figure S10 : Visualization of insert USB task.

Figure S11 : Visualization of insert block v1 task.

Figure S12 : Visualization of insert block v2 task (part 1).

Figure S13 : Visualization of insert block v2 task (part 2).

Figure S14 : Visualization of empty-can selection task.

Figure S15 : Visualization of gear assembly task.

Figure S16 : Visualization of grasp chip task.

Figure S17 : Visualization of pull drawer task.

Figure S18 : Visualization of insert USB task.

Figure S19 : Visualization of insert block v1 task.

Figure S20 : Visualization of insert block v2 task (part 1).

Figure S21 : Visualization of insert block v2 task (part 2).

Explore similar work

$N_0$ -VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
Jul 26, 2026NeoteAI Team, Fudan TEAI TeamTactile
NeoteAI Team · Fudan TEAI Team
TacCoRL: Integrating Tactile Feedback into VLA via Simulation
Jun 10, 2026Siyu Ma, Yuqi Liang, Chang Yu +5Tactile Simulation-Based Reinforcement Learning
University of California, Los Angeles · University of California, San Diego · 4Peking University +1
ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
Jul 24, 2026Yunao Huang, Shiyu Sang, Suting Ni +6Tactile World Model Robotic Manipulation
ShanghaiTech University · InstAdapt

Jul 26, 2026, cs.ROJ/K move · Enter open · S save

$N_0$ -VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

NeoteAI Team, Fudan TEAI Team

NeoteAI Team · Fudan TEAI Team

Tactile

Open paper AskPDF arXiv

We present

N_0

-VTLA, a vision-tactile-language-action (VTLA) foundation model capable of (1) fine-grained contact-rich manipulation with tactile perception and tactile-feedback control, and (2) offline policy improvement from stored deployment data. Building on current vision-based backbones, we propose a training recipe for tactile integration consisting of visuo-tactile pre-training, staged tactile-pathway integration, and advantage-conditioned offline policy improvement. During pre-training, the policy learns broad contact priors from NeoData, our large-scale visuo-tactile robot dataset; to our knowledge,

N_0

-VTLA is the first VTLA model pretrained on tactile data at scale. During post-training, we augment the policy with a predictive tactile pathway that distills the contact patterns learned at scale into the fine motion adjustments required by downstream tactile-centric manipulation. For offline policy improvement, we introduce ALTER, an advantage-conditioned offline reinforcement learning method that converts relative progress and trajectory-event comparisons into binary advantage labels for policy training on a fixed deployment corpus, further improving task-specific learning on contact-rich skills such as deformable object manipulation. Across contact-rich benchmarks,

N_0

-VTLA outperforms strong baselines by wide margins: it wins all nine real-robot NeoReal tasks and reaches 63.8% mean success on a twenty-task simulation suite, against 44.0% for the strongest baseline.

N_0

-VTLA policies trained with ALTER reach 75-95% success on three long-horizon real-robot tasks. These results lay a foundation for versatile tactile-driven manipulation policies.

Abstract

Explore similar work

N0N_0N0​-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

$N_0$ -VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens