SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

Bowen Jing; Mingxin Wang; Ruiyang Hao; Chenchen Ge; Hanwen Shen; Junjie He; Yang Cui; Yiming Hou

cs.ROAug 19, 2026

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

Authors: Bowen Jing, Mingxin Wang, Ruiyang Hao, Chenchen Ge, Hanwen Shen, Junjie He, Yang Cui, Yiming Hou,

Organizations: Tuojing Intelligence · Tsinghua University · King’s College London · Southeast University ·

Abstract

Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the task while allowing slip or causing excessive compression. A primary bottleneck is the absence of visuo-tactile datasets that pair policy-visible contact observations with independent physical ground truth over complete tasks. We introduce SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation. It contains 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions, alongside evaluator-only finite-element (FEM) states. Building upon this dataset, we establish a closed-loop benchmark that uses fixed object-specific calibration to define the Deformation-aware Success Rate (DSR), which counts a rollout as successful only when it completes the task and keeps peak normalized deformation within tolerance. Across Diffusion Policy, $π_{0.5}$ , and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the deformation tolerance, accounting for 0.7--24% of each configuration's successes. Under distribution shift, visuo-tactile variants achieve higher task success in all six policy--suite comparisons and higher DSR in five, whereas their in-distribution benefits are mixed. These results show that making touch available does not by itself ensure effective multimodal fusion. SoftVTBench therefore provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

Figures & tables

Figure 1: Overview of SoftVTBench. Top: $4{,}000$ demonstrations over four diagnostic suites and more than $50$ assets, including volumetric deformable objects and visually matched rigid twins. Bottom left: controlled visual and physical shifts for closed-loop generalization evaluation. Bottom center: the physical-interaction regimes captured by touch and evaluated from object deformation, from slip-prone loose grasps to excessive compression. Bottom right: synchronized visual, tactile, proprioceptive, language, and action streams.

Benchmark	Complete Task	3D Deformable	Policy-Visible Touch	Evaluator-Only Deformation Scoring
LIBERO Liu et al. (2023)	✓	✗	✗	✗
ManiSkill2 Gu et al. (2023)	✓	✓	✗	✗
SoftGym Lin et al. (2020)	✓	❍	✗	✗
MoDeSuite Zhang et al. (2025c)	✓	❍	✗	✗
DefGraspSim Huang et al. (2022)	✗	✓	✗	✓
SoGraB Greenland et al. (2024)	✗	✓	✗	✓

Table 1: Benchmark positioning. Complete Task : the benchmark scores multi-stage manipulation (approach, grasp, transport, place) rather than an isolated grasp. 3D Deformable : the manipulated objects are volumetric soft bodies, as opposed to cloth or rope ( ❍ ). Policy-Visible Touch : tactile observations are available to the policy as input. Evaluator-Only Deformation Scoring : object deformation is measured from physical state hidden from the policy, and that measurement enters the success criterion, so an episode can complete the task and still be scored as a failure; ❍ marks benchmarks that constrain interaction on the input side (e.g. a force budget) without scoring the resulting deformation. Symbols denote full ( ✓ ), partial ( ❍ ), or no ( ✗ ) support. Benchmarks that expose touch do not score deformation from hidden state, and those that do evaluate isolated grasps; only SoftVTBench provides all four.

Figure 2: SoftVTBench construction and evaluation pipeline. Stage 1 constructs matched rigid–deformable objects and calibrates object-specific interaction constraints. Stage 2 generates controlled tasks and records policy-visible observations separately from evaluator-only physical states. Stage 3 applies automatic quality control and human verification, producing the released training, ID, and OOD splits. Task success and deformation compliance are labeled separately; DSR combines them during benchmark evaluation.

Suite	Object Type	Variation Axis	#Tasks	#Demos	ID Eval Episodes	OOD Conditions
Object-Soft	Deformable	Object identity	10	1,000	500	9
Spatial-Soft	Deformable	Spatial layout	10	1,000	500	9
Object-Rigid	Rigid twin	Object identity	10	1,000	500	–
Spatial-Rigid	Rigid twin	Spatial layout	10	1,000	500	–
Total	–	–	40	4,000	2,000	–

Table 2: Dataset composition and evaluation splits. The four suites form a matched

2\times 2

design over object type and variation axis. OOD evaluation uses nine single-factor conditions on the two deformable suites.

Figure 3: Tactile observations across deformable assets. Each panel shows a tactile RGB image with its marker-motion overlay at the moment of grasping, for the ten Object-Soft tasks. Objects that differ in geometry, compliance, and contact area produce distinct contact patches and shear fields under the same pick-and-place instruction. These are the cues available to a visuo-tactile policy and absent from external RGB views.

		Object-Soft		Spatial-Soft
Model	Input	TSR	DSR	TSR	DSR
Diffusion Policy	VO-C	37.4	33.6	15.6	13.4
Diffusion Policy	VT-C	40.0	30.4	33.0	25.0
$\pi_{0.5}$	VO-C	41.6	38.4	26.0	22.6
$\pi_{0.5}$	VT-C	41.4	35.0	27.6	22.0
FastWAM	VO-C	62.0	58.0	37.0	36.6

Table 3: In-distribution Task Success Rate and Deformation-aware Success Rate on the deformable suites (%). DSR is lower than TSR in every one of the twelve configurations; the difference is the fraction of rollouts that reach the target while leaving the calibrated interaction safety zone.

		Object variation		Spatial variation
Model	Input	Rigid	Soft	Rigid	Soft
Diffusion Policy	VO-C	40.0	37.4	14.0	15.6
Diffusion Policy	VT-C	35.0	40.0	11.0	33.0
$\pi_{0.5}$	VO-C	60.0	41.6	50.4	26.0
$\pi_{0.5}$	VT-C	59.6	41.4	54.0	27.6
FastWAM	VO-C	64.0	62.0	25.0	37.0

Table 4: Task success on deformable assets and their rigid twins (TSR, %). Twins match geometry, appearance, and mass under the same layouts and instructions, with negligible deformation, so a rigid–deformable difference isolates the effect of deformability. Because deformation is negligible on the twins, DSR coincides with TSR there and only TSR is reported.

	Object-Soft		Spatial-Soft
Configuration	TSR	DSR	TSR	DSR
VO-B	30.2	27.2	34.2	20.0
VO-C	41.6	38.4	26.0	22.6
VT-B	41.0	28.0	30.0	21.4
VT-C	41.4	35.0	27.6	22.0

Table 5: Matched sensing–control ablation for

\pi_{0.5}

(%). VO/VT denote vision-only and visuo-tactile inputs; B/C denote binary and continuous gripper control. Crossing the two factors separates a gain due to touch from a gain due to finer actuation.

		Object-Soft		Spatial-Soft
Model	Input	TSR $\uparrow$ ( $\Delta$ vs. ID)	DSR $\uparrow$ ( $\Delta$ vs. ID)	TSR $\uparrow$ ( $\Delta$ vs. ID)	DSR $\uparrow$ ( $\Delta$ vs. ID)
Diffusion Policy	VO-C	29.2 ( $-$ 8.2)	26.6 ( $-$ 7.0)	11.0 ( $-$ 4.6)	8.8 ( $-$ 4.6)
Diffusion Policy	VT-C	31.2 ( $-$ 8.8)	25.0 ( $-$ 5.4)	25.2 ( $-$ 7.8)	17.8 ( $-$ 7.2)
$\pi_{0.5}$	VO-C	35.8 ( $-$ 5.8)	33.2 ( $-$ 5.2)	24.4 ( $-$ 1.6)	19.4 ( $-$ 3.2)
$\pi_{0.5}$	VT-C	41.0 ( $-$ 0.4)	34.2 ( $-$ 0.8)	28.4 ( $+$ 0.8)	23.2 ( $+$ 1.2)
FastWAM	VO-C	54.4 ( $-$ 7.6)	53.8 ( $-$ 4.2)	27.8 ( $-$ 9.2)	27.2 ( $-$ 9.4)

Table 6: Out-of-distribution Task Success Rate and Deformation-aware Success Rate on the deformable suites (%), pooled over the nine held-out conditions of Section 3.1 .

\Delta

is the change relative to the corresponding in-distribution entry for the same model, input, and suite in Table 3 .

Figure 4: Task Success Rate under out-of-distribution shifts, resolved by factor. Rows correspond to the Object-Soft and Spatial-Soft suites; column groups correspond to Diffusion Policy, $\pi_{0.5}$ , and FastWAM, each resolved into illumination, object-mass, and Young’s-modulus shifts.

Supplementary section	Contents
Sec. 2 : Simulation and implementation details	Simulator, robot, sensing, FEM, and compute configuration.
Sec. 3 : Assets and interaction safety zones	Asset inventory, rigid twins, per-object calibration, and the calibration percentile.
Sec. 4 : Task suite details	Suite design, task mapping, and variation factors.
Sec. 5 : Baseline training details	Training and inference settings for all evaluated policy families.
Sec. 6 : Per-condition results	Out-of-distribution results per policy family and condition.
Sec. 7 : Additional analyses	Qualitative cases and stiffness-conditioned deformation statistics.

Table 7: Contents of the supplementary sections in document order.

Item	Value
Simulator	Isaac Sim 4.5.0 / Isaac Lab 0.41.3, PhysX 5 GPU pipeline
Physics / control rate	60 Hz physics, decimation 3, 20 Hz control
Robot	Franka arm with Panda parallel-jaw gripper
Controller	Task-space differential inverse kinematics
End-effector action	Absolute pose target: 3D position and 3D axis-angle orientation
Gripper action	Normalized closure command; continuous and binary encodings

Table 8: Simulation and sensing configuration.

Asset	$g^{\min}_{o}$	$g^{\max}_{o}$	Span (mm)	$\tau_{o}$ /diag. (%)
soft_pastry001	0.4000	0.8000	26.21	7.6
soft_pastry002	0.6547	0.6900	2.56	9.3
soft_pastry003	0.4400	0.8000	13.26	8.6
soft_pastry005	0.4700	0.5800	5.87	10.7
soft_pastry010	0.4200	0.6600	24.82	8.3
soft_pastry011	0.4400	0.5800	10.63	7.1

Table 9: Interaction safety zones of the ten deformable assets.

\tau_{o}

is the object-specific 90th-percentile threshold used by the released evaluator, expressed as a percentage of the reference bounding-box diagonal.

Suite	Task	Instruction	Asset
Object-Soft	0	Pick up the white swirled pastry and place it in the basket.	soft_pastry001
	1	Pick up the panda-face pastry and place it in the basket.	soft_pastry002
	2	Pick up the small chocolate rectangular pastry and place it in the basket.	soft_pastry003
	3	Pick up the soft cream-colored cube and place it in the basket.	soft_stw_cube_hq
	4	Pick up the yellow square layered pastry and place it in the basket.	soft_pastry005
	5	Pick up the soft tan cylinder and place it in the basket.	soft_stw_cylinder_hq

Table 10: Task specifications for the deformable suites. Task identifiers follow the released suite configuration and are zero-indexed; instructions are reproduced verbatim from that configuration, and object descriptions are therefore not normalized across suites.

Family	Factor	Training range	Held-out values
Visual	Dome-light intensity	135 (fixed nominal)	67.5, 180, 270
Spatial	Object placement	Per-task recorded support	—
	Target placement	Per-task recorded support	—
	Robot initialization	Per-task recorded support	—
Physical	Mass scale	$\times$ 1.0 (nominal)	$\times$ 1.25, $\times$ 1.75, $\times$ 2.5
	Young’s modulus scale	$\times$ 1.0 (nominal)	$\times$ 0.5, $\times$ 0.8, $\times$ 2.0

Table 11: Variation factors: training ranges and held-out out-of-distribution values. The nine out-of-distribution conditions are the three held-out levels of dome-light intensity, mass, and Young’s modulus.

Setting	Diffusion Policy	$\bm{\pi}_{0.5}$	FastWAM
Model and compute
Policy backbone	Image-conditioned diffusion U-Net	—	—
GPUs	$1\times$ NVIDIA A100-80GB	$8\times$ NVIDIA A100-80GB	$8\times$ NVIDIA A100-80GB
Precision	—	bfloat16 mixed precision	bfloat16
Distributed training	—	FSDP	ZeRO-1
Fine-tuning method	—	LoRA	—

Table 12: Training and inference configurations for the three baseline policy families.

Policy	Setting	Vision-only (VO)	Visuo-tactile (VT)
Diffusion Policy	Trainable parameters	$329.2$ M	$455.7$ M
	Batch size (micro / global)	$256/256$	$128/128$
	Input modalities	RGB; robot state	RGB; robot state; tactile RGB; marker motion
$\pi_{0.5}$	Trainable parameters	$467.0$ M	$538.0$ M
	Batch size (micro / global)	$32/256$	$32/256$
	Input modalities	RGB; robot state; language	RGB; robot state; language; tactile RGB; marker motion

Table 13: Differences between the vision-only and visuo-tactile policy variants. Trainable parameter counts include all optimized modules and exclude frozen model weights and preprocessing components.

		Object-Soft				Spatial-Soft
		VO-C		VT-C		VO-C		VT-C
Factor	Level	TSR	DSR	TSR	DSR	TSR	DSR	TSR	DSR
Lighting	$\times$ 0.5	0	0	1	1	0	0	19	14
	$\times$ 1.33	37	33	36	30	18	15	28	16
	$\times$ 2.0	0	0	0	0	16	9	17	13
Mass	$\times$ 1.25	41	38	42	34	5	5	18	17

Table 14: Per-condition OOD results for Diffusion Policy (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.

		Object-Soft				Spatial-Soft
		VO-C		VT-C		VO-C		VT-C
Factor	Level	TSR	DSR	TSR	DSR	TSR	DSR	TSR	DSR
Lighting	$\times$ 0.5	37	34	39	35	30	24	38	33
	$\times$ 1.33	37	34	40	33	29	25	33	28
	$\times$ 2.0	38	35	43	34	28	22	31	26
Mass	$\times$ 1.25	37	37	44	40	13	13	18	16

Table 15: Per-condition OOD results for

\pi_{0.5}

(%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.

		Object-Soft				Spatial-Soft
		VO-C		VT-C		VO-C		VT-C
Factor	Level	TSR	DSR	TSR	DSR	TSR	DSR	TSR	DSR
Lighting	$\times$ 0.5	63	63	54	54	30	29	39	37
	$\times$ 1.33	62	60	57	57	33	33	48	47
	$\times$ 2.0	47	45	56	56	33	33	45	45
Mass	$\times$ 1.25	59	59	61	61	17	17	21	21

Table 16: Per-condition OOD results for FastWAM (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.

Figure 5: Illustrative rollouts on three deformable objects across Object-Soft and Spatial-Soft . Each row shows synchronized third-person, wrist, and tactile observations at approach, peak interaction, and placement, followed by the normalized deformation trace $R_{t}$ . Circle, diamond, and square indicate the displayed steps. All rollouts achieve task success (TSR $=1$ ). Blue trajectories satisfy the calibrated deformation tolerance (DSR $=1$ ), while orange trajectories exceed $R_{t}=1$ despite success (DSR $=0$ ). Rows are independently sampled and not matched-initial-state comparisons.

Policy	$E$ scale	Input	TSR	DSR	$R_{\max}$ median [IQR]	$N$
DP	$\times 1.0$	VO-C	42	39	0.63 [0.47, 0.75]	100
		VT-C	42	30	0.69 [0.48, 0.97]	100
	$\times 0.5$	VO-C	33	32	0.66 [0.06, 0.76]	100
		VT-C	41	30	0.76 [0.57, 0.98]	100
	$\times 0.8$	VO-C	37	31	0.63 [0.06, 0.79]	100
		VT-C	45	33	0.75 [0.54, 0.96]	100

Table 17: Completion and deformation statistics under stiffness shift on Object-Soft (rates in %).

Deformable Object Manipulation

Deformable Object Manipulation studies how robots perceive, model, and control flexible objects such as cloth, ropes, cables, and soft tissue.

See all 104 papers

Jul 5, 2026

SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

Deformable object manipulation poses challenges beyond task completion: successful execution must also maintain safe physical interaction, holding the object stably without slip or drop while avoiding excessive deformation. However, existing manipulation benchmarks are predominantly success-oriented and rarely evaluate whether a policy remains physically safe throughout execution. We present SoftVTBench, a safety-aware visuo-tactile benchmark for physically constrained deformable object manipulation. Built in Isaac Sim with finite-element-simulated deformable objects, SoftVTBench provides multi-view RGB observations, RGB tactile sensing with marker motion, proprioception, and language instructions, and defines four matched task suites over object type (deformable vs. rigid) and variation axis (object vs. spatial). It separately reports Goal Success and Safety Success; the latter additionally requires no drop and peak deformation below a calibrated object-specific threshold, measured from policy-hidden privileged Finite Element Method (FEM) states. We implement pi0.5-based baselines under this protocol. Experiments show that success-only evaluation substantially overstates policy performance, as a large fraction of goal-completing rollouts still violate physical safety. Furthermore, incorporating tactile sensing improves Safety Success (e.g., from 21.4% to 35.6% on object-centric deformable tasks) and reduces object deformation during execution, while maintaining comparable Goal Success. SoftVTBench provides a reproducible benchmark for studying visuo-tactile deformable manipulation under physical interaction constraints.

Bowen Jing, Mingxin Wang, Ruiyang Hao +15

Tuojing Intelligence · Tsinghua University · King’s College London +10

Tactile Sensing Robotics Simulation Deformable Object Manipulation Robot Safety Robotic Manipulation Robot Manipulation Benchmarks+1

Ask PDF

Apr 22, 2026

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging. This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale. To address these limitations, we introduce the VTOUCH dataset. It leverages vision based tactile sensing to provide high-fidelity physical interaction signals, adopts a matrix-style task design to enable systematic learning, and employs automated data collection pipelines covering real-world, demand-driven scenarios to ensure scalability. To further validate the effectiveness of the dataset, we conduct extensive quantitative experiments on cross-modal retrieval as well as real-robot evaluation. Finally, we demonstrate real-world performance through generalizable inference across multiple robots, policies, and tasks.

Qianxi Hua, Xinyue Li, Zheng Yan +4

Humanoid Robot (Shanghai) Co., Ltd., Shanghai, China · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · School of Astronautics, Harbin Institute of Technology, Harbin, China

Vision-Based Tactile Sensing Robot Manipulation Datasets Tactile Sensing Bimanual Robotic Manipulation Tactile Sensing for Robotic Manipulation

Ask PDF

Sep 14, 2026

Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands

Tactile sensing provides contact information that can be difficult to infer from vision alone, but tactile hardware for dexterous hands has not converged to a common design. Dexterous hands differ in finger structure, contact surfaces, and sensor layouts, while simulated tactile signals still differ from measurements produced by physical sensors. These factors make it difficult to study visuo-tactile manipulation across diverse dexterous hands within a consistent experimental setting. We present Bench2Dex, a simulation benchmark for visuo-tactile bimanual manipulation across 12 dexterous hands. We adapt existing robot models with a shared simulated tactile interface that converts local contact geometry into image-like tactile observations. The interface provides a consistent observation format across different hand morphologies without attempting to reproduce the output of a specific physical tactile sensor. Bench2Dex includes 26 bimanual manipulation tasks that involve tool use, articulated-object interaction, and multi-stage manipulation, together with about 1.3K human-teleoperated demonstrations. The benchmark provides synchronized visual, tactile, proprioceptive, action, and object-state observations, together with executable task metrics. For robustness, we group seven perturbation types into invariance axis, where the correct action does not change, and equivariance axis, where the correct action changes together with the perturbation. We evaluate ACT, Diffusion Policy, pi0.5, and GR00T N1.5 on Bench2Dex and report their performance and failure modes. Bench2Dex is meant as a platform for studying visuo-tactile learning across dexterous hands. It does not assume that simulated tactile observations can replace real tactile sensing; it offers a shared setting for algorithm development while tactile hardware and simulation models are still evolving.

Zhenjie Yang, Yideng Zhang, Dongjie Zhang +21

The University of Hong Kong · Shanghai Jiao Tong University · Fudan University +4

Dexterous Manipulation Bimanual Robotic Manipulation Robotics Simulation Robotic Manipulation Robot Manipulation Benchmarks Tactile Sensing for Robotic Manipulation

Ask PDF

Abstract

Deformable Object Manipulation

Explore similar work

SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands