ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations

Yiwen Liu; Yujun Zhu; Kui Jia; Zhao Liao; Yangwei You; Shuaijun Wang

cs.ROAug 21, 2026

ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations

Authors: Yiwen Liu, Yujun Zhu, Kui Jia, Zhao Liao, Yangwei You, Shuaijun Wang

Organizations: Xiaomi Robotics

Abstract

Recent vision-based action models have demonstrated strong capabilities in complex manipulation, but they rarely leverage explicit object physical properties to adapt their policies. We introduce ViTacPhys, a visual-tactile framework and data acquisition system that estimates object mass and friction-coefficient classes, together with continuous stiffness, from human manipulation demonstrations. Trained on data from 60 rigid and deformable objects, ViTacPhys combines temporal visual-tactile modeling, cross-attention multimodal fusion, and a semantic prior derived from a vision-language model. On seen objects, it achieves 97.2% mass classification accuracy, 98.8% friction-coefficient classification accuracy, and a stiffness mean absolute percentage error (MAPE) of 5.51%. On held-out objects from known categories, it achieves 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and a stiffness MAPE of 9.08%. We transfer ViTacPhys from the human domain to the robot domain using limited robot teleoperation data, robot-style video augmentation, and human demonstrations with matched actions, and deploy it as an online module for adaptive grasping. The resulting physical-property-conditioned policy achieves total grasping success rates of 95.0% on in-distribution objects and 83.4% on out-of-distribution objects. For out-of-distribution objects successfully grasped by both methods, its force profiles are more consistent with human teleoperation than those produced by ACT. These results demonstrate the feasibility of explicitly estimating and conditioning on object physical properties for real-world adaptive grasping.

Figures & tables

Fig. 1: ViTacPhys predicts object mass class, stiffness, and friction-coefficient class from visual–tactile observations and conditions a downstream adaptive grasping policy on these estimates. The examples compare grasping outcomes for visually similar objects with different physical properties against ACT and ViTacFormer [ 1 , 2 ] .

Fig. 2: Panel (a) shows the human visual–tactile data collection setup, which synchronizes wrist-view video, fingertip pressure maps, and motion-capture measurements. Panel (b) shows the everyday objects in the dataset, ranging from rigid mugs to soft rubber toys. Panel (c) shows the corresponding sensor locations on the wearable device and robot platform.

Fig. 3: Object-level label distributions in the ViTacPhys dataset. Mass and friction coefficient are discretized into three ordered classes, whereas stiffness is a continuous regression target shown on a

\log_{10}

scale. The displayed dataset-level class boundaries are fitted once using all annotated objects and fixed across evaluation splits.

Fig. 4: Overview of ViTacPhys. (A) The predictor combines temporal visual–tactile observations, flow features, and a VLM-derived semantic prior computed from pre-contact RGB frames. (B) Human-to-robot transfer uses robot teleoperation, matched-action human demonstrations, and visually augmented human demonstrations. (C) During deployment, the VLM prior is computed before contact. After contact, the system immediately begins rolling prediction from a 30-frame visual–tactile queue; unavailable initial entries repeat the earliest post-contact observation before 15 frames and 14 adjacent flow fields are sampled. Cumulative temporal voting stabilizes the predicted classes and stiffness bin before they are passed to the downstream policy.

	Mass		Stiffness						Friction Coeff.
Setting	Acc. $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	MAPE $\downarrow$	Pearson $r$ $\uparrow$	Within 10% $\uparrow$	Within 20% $\uparrow$	Acc. $\uparrow$	Macro-F1 $\uparrow$
In distribution	0.972 $\pm$ 0.008	0.970	0.248 $\pm$ 0.032	0.386	5.51 $\pm$ 0.45	0.980	84.22	96.56	0.988 $\pm$ 0.005	0.984
Held-out-object	0.875 $\pm$ 0.019	0.866	0.435 $\pm$ 0.127	0.577	9.08 $\pm$ 2.31	0.947	61.56	89.64	0.975 $\pm$ 0.003	0.971
One-shot	0.492 $\pm$ 0.072	0.480	0.837 $\pm$ 0.045	1.457	16.84 $\pm$ 0.95	0.680	49.47	75.19	0.569 $\pm$ 0.089	0.527

TABLE II: Physical-property prediction results averaged over three independent training runs. The main evaluation metrics are highlighted with a gray background.

	Mass		Stiffness						Friction Coeff.
Experiment	Acc. $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	MAPE $\downarrow$	Pearson $r$ $\uparrow$	W10 $\uparrow$	W20 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
w/o tactile content, w text	0.660	0.648	0.756	1.152	16.70	0.769	46.39	80.69	0.915	0.905
w/o visual content, w text	0.660	0.640	0.668	0.853	13.12	0.887	45.97	79.79	0.844	0.862
w/o visual flow, w text	0.757	0.751	0.457	0.579	9.44	0.942	59.70	88.39	0.905	0.886
w/o tactile flow, w text	0.736	0.727	0.412	0.563	8.54	0.950	70.87	89.52	0.888	0.870
w/o both flow, w text	0.661	0.662	0.858	1.366	18.91	0.633	56.94	73.47	0.898	0.857

TABLE III: Input-modality ablation. Bold and underlined entries indicate the best and second-best results. “w/o” denotes without, while “w” denotes with.

	Mass		Stiffness						Friction Coeff.
Experiment	Acc. $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	MAPE $\downarrow$	Pearson $r$ $\uparrow$	W10 $\uparrow$	W20 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Manual loss weights	0.778	0.799	0.540	0.714	11.01	0.922	59.51	83.06	0.935	0.938
DWA loss weights	0.762	0.772	0.576	0.764	12.38	0.880	54.72	84.44	0.947	0.950
Cross entropy	0.635	0.657	0.534	0.730	10.07	0.945	59.38	85.14	0.979	0.978
Ours	0.875	0.866	0.435	0.577	9.08	0.947	61.56	89.64	0.975	0.971

TABLE IV: Ablation of multi-task loss weighting strategies and loss functions.

Fig. 5: Robot teleoperation setup for downstream grasping. A Manus glove controls the dexterous hand to collect demonstrations in which the robot grasps and lifts objects with property-dependent forces.

Split	Init.	Training Data			Mass		Stiffness		Friction Coeff.
Split	Init.	Tele.	Aug.	H	Acc. $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	MAPE $\downarrow$	Acc. $\uparrow$	Macro-F1 $\uparrow$
ID	Train from scratch	$\checkmark$			0.998	0.996	0.217	5.10	1.000	1.000
	Train from scratch	$\checkmark$	$\checkmark$	$\checkmark$	0.960	0.966	0.315	6.78	0.965	0.973
	Fine-tuned	$\checkmark$			0.907	0.912	0.201	4.66	0.949	0.960
		$\checkmark$		$\checkmark$	0.926	0.927	0.164	3.66	0.986	0.989
		$\checkmark$	$\checkmark$		0.922	0.908	0.171	3.97	0.950	0.962

TABLE V: Human-to-robot transfer. Bold and underlined entries denote the best and second-best results.

Fig. 6: Adaptive grasping outcomes on ID and OOD objects. “GT Properties” conditions our policy on ground-truth properties, whereas “ViTacPhys Pred.” uses predicted properties. Clean success is the primary metric. Total success is the sum of clean and over-force outcomes and is reported as lift-and-hold success.

Comparison	Split	Method	Evaluated-object ratio	Abs. Err. $\downarrow$	Pearson $r$ $\uparrow$	Pairwise $\uparrow$
ACT vs. Ours	ID	ACT [ 1 ]	$37/40$ (92.5%)	0.040	0.889	0.852
	ID	Ours	$37/40$ (92.5%)	0.035	0.862	0.843
	OOD	ACT [ 1 ]	$8/12$ (66.7%)	0.076	0.438	0.639
	OOD	Ours	$8/12$ (66.7%)	0.061	0.852	0.806
ViTacFormer vs. Ours	ID	ViTacFormer [ 2 ]	$17/40$ (42.5%)	0.077	0.503	0.733
	ID	Ours	$17/40$ (42.5%)	0.044	0.851	0.844

TABLE VI: Normalized grasping force comparison.

Fig. 7: Grasping force comparison between our method and baselines, grouped by the ground-truth mass, stiffness, and friction coefficient bins used as physical-property inputs to the downstream policy. “L”, “M”, and “H” denote low, medium, and high, respectively. For a given method, “X” indicates that all objects in the corresponding bin failed to be grasped by that method, while an empty bin indicates that no evaluated object in the corresponding split/subset belongs to that bin.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 311 papers

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
Jul 24, 2026Yunao Huang, Shiyu Sang, Suting Ni +6World Models for Robotics Contact-Rich Robotic Manipulation
ShanghaiTech University · InstAdapt
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
Mar 19, 2026Yuhang Zheng, Songen Gu, Yupeng Zheng +11Contact-Rich Robotic Manipulation Tactile World Models
National University of Singapore · Fudan University · CASIA +4
GraspTune: Tactile-Driven Execution Refinement for Robust Grasping
Sep 21, 2026Juntao Li, Xingke Xia, Sichao Liu +1Robotic Grasping Generalization in Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · KTH Royal Institute of Technology, Stockholm, Sweden

Jul 24, 2026J/K move · Enter open · S save

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Yunao Huang, Shiyu Sang, Suting Ni, Haotao Lu+5

ShanghaiTech University · InstAdapt

World Models for Robotics Contact-Rich Robotic Manipulation World Models Synthetic Data Generation for Robotics Tactile Sensing for Robotic Manipulation Tactile World Models

Open paper AskPDF arXiv

Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/

Abstract

Robotic Grasping

Explore similar work

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

GraspTune: Tactile-Driven Execution Refinement for Robust Grasping