HapticWorld: an Interactive World Simulator with Real-time Torque Feedback

Shaoting Peng; Litian Liang; Yixuan Wang; Ming Yang; Katherine Driggs-Campbell; Mark Cutkosky; James Jingxi Xu

cs.ROSep 25, 2026

HapticWorld: an Interactive World Simulator with Real-time Torque Feedback

Authors: Shaoting Peng, Litian Liang, Yixuan Wang, Ming Yang, Katherine Driggs-Campbell, Mark Cutkosky, James Jingxi Xu

Organizations: Ant Group · Coordinated Science Laboratory, Electrical and Computer Engineering, University of Illinois Urbana-Champaign · NVIDIA · Mechanical Engineering, Stanford University

Abstract

Contact-rich manipulation depends on force sensing that is hard to infer from visual signals alone, both for collecting demonstrations and for training policies. Force-annotated data, however, remains hard to obtain at scale: real-robot collection ties every demonstration to physical hardware, physics simulators report contact forces that deviate systematically from real measurements, and learned world simulators, though scalable and realistic, are vision-only, so operators feel nothing during data collection and the data carries no force/torque (F/T) labels. We present HapticWorld, an interactive world simulator that predicts joint torque together with observations and renders it back to the operator in real time, closing the haptic loop between a human and a learned world model. Across three contact-rich tasks, torque feedback raises data collection throughput by 1.6 times on average. Policies trained on HapticWorld-generated demonstrations succeed in 54/60 real-world trials, approaching the 56/60 upper bound of real-world data, and far exceeding the 19/60 success rate of the vision-only baseline. Moreover, the success rates measured inside HapticWorld closely match real-world evaluation, demonstrating that HapticWorld can serve as a stand-alone F/T-conditioned policy evaluation platform.

Figures & tables

Fig. 2: HapticWorld Model Architecture. At each step, the observation history is encoded into latents and fed to the consistency dynamics model together with the operator’s commanded joint positions $a_{t-1}$ and the noise level. The model generates the next latent, which is decoded into the next observation $\hat{o}_{t}$ ; the torque head reads the widest intermediate features from the middle of the block stack and maps them to the joint torque $\hat{\tau}_{t}$ , which is rendered back to the leader arm.

Fig. 3: Hardware setup and tasks. (a) Play data is collected by teleoperating the follower arm with a synchronized leader arm, recording the head camera view, the commanded joint positions, and the measured joint torques. (b–d) The three evaluated tasks: open the microwave, wipe the marks on the whiteboard, and flip the box against a fixture. Cyan arrows indicate the equivalent Cartesian force corresponding to the joint-torque feedback.

Fig. 4: Torque prediction on held-out validation episodes. Predicted (dashed orange line) and measured (solid blue line) torque for three representative joints on each task.

Fig. 5: Real-world rollouts of policies trained without and with torque modality, using data collected from IWS and HapticWorld, respectively. Left: failure modes of policies trained without torque: the gripper grasps the microwave handle too weakly and slips, leaving the door unopened (Task 1); the eraser is not pressed hard enough, leaving marks on the whiteboard (Task 2); and the box is not pushed firmly enough and drops back during flipping (Task 3). Right: policies trained with HapticWorld data apply sufficient and consistent force and complete all three tasks. Cartesian force traces derived from the joint torques are shown below each rollout, with the cyan shading marking the phases where torque matters most, and colored vertical lines corresponding to each frame shown above.

Collection condition	Microwave Opening	Whiteboard Wiping	Box Flipping
With haptic feedback	26	19	22
Without haptic feedback	21	8	14
Relative throughput	1.2 $\times$	2.4 $\times$	1.6 $\times$

TABLE I: Demonstrations collected in 10 minutes of teleoperation inside HapticWorld.

Data source	IWS [ 17 ]	HapticWorld w/o feedback	Real-world data	HapticWorld (Ours)
Task 1 – Microwave opening	$10/20$	$14/20$	$19/20$	${19/20}$
Task 2 – Whiteboard wiping	$1/20$	$5/20$	$20/20$	${19/20}$
Task 3 – Box flipping	$8/20$	$6/20$	$17/20$	${16/20}$

TABLE II: Real-world success rates (out of 20 trials) of ACT policies trained on 50 demonstrations per configuration. Policies trained on real-world data serve as a reference upper bound.

Training data source	Real-world data		HapticWorld data
Evaluation platform	Real-world	HapticWorld	Real-world	HapticWorld
Task 1 – Microwave opening	$19/20$	$19/20$	${19/20}$	${19/20}$
Task 2 – Whiteboard wiping	$20/20$	$20/20$	${19/20}$	${20/20}$
Task 3 – Box flipping	$17/20$	$14/20$	${16/20}$	${14/20}$

TABLE III: Policy success rates (out of 20 trials) for torque-conditioned ACT policies trained on real-world or HapticWorld demonstration data and evaluated both in the real world and inside HapticWorld.

Haptic Feedback

Haptic feedback conveys touch, force, motion, or material properties through devices that let people and robots perceive and respond to interaction.

See all 60 papers

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
Jul 24, 2026Yunao Huang, Shiyu Sang, Suting Ni +6World Models for Robotics Contact-Rich Robotic Manipulation
ShanghaiTech University · InstAdapt
WorldContact: A Contact-Centric World Model for Scalable Robot Learning
Sep 17, 2026Caoliwen Wang, Mengdi Wang, Heng Zhang +7Robot Policy Adaptation World Models for Robotics
University of British Columbia · Westlake University · Style3D Research
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
Oct 16, 2026Yuxiang Ma, Weihan Xu, Pengfei Ye +5Tactile World Models Contact-Rich Robotic Manipulation
Massachusetts Institute of Technology · University of Washington · Harvard University +2

Jul 24, 2026J/K move · Enter open · S save

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Yunao Huang, Shiyu Sang, Suting Ni, Haotao Lu+5

ShanghaiTech University · InstAdapt

World Models for Robotics Contact-Rich Robotic Manipulation World Models Synthetic Data Generation for Robotics Tactile Sensing for Robotic Manipulation Tactile World Models

Open paper AskPDF arXiv

Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/

Abstract

Haptic Feedback

Explore similar work

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

WorldContact: A Contact-Centric World Model for Scalable Robot Learning

ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction