HapticWorld: an Interactive World Simulator with Real-time Torque Feedback
Authors: Shaoting Peng, Litian Liang, Yixuan Wang, Ming Yang, Katherine Driggs-Campbell, Mark Cutkosky, James Jingxi Xu
Organizations: Ant Group · Coordinated Science Laboratory, Electrical and Computer Engineering, University of Illinois Urbana-Champaign · NVIDIA · Mechanical Engineering, Stanford University
Contact-rich manipulation depends on force sensing that is hard to infer from visual signals alone, both for collecting demonstrations and for training policies. Force-annotated data, however, remains hard to obtain at scale: real-robot collection ties every demonstration to physical hardware, physics simulators report contact forces that deviate systematically from real measurements, and learned world simulators, though scalable and realistic, are vision-only, so operators feel nothing during data collection and the data carries no force/torque (F/T) labels. We present HapticWorld, an interactive world simulator that predicts joint torque together with observations and renders it back to the operator in real time, closing the haptic loop between a human and a learned world model. Across three contact-rich tasks, torque feedback raises data collection throughput by 1.6 times on average. Policies trained on HapticWorld-generated demonstrations succeed in 54/60 real-world trials, approaching the 56/60 upper bound of real-world data, and far exceeding the 19/60 success rate of the vision-only baseline. Moreover, the success rates measured inside HapticWorld closely match real-world evaluation, demonstrating that HapticWorld can serve as a stand-alone F/T-conditioned policy evaluation platform.
Figures & tables
Fig. 2: HapticWorld Model Architecture. At each step, the observation history is encoded into latents and fed to the consistency dynamics model together with the operator’s commanded joint positions at−1 and the noise level. The model generates the next latent, which is decoded into the next observation o^t ; the torque head reads the widest intermediate features from the middle of the block stack and maps them to the joint torque τ^t , which is rendered back to the leader arm.
Fig. 3: Hardware setup and tasks. (a) Play data is collected by teleoperating the follower arm with a synchronized leader arm, recording the head camera view, the commanded joint positions, and the measured joint torques. (b–d) The three evaluated tasks: open the microwave, wipe the marks on the whiteboard, and flip the box against a fixture. Cyan arrows indicate the equivalent Cartesian force corresponding to the joint-torque feedback.
Fig. 4: Torque prediction on held-out validation episodes. Predicted (dashed orange line) and measured (solid blue line) torque for three representative joints on each task.
Fig. 5: Real-world rollouts of policies trained without and with torque modality, using data collected from IWS and HapticWorld, respectively. Left: failure modes of policies trained without torque: the gripper grasps the microwave handle too weakly and slips, leaving the door unopened (Task 1); the eraser is not pressed hard enough, leaving marks on the whiteboard (Task 2); and the box is not pushed firmly enough and drops back during flipping (Task 3). Right: policies trained with HapticWorld data apply sufficient and consistent force and complete all three tasks. Cartesian force traces derived from the joint torques are shown below each rollout, with the cyan shading marking the phases where torque matters most, and colored vertical lines corresponding to each frame shown above.
Collection condition
Microwave Opening
Whiteboard Wiping
Box Flipping
With haptic feedback
26
19
22
Without haptic feedback
21
8
14
Relative throughput
1.2 ×
2.4 ×
1.6 ×
TABLE I: Demonstrations collected in 10 minutes of teleoperation inside HapticWorld.
Data source
IWS [ 17 ]
HapticWorld w/o feedback
Real-world data
HapticWorld (Ours)
Task 1 – Microwave opening
10/20
14/20
19/20
19/20
Task 2 – Whiteboard wiping
1/20
5/20
20/20
19/20
Task 3 – Box flipping
8/20
6/20
17/20
16/20
TABLE II: Real-world success rates (out of 20 trials) of ACT policies trained on 50 demonstrations per configuration. Policies trained on real-world data serve as a reference upper bound.
Training data source
Real-world data
HapticWorld data
Evaluation platform
Real-world
HapticWorld
Real-world
HapticWorld
Task 1 – Microwave opening
19/20
19/20
19/20
19/20
Task 2 – Whiteboard wiping
20/20
20/20
19/20
20/20
Task 3 – Box flipping
17/20
14/20
16/20
14/20
TABLE III: Policy success rates (out of 20 trials) for torque-conditioned ACT policies trained on real-world or HapticWorld demonstration data and evaluated both in the real world and inside HapticWorld.
Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/
Adapting robots to new objects and tasks requires interaction experience that can be costly to obtain. We present WorldContact, a contact-centric world model for deformable-object manipulation, constructed from a limited set of high-quality trajectories to generate additional training data efficiently. It predicts object dynamics using larger time steps than the source numerical simulator, which requires small integration steps to resolve rapid motion and prevent interpenetration. We evaluate WorldContact across 16 shopping-bag manipulation tasks. State-rollout measurements on a single H100 GPU show a 10× speedup over the source simulator, excluding rendering and disk I/O. We use the generated data to fine-tune an existing vision-language-action policy and deploy it directly on a real robot. In bag lifting, the same policy achieves 65% single-attempt success when fine-tuned on source simulation data alone, compared with 95% when fine-tuned on the dataset expanded with WorldContact. These results support efficient data generation with WorldContact for robot policy adaptation.
Caoliwen Wang, Mengdi Wang, Heng Zhang +7
University of British Columbia · Westlake University · Style3D Research
Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.
Yuxiang Ma, Weihan Xu, Pengfei Ye +5
Massachusetts Institute of Technology · University of Washington · Harvard University +2