ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction

Yuxiang Ma; Weihan Xu; Pengfei Ye; Anna Mei; Yuxin Ray Song; Tingle Li; Haonan Chen; Edward H. Adelson

cs.ROOct 16, 2026

ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction

Authors: Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson

Organizations: Massachusetts Institute of Technology · University of Washington · Harvard University · University of California, Berkeley ·

Abstract

Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.

Figures & tables

Figure 1: Force-informed sensor actions. (a) Top: scene view, with the sensor of (b) boxed. Estimated normal force $f$ shifts the sensor target by $k_{f}f\,\mathbf{n}_{s}$ along the gel normal, so each action includes $k_{f}\Delta f\,\mathbf{n}_{s}$ (Eq. 3 ). (b) On a held-out PushT sequence, the force-conditioned model forms the tactile imprint at contact, while the pose-only model does not. Tactile crops are shown at 2 $\times$ contrast.

Figure 2: Overview of ACTS. Frozen encoders produce observation latents, whose spatial patches are mapped to tokens by input projectors $P^{\mathrm{in}}$ . A generator initialized from pretrained Cosmos3 weights jointly processes noisy future tokens conditioned on clean history, force-aware sensor actions, and view-aligned actions. Output projectors $P^{\mathrm{out}}$ map target-token features to latent velocity predictions. The scene and two wrist streams share the RGB projector pair $(P_{V}^{\mathrm{in}},P_{V}^{\mathrm{out}})$ , while both tactile streams share $(P_{T}^{\mathrm{in}},P_{T}^{\mathrm{out}})$ . Sampling updates only future latents, which frozen decoders reconstruct into synchronized multimodal predictions.

Figure 3: Qualitative results on a held-out PushT window. Given eight observed frames (left) and the recorded sensor actions, ACTS predicts the next eight frames of the scene view, both wrist views, and both GelSight streams. Ground truth in blue boxes, predictions in orange boxes. Videos and results on all tasks: https://acts-anon.github.io/ .

		Video		Wrist (L)		Wrist (R)		Tactile (L)		Tactile (R)
Task	Metric	Copy	Ours	Copy	Ours	Copy	Ours	Copy	Ours	Copy	Ours
Rope	PSNR $\uparrow$	25.10	28.92	17.66	22.30	19.66	23.70	44.74	46.08	46.46	46.51
	SSIM $\uparrow$	0.848	0.907	0.657	0.769	0.704	0.796	0.986	0.988	0.990	0.990
	LPIPS $\downarrow$	0.115	0.045	0.261	0.131	0.234	0.129	0.033	0.021	0.027	0.020
PushT	PSNR $\uparrow$	25.21	29.69	19.32	22.86	19.20	22.76	46.24	46.42	48.73	48.37
	SSIM $\uparrow$	0.861	0.925	0.762	0.834	0.766	0.844	0.986	0.988	0.991	0.991

Table 1: Short-horizon prediction quality over 8 frames (

0.53

s). Copy repeats the last observed frame across the prediction window. ACTS consistently improves the camera streams, while high temporal persistence makes full-frame tactile metrics less discriminative.

Task	Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Rope	ContactWorld ( 2026 )	$26.03$	$0.846$	$0.164$
	VT-WM-style ( 2026 )	$28.05$	$0.8878$	$0.1118$
	Ours	28.92	0.907	0.045
PushT	ContactWorld ( 2026 )	$26.35$	$0.858$	$0.139$
	VT-WM-style ( 2026 )	$27.69$	$0.8916$	$0.1047$
	Ours	29.69	0.925	0.038

Table 2: Scene-view prediction on held-out human data over 8 frames (

0.53

s). Both baselines are trained from scratch on React. Best results per object are shown in bold.

		Video		Wrist (L)		Wrist (R)		Tactile (L)		Tactile (R)
Task	Metric	Copy	Ours	Copy	Ours	Copy	Ours	Copy	Ours	Copy	Ours
Rope	PSNR $\uparrow$	31.57	32.31	27.29	27.88	27.70	27.28	42.74	42.56	44.73	44.65
	SSIM $\uparrow$	0.944	0.942	0.827	0.845	0.838	0.856	0.980	0.980	0.986	0.986
	LPIPS $\downarrow$	0.021	0.022	0.055	0.055	0.054	0.057	0.043	0.029	0.040	0.025
PushT	PSNR $\uparrow$	27.75	29.85	25.84	26.71	23.96	25.62	45.66	44.15	45.49	44.25
	SSIM $\uparrow$	0.915	0.924	0.850	0.863	0.827	0.853	0.980	0.982	0.984	0.984

Table 3: Zero-shot short-horizon prediction on robot teleoperation (8 frames,

0.53

s), scored against VAE-reconstructed ground truth. Copy repeats the last observed frame; slow robot motion makes it a strong baseline.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Human data collection setup. Left: the workspace and three fixed cameras. Top right: the two handheld devices and their sensing hardware. Bottom: the motherboard, PushT block, and rope used for collection.

Object	Recordings	Segments	Duration (min)
Motherboard	12	25	93.9
PushT	30	92	96.4
Rope	18	70	104.1
Total	60	187	294.4

Appendix

Table 4: Dataset statistics after curation. Duration is computed from the number of aligned time steps at 30 Hz and counts each bimanual interaction once.

Figure 5: Zero-shot rollout on robot teleoperation. A held-out robot PushT window in which the sensor breaks contact with the block, predicted without fine-tuning by the model trained on human data. Given eight observed frames (left) and the robot’s sensor actions, ACTS predicts the next eight frames of the scene view, both wrist views, and both GelSight streams. Ground truth in blue boxes, predictions in orange boxes. The scene and left tactile predictions follow the recording as the sensor lifts off. In the left wrist view, the recorded frames barely change, while the predicted block face grows and shifts from $t{+}4$ onward. More robot rollouts: https://acts-anon.github.io/ .

Task	Episodes			Windows			Steps/epoch
Task	Train	Val	Test	Train	Val	Test	Steps/epoch
PushT	56	4	8	55,199	4,439	5,756	6,900
Rope	53	8	9	77,188	6,363	8,468	9,648
Motherboard	21	2	5	71,386	5,411	16,699	8,923

Appendix

Table 5: Dataset splits and training sizes. Splits are defined by complete source episodes. Window counts correspond to examples used for training and evaluation.

Figure 6: Effect of the force term on tactile prediction. A held-out PushT window in which the right sensor touches the block. Rows: ground truth (blue), force-informed actions (orange), pose-only actions (gray) for the right wrist and GelSight streams; numbers are per-frame PSNR. Bottom: recorded normal force (black, left axis) and per-step action along the gel normal $\mathbf{n}_{s}$ (Eq. 3 ; right axis, broken), pose-only (gray) and force-informed (orange), with the force increment $k_{f}\,\Delta f$ above each orange bar; the dotted line marks contact. The force term adds 1.2 mm over the window, and only the model that receives it forms the imprint.

Task	Force	Stream	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Rope	–	Video	$28.50$	$0.9021$	$0.0495$
		Wrist (L)	$21.68$	$0.7583$	$0.1444$
		Wrist (R)	$23.62$	$0.7966$	$0.1353$
		Tactile (L)	$45.90$	$0.9874$	$0.0230$
		Tactile (R)	46.30	$0.9898$	$0.0188$
	$\checkmark$	Video	$28.56$	$0.9028$	$0.0490$

Appendix

Table 6: Ablation of force information in action conditioning. All observation streams are retained. A checkmark indicates that force information is included in the action representation. Within each dataset, bold values indicate a significant difference between the force and no-force configurations for the same observation stream (paired

t

-test,

p<0.05

). All configurations are evaluated after 10 training epochs for a fair comparison.

Tactile World Models

Tactile World Models predict how touch and contact change under robot actions to support planning and dexterous manipulation.

See all 29 papers

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
Jul 24, 2026Yunao Huang, Shiyu Sang, Suting Ni +6World Models for Robotics Contact-Rich Robotic Manipulation
ShanghaiTech University · InstAdapt
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
Mar 19, 2026Yuhang Zheng, Songen Gu, Yupeng Zheng +11Contact-Rich Robotic Manipulation Tactile World Models
National University of Singapore · Fudan University · CASIA +4
Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
Jun 7, 2026Yunfan Lou, Yifan Ye, Yankai Fu +7World Models for Robotics Contact-Rich Robotic Manipulation
Peking University Beijing, China · Nanjing University Nanjing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Beijing, China

Jul 24, 2026J/K move · Enter open · S save

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Yunao Huang, Shiyu Sang, Suting Ni, Haotao Lu+5

ShanghaiTech University · InstAdapt

World Models for Robotics Contact-Rich Robotic Manipulation World Models Synthetic Data Generation for Robotics Tactile Sensing for Robotic Manipulation Tactile World Models

Open paper AskPDF arXiv

Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/

Abstract

Tactile World Models

Explore similar work

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation