Organizations: Institute of Automation, Chinese Academy of Sciences · Anyverse Robotics · Open-Phoenix Team · National University of Singapore · Xi’an Jiaotong University · Beijing Institute of Technology
Evaluating and post-training robot policies through physical interaction is costly and difficult to scale, motivating action-conditioned world models as interactive simulators. However, visually plausible predictions can misrepresent contact dynamics, providing unreliable feedback for policy evaluation and learning. We introduce PhysCanvas, a state-consistent visuo-tactile world model for contact-rich robot manipulation. PhysCanvas unifies multi-view RGB, tactile images, marker displacement, task state, and actions within a unified vision space. All modalities share a video VAE and pretrained generative backbone, allowing optical touch to reuse visual and temporal priors. Motion images derived through forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. Our consistency-aware post-training uses the frozen Scene Reward Model, Modality Reward Model, and State Reward Model to supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes. The resulting world model supports closed-loop policy rollouts for evaluation and provides imagined interactions and rewards for GRPO policy updates. Across four real-world manipulation tasks, when policies are rolled out in the world model, State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model further increases π0.5's mean real-world success rate from a 59.1% SFT baseline to 70.8%. These results demonstrate the value of unified multimodal modeling and consistency-aware post-training for robot policy evaluation and learning. Project page: https://zebinx.github.io/PhysCanvas_HomePage/.
Figures & tables
Figure 1: PhysCanvas as a multimodal simulator. Action-conditioned visual–tactile rollouts support policy evaluation and learning.
Figure 2: Task overview. Real-world executions and corresponding tactile observations.
Figure 3: Overview of PhysCanvas . A unified canvas supports action-conditioned multimodal generation, with scene, state, and modality consistency guiding post-training.
Model
PSNR ↑
SSIM ↑
LPIPS ↓
SD ft-MSE †
30.73
0.9020
0.0600
SDXL fp16-fix †
32.28
0.9226
0.0399
UniT
31.72
0.9613
0.0210
Wan VAE †
35.51
0.9539
0.0139
Table 1: Tactile image reconstruction on the USB 3D-pose dataset released with UniT ( Xu et al., 2025 ) . † : zero-shot, without tactile fine-tuning.
Figure 4: Multimodal VAE reconstruction. Inputs (top) and reconstructions (bottom).
Figure 5: Evolution of multimodal token distributions. t-SNE embeddings of multimodal tokens at (a) the VAE output and after Transformer blocks (b) 1, (c) 15, and (d) 30. Colors denote camera RGB, tactile image, marker displacement, and task state.
Figure 6: Action intervention. Forced-open gripper predicts a falling block.
Figure 7: Scene and modality consistency. Left: temporal drift without Scene Reward Model supervision. Right: Rubik’s Cube RGB, tactile image, and marker displacement before and after Modality Reward Model supervision; red boxes mark discrepancies.
Method
Test Tube
Bottle
Blocks
Rubik’s Cube
FM
π0.5
FM
π0.5
FM
π0.5
FM
π0.5
SFT
0.0 ± 0.0
50.0 ± 7.0
20.0 ± 4.0
88.3 ± 4.7
25.0 ± 10.8
46.6 ± 2.3
0.0 ± 0.0
51.6 ± 6.2
RL in WM
0.0 ± 0.0
60.0 ± 4.0
33.3 ± 2.3
90.0 ± 0.0
33.3 ± 8.5
56.6 ± 4.7
0.0 ± 0.0
63.3 ± 4.7
RL in WM+RM
0.0 ± 0.0
60.0 ± 8.1
35.0 ± 4.0
86.6 ± 2.3
41.6 ± 6.2
63.3 ± 4.7
0.0 ± 0.0
73.3 ± 4.7
Table 4: Real-world policy success rates (%). Values are mean ± population standard deviation over three runs; bold marks the highest nonzero mean per column (including ties).
Figure 8: Effect of state supervision on policy evaluation. Each point compares world-model rollout and real-world success rates for one evaluation batch. Dashed blue and solid orange lines are task-wise fits for M2 and M3 , respectively; the dotted diagonal denotes ideal agreement.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Hardware layout. The dual-arm platform, three camera views, and fingertip tactile sensors.
Figure 11: Four bimanual tasks. Object photographs, rollout stages, and paired tactile-image/marker-displacement examples. L/R denotes the gripper and 0/1 its pad; green arrows show recorded displacement at 5× display scale.
Task
Training episodes
Test episodes
Test Tube
1,000
80
Blocks
1,000
80
Bottle
926
80
Rubik’s Cube
879
80
Total
3,805
320
Appendix
Table 5: Policy-rollout training and test data used by the world model.
Figure 12: Task-state canvas encoding. Each row shows state 0 (the success condition is not satisfied) and state 1 (the success condition is satisfied) for one task. Task-specific color identifies the task, while vertical and horizontal low-frequency bands distinguish states 0 and 1, respectively. For Blocks, partial completion remains state 0; state 1 requires both blocks to be placed.
Figure 13: Action projection. Projected URDF wireframes over the fixed and wrist views.
Figure 14: Tactile data distributions. Unified percentile-box summaries from sampled rollouts.
Figure 15: World-model data and interaction loop. Policy rollouts are processed offline into aligned action-image and visual–tactile training data. The trained world model then interacts with a policy for RL or evaluation.
Figure 16: Policy architecture overview. All policies use synchronized multi-view RGB, marker displacement, and robot state, but fuse these modalities differently. FM and ACT are task-specific; π0.5 is prompt-conditioned and shared across tasks.
Reward model
Input and question
Data provenance
State Reward Model
Three RGB views; is the task complete?
Human-labeled real rollouts with frame-level completion transitions.
Scene Reward Model
Initial GT and current WM views; is the scene preserved?
Human-labeled GT–WM pairs, including preserved and drifted frames.
Modality Reward Model
RGB, tactile image, and marker displacement; do modalities agree?
Aligned real bundles, controlled tactile mismatches, and labeled WM generation failures.
Appendix
Table 6: Reward-model supervision. “WM” denotes world-model-generated observations and “GT” denotes real observations.
1:
Y←[],q←q0,r←o0
2:
repeat
3:
o←Adapt(r,q)
4:
A←FirstT−∣Y∣(PolicyStep(π,o,ξ))
5:
for each S∈Split8(A) do
6:
C←Render(q,S)
7:
O←WM(o0cam,r,C,ξ)
Appendix
Table 23
Figure 17: Policy GRPO in PhysCanvas . Eight sampled trajectories share one real reset condition; three representative futures are shown. The frozen world and reward models provide group-relative advantages, while replayed actions provide current policy log-probabilities for the clipped update.
Parameter
Value
Parameter
Value
Group size
8 trajectories
Actor global batch
128
Rollout epochs/update
4
Trajectories/update
512
Micro batch
8
Actor update epochs
1
Learning rate
5×10−6
Adam (β1,β2)
(0.9,0.95)
Adam ϵ
10−8
Weight decay
0.01
Gradient clipping
1.0
Ratio clipping
[0.8,1.28]
Appendix
Table 7: Common GRPO configuration for FM and π0.5 .
Figure 18: Qualitative comparison across four tasks. GT and model rollouts are shown at aligned timestamps.
Figure 19: Layer-wise multimodal attention for Bottle case 01. Off-diagonal entries show cross-modal routing, while diagonal entries show within-modality attention.
Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/
Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.
Yuxiang Ma, Weihan Xu, Pengfei Ye +5
Massachusetts Institute of Technology · University of Washington · Harvard University +2
Evaluating robot manipulation policies through repeated physical trials is costly. We study whether an action-conditioned world model can serve as a closed-loop test environment whose policy success rates track those on real robots. Beyond realistic video, the model must return the camera views and proprioceptive state needed for the policy's next action and preserve failed-action outcomes to avoid overestimating policy capability. We introduce ProWorld, a world model that jointly predicts multi-view video and robot state, then feeds both back to a frozen policy for repeated interaction. ProWorld comprises a shared transformer for joint video-state flow prediction and Shared-Time View Encoding to distinguish cameras while preserving temporal synchronization. To reduce interaction hallucinations that turn missed grasps into successful lifts, we augment training with failed interactions collected in high-fidelity simulation, reducing inflated success rates in the generated environment. ProWorld achieves the highest overall TriWorldBench score among the compared methods. Across three policies and four local manipulation tasks, it attains a Pearson correlation of 0.95 and a Spearman correlation of 0.94 between generated and real success rates. These results support joint visual-proprioceptive feedback and failure-aware training as a basis for world-model-based policy evaluation.
Zijun Wang, Tianyi Yan, Haokun Lin +12
Sun Yat-sen University · University of Macau · Mohamed bin Zayed University of Artificial Intelligence +2