Organizations: Institute of Automation, Chinese Academy of Sciences · Anyverse Robotics · Open-Phoenix Team · National University of Singapore · Xi’an Jiaotong University · Beijing Institute of Technology
Evaluating and post-training robot policies through physical interaction is costly and difficult to scale, motivating action-conditioned world models as interactive simulators. However, visually plausible predictions can misrepresent contact dynamics, providing unreliable feedback for policy evaluation and learning. We introduce PhysCanvas, a state-consistent visuo-tactile world model for contact-rich robot manipulation. PhysCanvas unifies multi-view RGB, tactile images, marker displacement, task state, and actions within a unified vision space. All modalities share a video VAE and pretrained generative backbone, allowing optical touch to reuse visual and temporal priors. Motion images derived through forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. Our consistency-aware post-training uses the frozen Scene Reward Model, Modality Reward Model, and State Reward Model to supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes. The resulting world model supports closed-loop policy rollouts for evaluation and provides imagined interactions and rewards for GRPO policy updates. Across four real-world manipulation tasks, when policies are rolled out in the world model, State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model further increases π0.5's mean real-world success rate from a 59.1% SFT baseline to 70.8%. These results demonstrate the value of unified multimodal modeling and consistency-aware post-training for robot policy evaluation and learning. Project page: https://zebinx.github.io/PhysCanvas_HomePage/.
Figures & tables
Figure 1: PhysCanvas as a multimodal simulator. Action-conditioned visual–tactile rollouts support policy evaluation and learning.
Figure 2: Task overview. Real-world executions and corresponding tactile observations.
Figure 3: Overview of PhysCanvas . A unified canvas supports action-conditioned multimodal generation, with scene, state, and modality consistency guiding post-training.
Model
PSNR ↑
SSIM ↑
LPIPS ↓
SD ft-MSE †
30.73
0.9020
0.0600
SDXL fp16-fix †
32.28
0.9226
0.0399
UniT
31.72
0.9613
0.0210
Wan VAE †
35.51
0.9539
0.0139
Table 1: Tactile image reconstruction on the USB 3D-pose dataset released with UniT ( Xu et al., 2025 ) . † : zero-shot, without tactile fine-tuning.
Figure 4: Multimodal VAE reconstruction. Inputs (top) and reconstructions (bottom).
Figure 5: Evolution of multimodal token distributions. t-SNE embeddings of multimodal tokens at (a) the VAE output and after Transformer blocks (b) 1, (c) 15, and (d) 30. Colors denote camera RGB, tactile image, marker displacement, and task state.
Figure 6: Action intervention. Forced-open gripper predicts a falling block.
Figure 7: Scene and modality consistency. Left: temporal drift without Scene Reward Model supervision. Right: Rubik’s Cube RGB, tactile image, and marker displacement before and after Modality Reward Model supervision; red boxes mark discrepancies.
Method
Test Tube
Bottle
Blocks
Rubik’s Cube
FM
π0.5
FM
π0.5
FM
π0.5
FM
π0.5
SFT
0.0 ± 0.0
50.0 ± 7.0
20.0 ± 4.0
88.3 ± 4.7
25.0 ± 10.8
46.6 ± 2.3
0.0 ± 0.0
51.6 ± 6.2
RL in WM
0.0 ± 0.0
60.0 ± 4.0
33.3 ± 2.3
90.0 ± 0.0
33.3 ± 8.5
56.6 ± 4.7
0.0 ± 0.0
63.3 ± 4.7
RL in WM+RM
0.0 ± 0.0
60.0 ± 8.1
35.0 ± 4.0
86.6 ± 2.3
41.6 ± 6.2
63.3 ± 4.7
0.0 ± 0.0
73.3 ± 4.7
Table 4: Real-world policy success rates (%). Values are mean ± population standard deviation over three runs; bold marks the highest nonzero mean per column (including ties).
Figure 8: Effect of state supervision on policy evaluation. Each point compares world-model rollout and real-world success rates for one evaluation batch. Dashed blue and solid orange lines are task-wise fits for M2 and M3 , respectively; the dotted diagonal denotes ideal agreement.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Hardware layout. The dual-arm platform, three camera views, and fingertip tactile sensors.
Figure 11: Four bimanual tasks. Object photographs, rollout stages, and paired tactile-image/marker-displacement examples. L/R denotes the gripper and 0/1 its pad; green arrows show recorded displacement at 5× display scale.
Task
Training episodes
Test episodes
Test Tube
1,000
80
Blocks
1,000
80
Bottle
926
80
Rubik’s Cube
879
80
Total
3,805
320
Appendix
Table 5: Policy-rollout training and test data used by the world model.
Figure 12: Task-state canvas encoding. Each row shows state 0 (the success condition is not satisfied) and state 1 (the success condition is satisfied) for one task. Task-specific color identifies the task, while vertical and horizontal low-frequency bands distinguish states 0 and 1, respectively. For Blocks, partial completion remains state 0; state 1 requires both blocks to be placed.
Figure 13: Action projection. Projected URDF wireframes over the fixed and wrist views.
Figure 14: Tactile data distributions. Unified percentile-box summaries from sampled rollouts.
Figure 15: World-model data and interaction loop. Policy rollouts are processed offline into aligned action-image and visual–tactile training data. The trained world model then interacts with a policy for RL or evaluation.
Figure 16: Policy architecture overview. All policies use synchronized multi-view RGB, marker displacement, and robot state, but fuse these modalities differently. FM and ACT are task-specific; π0.5 is prompt-conditioned and shared across tasks.
Reward model
Input and question
Data provenance
State Reward Model
Three RGB views; is the task complete?
Human-labeled real rollouts with frame-level completion transitions.
Scene Reward Model
Initial GT and current WM views; is the scene preserved?
Human-labeled GT–WM pairs, including preserved and drifted frames.
Modality Reward Model
RGB, tactile image, and marker displacement; do modalities agree?
Aligned real bundles, controlled tactile mismatches, and labeled WM generation failures.
Appendix
Table 6: Reward-model supervision. “WM” denotes world-model-generated observations and “GT” denotes real observations.
1:
Y←[],q←q0,r←o0
2:
repeat
3:
o←Adapt(r,q)
4:
A←FirstT−∣Y∣(PolicyStep(π,o,ξ))
5:
for each S∈Split8(A) do
6:
C←Render(q,S)
7:
O←WM(o0cam,r,C,ξ)
Appendix
Table 23
Figure 17: Policy GRPO in PhysCanvas . Eight sampled trajectories share one real reset condition; three representative futures are shown. The frozen world and reward models provide group-relative advantages, while replayed actions provide current policy log-probabilities for the clipped update.
Parameter
Value
Parameter
Value
Group size
8 trajectories
Actor global batch
128
Rollout epochs/update
4
Trajectories/update
512
Micro batch
8
Actor update epochs
1
Learning rate
5×10−6
Adam (β1,β2)
(0.9,0.95)
Adam ϵ
10−8
Weight decay
0.01
Gradient clipping
1.0
Ratio clipping
[0.8,1.28]
Appendix
Table 7: Common GRPO configuration for FM and π0.5 .
Figure 18: Qualitative comparison across four tasks. GT and model rollouts are shown at aligned timestamps.
Figure 19: Layer-wise multimodal attention for Bottle case 01. Off-diagonal entries show cross-modal routing, while diagonal entries show within-modality attention.