DepthWorld: 3D World Model for Robot Manipulation

Jai Bardhan; Josef Sivic; Vladimir Petrik

cs.ROOct 6, 2026

DepthWorld: 3D World Model for Robot Manipulation

Authors: Jai Bardhan, Josef Sivic, Vladimir Petrik

Organizations: Czech Institute of Informatics, Robotics and Cybernetics Czech Technical University in Prague

Abstract

World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

Figures & tables

Figure 1: DepthWorld: The RGB-only baseline fails to maintain the shape of the cutlery, and the object disappears. DepthWorld capitalizes on the depth inputs to accurately maintain the cutlery’s shape throughout the rollout while producing high-quality metric depth maps.

Figure 2: DROID-3D calibration pipeline. Given raw stereo from two external cameras and a wrist-mounted camera together with the robot’s URDF and joint encoder readings, Stage 1 recovers per-view metric depth from each stereo pair via a learned global-matching network and uses dense correspondences to enforce multi-view consistency. Stage 2 renders the URDF into each external view to ground the rig to the robot frame, then forms a joint factor graph that pools all episodes of a robot to recover per-scene extrinsic corrections ( $\delta T_{\text{ext1}}^{s}$ , $\delta T_{\text{ext2}}^{s}$ ) alongside shared kinematic parameters (hand-eye correction $\delta T_{\text{wrist}}$ , joint encoder offsets $\delta\bm{q}$ ).

Figure 3: DepthWorld Architecture. Multi-view RGB and depth are independently encoded into a spatially tiled latent grid ( $72\times 80$ ) for joint U-Net denoising, preserving pretrained SVD priors. Predicted future latents ( $x_{0,f}$ ) pass through a DPT head to predict a 3D point map ( $X,Y,Z$ ) and confidence logit in the robot base frame, supervised by a robust geometric loss.

	All scenes ( $n=995$ )		PointWorld-converged subset ( $n=815$ )
Metric	PointWorld [ 14 ]	Ours	PointWorld [ 14 ]	Ours
EE median (px) $\downarrow$	5.84	0.25	5.67	0.25
WE median (px) $\downarrow$	13.19	1.03	12.52	1.02
RD median (mm) $\downarrow$	111.3	14.0	104.6	13.9
Mask IoU mean $\uparrow$	0.440	0.810	0.458	0.819

Table 1: Calibration quality on

\sim

1,000 evaluation scenes. Blue rows: 2D multi-view consistency (medians on joint-MAGSAC gold inliers). Amber rows: 3D grounding to the robot frame. The right block restricts to the subset where the baseline’s per-scene refinement converges.

Panel of Figure 4 — Figure 4: Qualitative Result: We show two rollouts by the baseline RGB model and DepthWorld, which also produces high-quality depth. Left: The RGB-only model fails to simulate the robot grasping the red cup as in the GT trajectory. DepthWorld accurately simulates the interaction and produces an aligned depth map. Right: Without depth supervision, the RGB-only world model cannot distinguish the open wardrobe door from a solid surface and simulates the arm colliding with it. DepthWorld accurately generates the arm going inside the wardrobe by accounting for the depth of the arm and the door. See the project website for additional results, including videos.

		RGB quality			Depth quality
View	Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}{\uparrow}$
External	RGB	22.63	0.8170	0.0930	—	—	—
	RGB+D	24.09	0.8333	0.0889	0.0765	0.220	0.9432
	RGB+D PM-DPT	24.11	0.8333	0.0891	0.0782	0.219	0.9434
Wrist	RGB	16.98	0.5737	0.3228	—	—	—
	RGB+D	17.96	0.6106	0.3137	0.2230	0.150	0.8182

Table 2: Ablation of world model prediction quality on held-out trajectories. Depth metrics are against obtained depth from Sec. 3 . The RGB-only baseline does not predict depth (—). Joint depth prediction (RGB+D) improves all RGB metrics on both view classes; the point-map auxiliary loss (PM-DPT) further refines depth accuracy while maintaining RGB generation quality, specially on wrist metrics.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: ZED SDK vs S 2 M 2 . The classical ZED ULTRA algorithm fails completely in the wrist cam view and produces sparse depth for the external views. ZED NEURAL produces comparatively denser results but fails at the boundaries and edges of objects. S 2 M 2 produces denser and sharper results and outperforms both the alternatives.

Figure A2: Depth Anything 3 (DA3) vs S 2 M 2 . DA3 produces generates noisy and overly smoothed depth maps, which fail to capture the details of the objects. The two methods are visualized with a different colormap (depth for DA3 vs disparity for S 2 M 2 ).

Figure A3: Foundation Stereo vs S 2 M 2 . We find that S 2 M 2 produces more stable results. Notice the missing gripper on the left image and the higher quality details of the objects within the cup in the right image.

Panel of Figure 10 — Figure A4: Correspondence quality across feature matchers . Each panel overlays the correspondences recovered by one matcher on a wrist–external camera pair; green points are MAGSAC [ 1 ] fundamental-matrix inliers. The dense matcher RoMa-v2 (a) recovers accurate correspondences across the extreme viewpoint change, whereas the sparse matchers (b–d) collapse to a handful of matches or fail outright on the textureless and specular regions that dominate the scene.

Panel of Figure 11 — Figure A5: Correspondence quality across feature matchers (continued). Panels and color coding as in Fig. A4 : (a) RoMa-v2, (b) LoMa, (c) SuperPoint + LightGlue, (d) SIFT + LightGlue; green points are MAGSAC inliers. RoMa-v2 again yields dense, accurate matches where the sparse matchers largely fail on the wrist–external pair.

Panel of Figure 12 — Figure A6: Correspondence quality across feature matchers (continued). Panels and color coding as in Fig. A4 : (a) RoMa-v2, (b) LoMa, (c) SuperPoint + LightGlue, (d) SIFT + LightGlue; green points are MAGSAC inliers. The gap is consistent across scenes—RoMa-v2 sustains coverage in the wrist view while the other matchers degrade.

Figure A7: RoMa-v2 vs others summary. Left: We show the number of usable correspondences (inliers from MAGSAC) for each method. RoMa-v2 consistently produces an order of magnitude more correspondences. Right: We show the coverage of each method. RoMa-v2 has approximately

2\times

the coverage on the extreme viewpoint pair of wrist and external cameras.

Figure A8: Effect of cropping the render and the real image before matching with LoMa-G. The yellow box indicates the crop region. The number of matches goes up from $92\rightarrow 277$ . Green dots indicate matches that are on the surface of the robot, red dots are for points that do not lie on the surface.

Figure A9: Visualization of the PCA of the DINOv2 features and the LoMa features. Here we show a comparison between the patch level features of the actual image, cropped around the robot arm, and a similar render of the robot URDF. The patch level features for both DINOv2 and LoMa match at the robot arm. However, the patch level features from LoMa contain more fine-grained details, since LoMa was specifically tuned for matching.

Weight tweaks
Full model $\pm$	EE $\downarrow$	WE $\downarrow$	RD $\downarrow$	IoU $\uparrow$
	(px)	(px)	(mm)	(mean)
Full (all factors)	0.251	1.032	14.15	0.808
$+$ amplify 3D	0.251	1.027	14.02	0.808
$+$ amplify FM	0.250	1.038	14.09	0.804
Leave-one-out (drop one factor)

Appendix

Table A1: Ablation relative to the full model on

\sim

1,000 scenes. All rows are the full model with the listed modification. Blue: 2D consistency (gold-inlier median px). Amber: 3D grounding.

+

AMP 3D,

-

FM is the deployed model.

Figure A10: Qualitative Results of the pose estimation of PointWorld [ 14 ] based baseline vs Ours. In all the cases our method produces significantly more aligned poses. The average robot depth error for PointWorld based approach on these examples is around 110-130mm while ours is around 15-20mm. We show only the points corresponding to segmented robot pixels in the image. In all the cases, it is clear that poses from the baseline method do not align, while they do for ours. Minor cleaning has been done to remove floaters due to imperfect segmentation.

		RGB quality			Depth quality
View	Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}$ $\uparrow$
External	Dual-branch	21.94	0.8039	0.1010	0.0861	0.232	0.9383
	Channel expansion	22.22	0.8017	0.1021	0.0796	0.220	0.9414
	Spatial tiling (ours)	23.66	0.8252	0.0945	0.0854	0.235	0.9383
Wrist	Dual-branch	16.05	0.5441	0.3554	0.2523	0.161	0.7685
	Channel expansion	16.34	0.5593	0.3422	0.2801	0.158	0.7930

Appendix

Table A2: Ablation: Spatial latent tiling vs. other integration architectures , at

\sim 70

k training steps under the autoregressive rollout protocol (Appendix D.3 ). All are joint RGB–depth models; depth metrics are against the VAE-round-tripped GT (Appendix C.8 ). Spatial tiling matches or beats both the baseline integration architectures architecture on nearly every metric and view despite leaving the U-Net unmodified. Best per view/metric in bold , second best underlined .

	External				Wrist
$K{=}10$ rollout	PSNR $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$
RGBD, S 2 M 2 depth (ours)	24.07	0.0877	0.074	0.946	17.68	0.3270	0.266	0.797
RGBD, ZED NEURAL depth	24.26	0.0881	0.079	0.939	17.77	0.3260	0.254	0.794
RGBD, ZED ULTRA depth	22.11	0.0947	0.194	0.830	16.28	0.3395	0.461	0.655

Appendix

Table A3: Ablation: Depth source used to train DepthWorld , at

50

k training steps under the autoregressive rollout protocol (Appendix D.3 ). All are joint RGB–depth models with identical architecture and training protocol; only the depth maps used for training differ. Best per view/metric in bold , second best underlined .

	External				Wrist
$K{=}10$ rollout	PSNR $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$
PM-DPT, factory extrinsics	24.10	0.0891	0.075	0.945	18.01	0.3131	0.233	0.817
PM-DPT, refined extrinsics	24.13	0.0890	0.075	0.945	18.03	0.3124	0.220	0.819

Appendix

Table A4: Ablation: Extrinsics used to build the point-map targets , under the autoregressive rollout protocol (Appendix D.3 ). Both are RGB+D PM-DPT models trained identically; the robot-frame point-map targets are backprojected either with DROID’s factory extrinsics or with the refined DROID-3D extrinsics of Appendix B . Best per view/metric in bold .

Figure A11: Qualitative comparison with PointWorld on a held-out window in which the robot opens a door. Top: observed external RGB at the input frame and at $2$ , $4$ , $6$ , and $8$ s. Rows 2–5: workspace point clouds in the robot frame at the same times: the ground truth lifted from $S^{2}M^{2}$ depth; PointWorld’s dense rollout seeded at $t=0$ ; the ground truth after the SVD-VAE round trip, which is DepthWorld’s own reference in Table A5 ; and DepthWorld’s predicted depth unprojected into the robot frame. PointWorld follows the arm at short horizon, but from about $4$ s its cloud scatters and drifts, consistent with its rising whole-scene Chamfer distance. DepthWorld stays coherent over the full $8$ s and reconstructs the opening door in agreement with its VAE reference, at the cost of thin structures and some foreground geometry that are present in the raw ground truth.

	Tracked-point $\ell_{2}$ , moving		Chamfer, moving		Chamfer, whole scene
Model	2 s	8 s	2 s	8 s	2 s	8 s
PointWorld [ 14 ] vs. raw GT	23	56	21	58	8	18
DepthWorld vs. VAE-GT	31	46	25	32	8	9

Appendix

Table A5: Comparison with PointWorld on the shared

256

-window split (

141

windows for the moving-point metrics,

147

for the whole scene), fair diagonal of the own-ceiling protocol: PointWorld is scored against raw ground-truth depth, DepthWorld (the RGB+D PM-DPT variant) against the VAE round-tripped ground truth. Medians over windows in mm at horizons of

2

s and

8

\ell_{2}

is averaged over the frames up to the horizon, Chamfer is evaluated at the horizon frame. Lower is better; best per column in bold .

Explore similar work

PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation
Jun 16, 2026Yuhang Huang, Xuan Lv, Junyan Xu +25Video World Models Robotic Manipulation
Institute of AI for Industries, Chinese Academy of Sciences
RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
Jul 7, 2026Haoyu Zhao, Xingyue Zhao, Siteng Huang +3Robotic Manipulation Four-Dimensional
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation
Aug 13, 2026DreamX Team, Rui Chen, Xiangxiang Chu +7Efficient World-Action Model Robotic Manipulation

Jun 16, 2026, cs.ROJ/K move · Enter open · S save

PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation

Yuhang Huang, Xuan Lv, Junyan Xu, Zhiyuan Yu+24

Institute of AI for Industries, Chinese Academy of Sciences

Video World Models Robotic Manipulation Many Worlds

Open paper AskPDF arXiv

World foundation models (WFMs) are powerful simulators, yet they predominantly operate in a single-view setting and lack the multi-view 3D consistency required for robotic manipulation. While robotic systems rely on multiple cameras (egocentric, eye-to-hand, and wrist-mounted) for policy learning, current multi-view world models simply concatenate view tokens without explicit geometric reasoning. This causes cross-view object drift, depth inconsistency, and texture misalignment. We trace these failures to two deficiencies: the absence of an explicit inter-view communication mechanism and the lack of a 3D geometric prior. We argue that resolving both simultaneously is necessary and sufficient. To address this, we present PAIWorld, a framework that augments diffusion-transformer world models via three core components: (1) Geometry-Aware Cross-View Attention blocks that establish an explicit pathway across views, (2) Geometric Rotary Position Embedding that encodes camera ray directions and extrinsic poses into the attention mechanism, and (3) Latent 3D-REPA, which distills 3D-aware features from frozen 3D foundation models to ensure 3D consistency. Built upon a DiT-based world foundation model, PAIWorld achieves state-of-the-art multi-view 3D consistency on robotic manipulation benchmarks, ranking 1st on the WorldArena leaderboard and 2nd on the AgiBot-Challenge2026 leaderboard, while enabling downstream applications such as model-based planning, world action models, and multi-view policy post-training.

Abstract

Explore similar work

PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation