Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.
Figures & tables
Figure 1: WorldMirror constructs simulation-ready digital twins with high visual and interaction fidelity. GPT-6 Astra is competitive in spatial structure and layout, while WorldMirror achieves better visual fidelity, collision geometry, and physical consistency through agentic reconstruction.
Capability
RoLA
RoboSnap
SimFoundry
SAGE
Astra
Agentic Real2Sim
WorldMirror
Scene Reconstruction
Scene Appearance
✓
✓
✓
✓
✗
✓
✓
Interactive Assets
✓
✓
✓
✓
✓
✓
✓
Robot–World Alignment
Camera Calibration
✗
✓
✓
✗
✓
✓
✓
Camera-Pose-Anchored World Representation
✗
✗
✗
✗
✗
✗
✓
Table 1: Capability comparison of representative real-to-sim and agentic world construction systems, covering scene reconstruction, robot–world alignment, and agentic construction.
Figure 2: WorldMirror framework. A coding agent maintains an editable world, invokes scene and object generation tools, and aligns camera, assets, and robot through assembly code. Rendered observations and execution feedback guide targeted revisions. The resulting world supports trajectory replay, policy evaluation, and simulation-data generation.
Type
Method
PSNR ↑
SSIM ↑
LPIPS ↓
DISTS ↓
Aesthetic ↑
Open-source
RoLA ( Zhao et al., 2025 )
13.53
0.587
0.506
0.206
0.480
RoboSnap ( Zhang et al., 2026 )
10.30
0.483
0.641
0.209
0.532
Proprietary
GPT-6 Astra
11.90
0.515
0.606
0.299
0.472
Ours
WorldMirror
13.57
0.570
0.427
0.168
0.570
Table 2: Visual quality comparison on 10 DROID scenes. Higher is better for ↑ metrics, while lower is better for ↓ metrics.
Figure 3: Qualitative RGB comparison with existing baselines on DROID scenes. WorldMirror produces more faithful visual reconstructions with improved appearance consistency and scene details compared with open-source and proprietary baselines.
Figure 4: Real-to-sim wrist-view comparison over time, showing visual alignment during replay.
Category
Method
IoU ↑
Pos. Err. ↓
Rel. Vec. ↓
Rel. Dist. ↓
Open-source
RoLA ( Zhao et al., 2025 )
0.232
30.65
0.814
0.542
RoboSnap ( Zhang et al., 2026 )
0.313
27.79
0.644
0.583
SimFoundry ( Ranawaka et al., 2026 )
0.169
19.54
0.627
0.507
Proprietary
GPT-6 Astra ( OpenAI, 2026 )
0.598
8.72
0.552
0.331
Ours
WorldMirror
0.469
9.66
0.557
0.219
Table 3: Object layout fidelity comparison. Higher is better for ↑ metrics, while lower is better for ↓ metrics.
Motion Replay
Physical Interaction
Method
Disp. (cm) ↓
Final (cm) ↓
Dir. ( ∘ ) ↓
DTW (cm) ↓
F1 ↑
SR ↑
RoLA ( Zhao et al., 2025 )
24.70
40.80
53.00
10.30
–
–
GPT-6 Astra ( OpenAI, 2026 )
11.17
21.98
81.30
10.82
0.653
40%
WorldMirror
1.79
5.83
10.90
1.64
0.873
100%
Table 4: Motion replay and physical interaction evaluation on DROID demonstrations. F1 denotes contact F1 score, and SR denotes replay success rate.
Strategy
Success ↑
Final (cm) ↓
Trans. (cm) ↓
Lift (cm) ↓
Dir. ( ∘ ) ↓
DTW (cm) ↓
Replay Iter. ↓
One-pass
✗
33.86
0
0
–
14.88
1
Fixed Iterative
✗
31.77
2.16
0.40
13.9
13.90
2
Full
✓
2.34
32.98
11.11
3.70
0.95
4
Table 5: Effect of verification-driven adaptive refinement on replay performance. One-pass performs reconstruction followed by a single replay attempt. Fixed Iterative repeatedly applies refinement without failure-aware diagnosis. Full WorldMirror performs structured failure diagnosis and targeted refinement.
Variant
Success ↑
Final (cm) ↓
Trans. (cm) ↓
Lift (cm) ↓
Dir. ( ∘ ) ↓
DTW (cm) ↓
Full
✓
2.34
32.98
11.11
3.70
0.95
w/o Stateful Memory
✓
2.35
33.00
11.23
3.75
0.97
w/o Adaptive Routing
✗
9.86
42.23
6.45
7.93
2.60
w/o Feedback Refinement
✗
33.86
0
0
–
14.88
Table 6: Component ablation of WorldMirror on the rail manipulation task. The ablations remove stateful memory, adaptive routing, or feedback-driven refinement. The latter corresponds to the One-pass setting, which performs only one replay without subsequent refinement.
Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on brittle workflow glue across visual perception tools and simulators: manual tuning of visual foundation models, mesh cleanup, coordinate frame alignments, etc. We introduce Agentic Real2Sim, a framework for generalized physical world modeling with vision-language agents that converts a real-world recording of object-robot interaction into a simulatable episodic twin, and connects the resulting twin to downstream policy fine-tuning and evaluation. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining a comparable conversion success rate. The framework further supports custom scene conversion, fine-tuning of a pretrained policy with data generated from converted episodes, and works effectively as a surrogate for real-world policy evaluation. The project site, including code is available at https://agentic-real2sim.github.io.
Guanxiong Chen, Qianjun Xia, Jiawei Peng +24
University of British Columbia. · National University of Singapore. · Johns Hopkins University. +4
A simulation of a real robot workspace must preserve task-relevant interactions, while policies developed in it must operate on observations available to the real robot. Yet scene reconstruction and policy development are often treated separately. We present Agentic Real-to-Sim-to-Real (Agentic RSR), a framework that links scene reconstruction, policy development, and real-robot execution through the same manipulation task. Given a workspace video, a task description, and a known robot model, an agent recovers metric scale, iteratively refines the scene using visual feedback, and checks task-relevant interactions in MuJoCo. A coding agent then develops an executable policy, progressing from privileged object poses to visual observations and randomized simulation. The policy can interleave multiple observations and actions within one invocation, while the agent uses execution feedback to continue, retry, or revise its approach. A shared task-level interface carries the policy and accumulated experience to the real robot, where fresh observations and safety checks guide execution. Using Claude Opus 5.5 through DeepSeek Harness (DSH), all 18 scenes across two robots pass the joint visual check; mean four-view Depth MAE against reference depth estimates is 0.0808 m, mean Lab ΔE76 is 8.01, and mean grayscale SSIM is 0.7338. In real-robot experiments, the aggregate task success rate reaches 80% of the simulation task success rate, indicating substantial retention of simulated performance on hardware. Code and reconstructed scene data will be made publicly available.
Yihan Li, Yating Feng, Shengjiu Sun +7
Sun Yat-sen University · The Hong Kong University of Science and Technology (Guangzhou)
Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK provides the underlying computational engine for low-level geometries, part composition, differentiable parameter refinement, and physics validation. At the same time, NeoMCP exposes these capabilities as structured, tool-callable APIs for vision-language agents to visually inspect, edit code, and iteratively refine programs. Through a hierarchical coordinator-worker architecture, the framework scales seamlessly from standalone articulated objects to complex compositional scenes. Evaluations across object-level and scene-level reconstruction benchmarks demonstrate that NeoWorld-3 achieves state-of-the-art geometric fidelity and physical readiness. Crucially, manipulation trajectories planned within our reconstructed twins achieve successful zero-shot execution on a physical dual-arm robot, completing the Real-Sim-Real loop.