LabWorld: Learning Interactive 4D Worlds for Scientific Experimentation

Suozhi Huang; Kelsey Fu; Binxu Li; Luna Lyu; Kaize Wu; Wenbo Lu; David Smerkous; Jiahan Li

cs.CVOct 27, 2026

LabWorld: Learning Interactive 4D Worlds for Scientific Experimentation

Authors: Suozhi Huang, Kelsey Fu, Binxu Li, Luna Lyu, Kaize Wu, Wenbo Lu, David Smerkous, Jiahan Li,

Organizations: Princeton University · Stanford University · University of Chicago · New York University ·

Abstract

Autonomous scientific agents must not only reason about experiments but also interact with the physical environments in which they are executed. Existing laboratory simulators typically rely on manually constructed assets and predefined procedures, while laboratory videos capture real equipment, layouts, and interactions but leave this information implicit in pixels. We introduce LabWorld, an agent-driven Real2Sim framework for reconstructing interactive 4D laboratory worlds from monocular RGB video. LabWorld recovers reusable 3D assets, a metric scene, time-varying object states, and procedure events. Multi-frame observations provide complementary evidence for asset and scene reconstruction, while motion reconstruction combines visual and geometric observations with event constraints encoding the spatial and temporal requirements of laboratory interactions. We evaluate LabWorld on real pipetting and vortexing videos collected in our laboratory against general-purpose agentic and dedicated reconstruction baselines. For pipetting, LabWorld achieves a mean Mask IoU of 0.501, exceeding the strongest baseline by 0.033. For vortexing, it achieves a mean Tube-Length-Normalized PCK-AUC of 0.638, exceeding the strongest baseline by 0.035. On five pipetting videos, event-conditioned reconstruction increases the fraction of modeled physical constraints satisfied from 14.3% (5/35) to 100% (35/35). The resulting worlds expose geometry, motion, articulation, and events as explicit, editable state for replay, inspection, and future embodied scientific systems.

Figures & tables

Figure 1: From laboratory video to interactive 4D worlds. Each video is selected from the collection on the left. Source frames are paired with reconstructed views at matching video times $t$ , measured in seconds. The enlarged scene overlays the same pipette, with matching colors linking its reconstructed states to the source frames. The lower panels illustrate video understanding, reusable asset generation, and potential uses in path planning and event editing. Object boxes and interaction arrows in the video-understanding panel are explanatory annotations. The planning and editing panels are conceptual illustrations rather than evaluated downstream results.

Method	Laboratory assets	Scene reconstruction	Motion reconstruction	Event-conditioned reconstruction
AutoBio ( Lan et al., 2026 )	$\checkmark$	$\times$	$\times$	$\times$
SafeLab ( Bai et al., 2026 )	$\checkmark$	$\times$	$\times$	$\times$
TableVerse ( Wang et al., 2026a )	$\times$	$\checkmark$	$\times$	$\times$
SimFoundry ( Ranawaka et al., 2026 )	$\times$	$\checkmark$	$\times$	$\times$
Agentic Real2Sim ( Chen et al., 2026a )	$\times$	$\checkmark$	$\checkmark$	$\times$
GPT-6 Astra	$\checkmark$	$\checkmark$	$\checkmark$	$\times$

Table 1: Capabilities of related methods and evaluated baselines. Columns denote reusable 3D assets, reconstruction of the observed scene and motion, and explicit procedure-level constraints during motion reconstruction.

\checkmark

indicates a demonstrated capability under these definitions;

\times

indicates that it is not demonstrated. For GPT-6 Astra, the entries describe the video-based reconstruction agent evaluated in this study rather than the underlying model in isolation. Its inputs and reconstruction procedure are described in Appendix G .

Figure 2: Overview of the LabWorld reconstruction pipeline. SAM3 provides visible object masks, while MapAnything estimates scene geometry from the source video. Observed equipment is represented by generated or reused assets and assembled into a shared 3D scene. An agent-generated event list specifies interaction types and time windows for motion reconstruction, which combines visual and geometric observations with temporal regularization and event constraints. The colored curve shows the reconstructed pipette-tip trajectory, with markers identifying selected times. Source-video and reconstruction crops support visual inspection of a beaker interaction, without independently establishing event completion.

Figure 3: Source-video and reconstruction comparison. Source frames (top) and LabWorld reconstructions (bottom) from pipetting video 108 are shown at matched times using the same tabletop crop. The columns show tool transport, tip-box interaction, and tube-rack interaction. Green boxes (Ref.) enclose SAM3 visible reference masks, while red boxes (3D) enclose rendered tool silhouettes projected through the fixed reconstruction camera. The overlays support comparison of projected position and extent, although occlusion can affect the visible reference masks. These selected frames illustrate reconstruction states rather than verify complete events or liquid transfer.

Method	Pipetting IoU $\uparrow$	Vortexing PCK-AUC $\uparrow$
GPT-6 Astra	0.156	0.479
FoundationPose	0.026	0.232
Do as I Do	0.468	0.603
LabWorld	0.501	0.638

Table 2: Reconstruction accuracy across baselines. Pipetting is evaluated using mean Mask IoU and vortexing using Tube-Length–Normalized PCK-AUC. Higher is better.

Figure 4: Pipette reconstruction agreement under reduced visibility. (a) Mask IoU for higher-visibility ( $v\geq 0.7$ , $n=503$ ) and lower-visibility ( $v<0.7$ , $n=144$ ) observations across ten LabWorld reconstructions. Points denote observations, violins show their distributions, boxes indicate medians and interquartile ranges, and black diamonds mark pooled means (0.545 and 0.349). (b) Mask IoU versus the occlusion proxy $o=1-v$ across all 647 observations. The red line shows the pooled linear fit with a pointwise 95% confidence interval from whole-video resampling. Pearson $r=-0.347$ and Spearman $\rho=-0.070$ show that the group-level difference does not correspond to a consistently monotonic relationship. Visibility is a prediction-dependent proxy, not a calibrated measure of physical occlusion.

Method	Mask IoU $\uparrow$	Physical constraints $\uparrow$
LabWorld	0.508	35/35
w/o event-conditioned reconstruction	0.572	5/35

Table 3: Ablation of event-conditioned reconstruction. Removing event constraints improves image-space Mask IoU but causes physical constraint satisfaction to collapse, demonstrating that visual agreement alone does not ensure a physically valid reconstruction. The complete component-wise ablation is reported in Appendix J .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Event	Functional parts	Reconstruction constraint
Grasp	handle, hand	Object and hand motion become locally coupled, and the object is released from its previous support.
Transport	rigid object	Preserve object dimensions and orientation continuity while allowing free motion above the table.
Dispense	tool tip, opening	Enforce the required approach, entry, dwell, and withdrawal phases while retaining the observed tool axis.
Insert	tube axis, slot	Align axes, constrain lateral offset, and enforce a valid insertion depth without penetration.
Open/close	lid, hinge, base	Keep the base supported and restrict the lid to a revolute joint with fixed axis and joint limits.
Place	object support, surface	Reach support contact with low terminal velocity and remain non-penetrating after release.

Appendix

Table 4: Scientific events and the geometric relations they instantiate.

Method	Mask IoU $\uparrow$	Physical constraints $\uparrow$
LabWorld	0.508	35/35
w/o event-conditioned reconstruction	0.572	5/35
w/o temporal regularization	0.514	35/35
w/o depth-derived initialization	0.487	35/35
w/o iterative visual refinement	0.223	35/35

Appendix

Table 5: Complete component-wise ablation of LabWorld . Mask IoU measures image-space agreement, while physical constraints measure satisfaction of the modeled experimental interaction geometry. Event conditioning is required for physical validity despite higher IoU without it, while the remaining ablations characterize complementary contributions to visual accuracy and trajectory coherence.

4D Reconstruction

4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.

See all 139 papers

WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations
Oct 22, 2026Tianling Xu, Weijie Wan, Honghao Su +9Robotics Simulation Digital Twins
Xspark AI · University College London · Tsinghua University +2
SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments
Oct 26, 2026Luke Hollis, Tianxing Fan, Heng YangRobotics Simulation Physics-Aware 3D Scene Reconstruction
Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Oct 3, 2026Jinzhou Tang, Zijun Zhang, Jing Yang +10Interactive World Models Coding Agents
Aether AI · University of California, San Diego

Oct 22, 2026J/K move · Enter open · S save

WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

Tianling Xu, Weijie Wan, Honghao Su, Kailun Su+8

Xspark AI · University College London · Tsinghua University+2

Robotics Simulation Digital Twins Agentic AI for Robotics Interactive World Models

Open paper AskPDF arXiv

Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.

Abstract

4D Reconstruction

Explore similar work

WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos