cs.ROOct 26, 2026

SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments

Authors: Luke Hollis, Tianxing Fan, Heng Yang

Organizations: Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.

Figures & tables

Figure 2: SceneAgent pipeline. Agents coordinate reconstruction, segmentation, physical modeling, and scene export, using simulator renders to inspect and repair intermediate outputs. Inputs include RGB image sequences, 3DGS and photogrammetry assets, AI-generated 3DGS scenes, and LiDAR scans. Section 3.6 extends the pipeline to videos of dynamic scenes.
Figure 3: 4DCodeBench dynamic IoU against tokens per task on the synthetic tasks Z_06, B_09 and Z_12. Baseline scores come from the supplied 4DCodeBench reference table. Baseline tokens are the published averages over 100 tasks.
Figure 4: 4DCodeBench mean tokens per task by conversion stage, with and without SceneAgent skills, on the synthetic tasks Z_06, B_09 and Z_12.
Table 1: System comparison. Capabilities of SceneAgent and recent real-to-sim systems for policy learning and evaluation.
Figure 5: Reconstruction of a cluttered bin scene from GraspClutter6D (hard/000091, 20 YCB objects) with and without SceneAgent skills, and with SimFoundry. Higher F1 at 1 cm and lower Chamfer distance indicate closer agreement with the dataset’s ground-truth object meshes.
Figure 6: Tokens used in each conversion stage of the GraspClutter6D bin scene (hard/000091), with and without SceneAgent skills.
Figure 7: Agent workflow and available skills. The agents coordinate tools for capture processing, object reconstruction, physical modeling, and simulation, and inspect intermediate results to guide corrections.
Figure 8: Marker reconstruction and interaction. A single product image is converted into object geometry and parts, including completion of the tip beneath the cap. Two Franka Emika Panda arms with Sharpa Wave hands manipulate the resulting marker and write on a whiteboard in simulation.
Figure 9: A corduroy cap captured with photogrammetry, annotated with estimated rigidity, and simulated under contact. The example illustrates the conversion from visual geometry to a deformable asset.
Figure 10: Deformable assets converted from Polycam [ 42 ] captures. Each object is shown at rest and under simulated deformation.
Figure 11: Contact and fluid simulation. A Franka Emika Panda arm with a Sharpa Wave hand presses a plush toy (01). A cup tips and pours milk onto a table (02). The milk is simulated with MPM.
Figure 12: Fracture simulation. A Dexmate Vega with two Sharpa Wave hands tears a parchment sheet from a roll and covers a bowl of fruit (01–02). Ceramic jars fracture after being knocked from a pedestal or dropped (03–04). The parchment parameters are engineering estimates.
Figure 13: Fracture and cutting. A watermelon breaks apart on impact after a drop (01). A Franka Emika Panda arm with a Sharpa Wave hand presses into a jelly, which cracks (02), and cuts salmon with a chef’s knife (03). The watermelon and jelly use cohesive finite elements in Newton.
Figure 14: The interactive viewer. (01) The scene editor for the lab scene with a Dexmate Vega, with the scene graph, the Three.js stage, the properties of the selected object, and the agent operator. Lettered callouts mark its main operations. (02) The experiment view of a digital-sister evaluation. Each cell is one object sister under one lighting setup and reports place success over ten trials.
Figure 15: Digital sisters of reconstructed objects. Each row shows one converted object followed by seven generated variants.
Figure 16: Hybrid environments combining captured objects with artist-authored assets. Franka Emika Panda arms with UMI grippers are shown during and after contact-rich and deformable-object tasks in simulation.
Figure 17: Dexterous manipulation in converted environments. Sharpa hands mounted on Panda arms and a Dexmate mobile manipulator interact with reconstructed objects during multi-step simulated tasks.
Figure 18: A Dexmate Vega opens a laptop (01) and types a command on its keyboard (02–03).
Figure 19: Agentic real2sim2real policy autoresearch, illustrated with a Franka Emika Panda and Sharpa Wave hand. First, real task recordings expose failures then agents construct a digital twin, generate task and scene sisters, and evaluate the current policy. Coding agents revise expert controllers and rerun simulated tasks. Task-verified recordings feed policy fine-tuning, and fixed paired validation trials compare checkpoints before real-world re-evaluation.
Figure 20: Converted scenes shown during simulated rollouts. Captured environments and objects can be combined to construct multi-step manipulation tasks. In the Matterport observatory, a Dexmate with two Sharpa hands pulls both guide ropes to slew the telescope.
Figure 21: Six objects converted from public Polycam captures, with close-up views used during agent inspection. The examples illustrate reconstruction of detailed geometry and assembly of simulation assets.
Figure 22
training strategiesperformance in simsuccess rate in simreal-world performancereal-world success rate
policy 1114/20057.0 %11/5022.0 %
policy 2140/20070.0 %31/5062.0 %
policy 3170/20085.0 %44/5088.0 %
Table 3: Comparison of different training strategies. Policy 1 is trained on 2k eggplant demonstration episodes from the dataset with variations. Policy 2 is trained for 3 tasks, each task with 2k demonstration episodes with variations. Policy 3 is finetuning policy 2 on 2k eggplant demonstration episodes without variation.
Figure 23: Preliminary real-world deployment on a Franka Emika Panda after simulation-based fine-tuning. Three episodes are shown with corresponding simulated scenes and wrist-camera views, including the eggplant and basket layouts.
More in this topic

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 319 papers

Explore similar work

CardsList

  1. SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

    Oct 20, 2026Rundong Luo, Yunong Liu, Di Cao +4Physics-Aware 3D Scene Reconstruction3D Reconstruction

    Luma AI · Cornell University

  2. Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies

    Oct 7, 2026Yihan Li, Yating Feng, Shengjiu Sun +7Sim-to-Real TransferRobotic Manipulation

    Sun Yat-sen University · The Hong Kong University of Science and Technology (Guangzhou)

  3. LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction

    Oct 1, 2026Zhening Huang, Yueyan Li, Johnathan Chiu +53D Reconstruction3D Scene Representation

    University of Cambridge · Imperial College London · Independent Researcher