Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction

Lintao Wang; Mingyang Sun; Yang Liu; Dingkang Yang; Lihua Zhang

cs.CVSep 21, 2026

Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction

Authors: Lintao Wang, Mingyang Sun, Yang Liu, Dingkang Yang, Lihua Zhang

Organizations: Physical Superintelligence Lab, Fysics AI · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Multimedia Laboratory (MMLab), The Chinese University of Hong Kong · College of Electronic and Information Engineering, Tongji University

Abstract

Agentic recognition requires visual perception to move beyond static scene understanding and produce structured scene representations that support the perception--reasoning--action loop. Existing single-image 3D generation methods, however, mainly produce visually plausible object assets rather than simulation-ready scene states. When independently generated meshes are composed in a shared space, they may fail to align with the input camera, violate gravity, interpenetrate nearby objects, or become unstable under physics simulation. We present Fysiverse-3D-SimReady, a grounded refinement framework for reconstructing simulation-ready multi-object scenes from a single RGB image with instance and ground prompts. The method places generated object meshes into a shared gravity-aligned scene, refines their camera-space poses through differentiable rendering, and corrects scene-level supports and contacts for stable physical execution. The scene is then used by an agentic simulation workflow, which converts a scene-specific task goal into an executable physics rollout rendered from the original camera view. Experiments show that Fysiverse-3D-SimReady improves input-view alignment, contact plausibility, and physical stability over existing single-image reconstruction and scene generation baselines, while enabling goal-conditioned physical interactions from a single image.

Figures & tables

Method	PSNR $\uparrow$	mIoU $\uparrow$	$R_{\mathrm{pen}}$ $\downarrow$	$K_{\max}$ (J) $\downarrow$	$D_{\max}$ (cm) $\downarrow$
SAM3D [ 3 ]	10.80	0.74	4.44	$1.30{\times}10^{3}$	$6.06{\times}10^{2}$
TabletopGen [ 40 ]	6.63	0.31	0.00	10.54	298.15
PAT3D [ 22 ]	7.13	0.47	0.00	3.13	5.31
Fysiverse-3D-SimReady	11.39	0.71	0.00	0.04	1.09

Table 1: Main quantitative comparison on the evaluation subset.

Variant	PSNR $\uparrow$	mIoU $\uparrow$	$R_{\mathrm{pen}}$ $\downarrow$	$K_{\max}$ (J) $\downarrow$	$D_{\max}$ (cm) $\downarrow$
w/o differentiable rendering	11.29	0.67	0.00	0.16	1.58
w/ differentiable rendering	11.39	0.71	0.00	0.04	1.09

Table 2: Ablation of differentiable per object alignment.

LLM Backend	Simulator Assistance	E2E Pass (%) $\uparrow$	Prompt Following (%) $\uparrow$	Avg. E2E Time $\downarrow$
GPT-5.5	w/o Agentic Roles	100.0	86.2	14.11
GPT-5.5	w/ Agentic Roles	100.0	93.1	13.44
Qwen3.7-Plus	w/o Agentic Roles	93.1	0.0	12.95
Qwen3.7-Plus	w/ Agentic Roles	93.1	66.7	11.46

Table 3: Ablation of Planner, Reasoner, and Builder roles across LLM backends.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 950 papers

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images
Oct 20, 2026Rundong Luo, Yunong Liu, Di Cao +4Physics-Aware 3D Scene Reconstruction 3D Reconstruction
Luma AI · Cornell University
REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image
May 28, 2026Xiaoxuan Ma, Jiashun Wang, Nicolas Ugrinovic +23D Reconstruction 3D Scene Representation
Carnegie Mellon University
SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image
Jun 2, 2026Inhee Lee, Sangwon Baik, Sungjoo Kim +33D Reconstruction Robotics Simulation
Seoul National University

Oct 20, 2026J/K move · Enter open · S save

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

Rundong Luo, Yunong Liu, Di Cao, Matthew Tancik+3

Luma AI · Cornell University

Physics-Aware 3D Scene Reconstruction 3D Reconstruction 3D Scene Representation Single-View 3D Reconstruction Robotics Simulation

Open paper AskPDF arXiv

Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.

Abstract

3D Reconstruction

Explore similar work

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image

SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image