Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation

Sanya Verma; Luca Cilio; Velissarios Christodoulou

cs.ROSep 30, 2026

Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation

Authors: Sanya Verma, Luca Cilio, Velissarios Christodoulou

Organizations: Kaedim

Abstract

Simulated evaluation is increasingly used alongside real-world evaluation of robot policies because it is cheaper and easier to repeat; however, its value depends on how closely its outcomes track the real robot's. We test whether our reconstruction pipeline, combining metrically scaled object geometry, authored physical parameters and scene reconstruction, reduces disagreement between simulated and real robot scores relative to a default open-source recipe. We constructed two simulated versions of one bimanual robot cell: an authored reconstruction, using object geometry at estimated metric scale, projected textures, authored physics and our own scene splat; and a baseline, referred to as the default reconstruction, using the open-source recipe of a generative single-image mesh, engine-default physics and a Gaussian-splat scene. Both reconstructions use the same object photographs and scene video. Two policies ran five tasks each, giving ten task-policy pairs, which we call cells; each cell was run twenty times in each reconstruction with all other settings held fixed. Both reconstructions were scored against the same real trials, graded by a third-party evaluator. Pearson correlation between the ten simulated and real cell means is r = 0.90 for the authored reconstruction and 0.51 for the default. Mean score error is 6.97 percentage points for the authored reconstruction and 17.54 for the default, a reduction of 10.56 percentage points. These results show that improving the quality of the environment reconstruction through higher visual fidelity, authored physics and metric scale makes the simulation more faithful to the real world and narrows the sim-to-real gap. We release the harness, the per-trial scores, every reported run's configuration, and the assets and scenes of both reconstructions.

Figures & tables

	authored reconstruction	default reconstruction
object geometry	mesh reconstructed with our pipeline from a photograph	mesh generated by TRELLIS from the same photograph
texture	projected from the same photograph	generated by TRELLIS with the mesh
metric scale	estimated by our pipeline from a printed ChArUco board of known size captured with the object	typed in by hand to match the object’s apparent size in the same photograph; no real measurement
collision geometry	signed distance field (bowl, cup, can, spray bottle, cube, mug, plate); authored convex hulls (bottles, blocks, cutting board); bounding sphere (baseball); convex decomposition (grapes, bin)	convex decomposition
mass, friction, restitution	mass and restitution estimated by a vision-language model per object; friction from a per-class table; material class from the same model	not authored; PhysX defaults, density 1000 kg/m 3 , friction 0.5, restitution 0
scene reconstruction, from the same handheld video	3D Gaussian splat [ 14 ] trained from the handheld video, aligned by a table-plane fit, cut above the table	2D Gaussian splat [ 16 ] by a PolaRiS-inspired recipe from the same video, same alignment and cut

Table 1: The two reconstructions.

Panel of Figure 1 — Figure 1: Two objects, the ribbed bowl (top) and the red mug (bottom), in both reconstructions. Per reconstruction, left to right: textured render, mesh (flat shaded), what PhysX collides with, and the values read from each USD; the authored objects are a signed distance field of their own mesh, the default objects are a convex decomposition that fills the bowl’s cavity and the mug’s interior (32 hulls each, CoACD [ 15 ] as a stand-in for the decomposition PhysX computes at load). All panels of an object share one camera.

Figure 2: First top-camera frame of each of the ten cells, real against both reconstructions.

setting	value
control	30 Hz; absolute joint targets clipped to each joint’s range, as on the evaluator’s controller
action chunk	16 steps per inference ( $\pi$ 0.5), 30 (MolmoAct2)
MolmoAct2 solver steps	10
step budget per trial	bottles 6400, bowls 3600, blocks 3600, latte 3600, clear table 12800, matching the evaluator
top camera	rendered 640x480, served at 640x360; 70 degrees below horizontal; 0.72 m ( $\pi$ 0.5) or 0.88 m (MolmoAct2); measured D435 intrinsics per rig
object placement	one layout per cell refit from the evaluator’s start frame; up to 1 cm seeded jitter in x and y per trial

Table 2: Settings held constant, each read from the reported runs’ logs and job records.

task (max)	task string	stage	points
bottles in bin (20)	“put 2 glass bottles in the blue bin”	grasped and lifted	5
		in bin, on its side	8
		in bin, upright	10
		total score of two bottles in bin	20
stack blocks (30)	“stack the red block on the blue block”	attempt	5
		red lifted	10

Table 3: Rubric per stage. Task strings and points are the evaluator’s [ 11 ] .

measure	authored	default	paired difference, 95% CI	excludes zero
score error (percentage points)	6.97	17.54	+10.56 [+5.03, +17.11]	yes
progress disagreement (percentage points)	15.90	24.23	+8.32 [+1.78, +16.74]	yes
Pearson correlation (r)	0.90	0.51	+0.39 [-0.08, +0.87]	no
failure-stage disagreement (percentage points)	42.00	48.50	+6.50 [-3.50, +17.00]	no

Table 4: The four agreement measures, paired on the cell. Correlation is higher when closer to real; the three disagreement measures are lower. A positive difference means the authored reconstruction is closer to real.

task	policy	real	real 95% CI	authored	authored consistent	default	default consistent	margin
bottles in bin (20)	MolmoAct2	9.50	7.15 to 11.75	5.95	yes	6.25	yes	-0.30
bottles in bin (20)	$\pi$ 0.5	10.05	7.65 to 12.35	9.40	yes	8.90	yes	0.50
stack bowls (50)	MolmoAct2	29.75	23.50 to 35.75	25.75	yes	16.75	no	9.00
stack bowls (50)	$\pi$ 0.5	29.25	23.00 to 35.50	30.50	yes	23.00	yes	5.00
stack blocks (30)	MolmoAct2	3.50	1.50 to 5.50	7.75	no	12.75	no	5.00
stack blocks (30)	$\pi$ 0.5	6.00	4.00 to 8.25	5.75	yes	5.50	yes	0.25

Table 5: Mean score per cell in rubric points; the task maximum is in brackets after the task name, so 9.50 for bottles is 9.50 out of 20. “consistent” means the 95% bootstrap interval on the simulated minus real mean includes zero (Section 4 ). Margin is how much closer to the real mean the authored reconstruction is than the default, in rubric points; a negative margin means the default is nearer.

Figure 3: Simulated against real mean score per cell, percent of task maximum. Each point is one cell; the dashed line is perfect agreement. The two labelled points are the largest departures: blocks with MolmoAct2, where the default reconstruction scores 42% against 12% real, and latte with $\pi$ 0.5, where it scores near zero against 37% real.

cell	authored	default	stages
bottles, MolmoAct2	22.0	20.0	5
bottles, $\pi$ 0.5	15.0	17.5	4
bowls, MolmoAct2	7.5	25.8	6
bowls, $\pi$ 0.5	6.7	10.0	6
blocks, MolmoAct2	20.0	60.0	1
blocks, $\pi$ 0.5	13.3	15.0	3

Table 6: Progress disagreement per cell, the mean absolute gap in percentage points between the fractions of simulated and of real trials reaching each rubric stage. Lower is closer. “stages” is the number of rubric stages the real robot reached.

Figure 4: Fraction of trials reaching at least each rubric stage, per cell. Stages are the distinct non-zero scores the real robot produced in that cell, so the ladder is the real robot’s. A curve that falls earlier is a system that stops earlier. Table 6 is the mean absolute gap between each simulated curve and the real curve.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

cell	authored, all gate-passing batches	default, all gate-passing batches
bottles, MolmoAct2	3.45 (V2_K_bottles_molmo), REPORTED 5.95 (K2_bot_molmo_plain), 6.50 (A1_bot_molmo_plain)	REPORTED 6.25 (S1_PS_bot_molmo)
bottles, $\pi$ 0.5	REPORTED 9.40 (K1_bot_pi05_plain), 10.80 (FINAL_K_bottles_pi05), 10.85 (A3_bot_pi05_plain)	REPORTED 8.90 (B0_PS_bot_pi05_r2), 9.50 (S6_PS_bot_pi05), 9.65 (A0_PS_bot_pi05_rerun)
bowls, MolmoAct2	19.75 (six jitter-sweep shards pooled: 04_NS10, JIT25, JIT40, DBG_JIT25, DBG_JIT40, PROBE_NS10; the gate cannot see the placement shift, so they pass it; listed, not reported), 20.00 (N20_K_bowls_molmo), 23.25 (V2_K_bowls_molmo), REPORTED 25.75 (RB_BOWLS_a + RB_BOWLS_b)	REPORTED 16.75 (S2_PS_bowls_molmo)
bowls, $\pi$ 0.5	28.50 (FINAL_K_bowls_pi05), REPORTED 30.50 (C2_bowls_pi05)	REPORTED 23.00 (S7_PS_bowls_pi05)
blocks, MolmoAct2	REPORTED 7.75 (C1_blocks_molmo), 8.00 (N20_K_blocks_molmo), 9.25 (V2_K_blocks_molmo)	10.50 (N20_P_blocks_molmo), 12.50 (V2_P_blocks_molmo), REPORTED 12.75 (S3_PS_blocks_molmo)
blocks, $\pi$ 0.5	REPORTED 5.75 (T1_blocks_pi05), 6.00 (FINAL_K_blocks_pi05)	REPORTED 5.50 (ZZ1_PS_blocks_pi05)

Appendix

Table 7: Every gate-passing batch per cell and reconstruction, mean score in rubric points with the run id in brackets; REPORTED marks the batch used in Tables 4 to 6.

Figure 5: Per cell, in percent of task maximum: the real robot’s mean (line) and its 95% bootstrap interval (grey band); the reported batch for the authored (green) and default (orange) reconstruction; and, as a faint bar behind each marker, the range spanned by every batch that passed the configuration gate for that cell and reconstruction. A cell with no bar had a single passing batch. A marker inside the grey band is not the consistency test used in the text, which compares the two means with their combined uncertainty (Section 4 ). The batches themselves are listed in the table above.

Explore similar work

Sep 20, 2026cs.RO

ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluation

Simulation-based evaluation provides a scalable and repeatable alternative to real-world evaluation of vision-language-action (VLA) policies. However, reconstruction errors can cause simulated policy performance to diverge from real-world performance, motivating the need to assess reconstructed environments for downstream VLA policy evaluation. We present ReVeal, a real-to-sim assessment framework combining workspace reconstruction, reconstruction-level assessment, and matched closed-loop policy evaluation. Novel-View Mesh Fidelity (NVMF) and Annotated Planar Geometry Fidelity (APGF) assess observation and planar geometric fidelity, respectively. We also develop PGSR-D, a reconstruction pipeline incorporating monocular depth supervision to improve geometry where multi-view visual cues are limited. Across 8 assessment scenes, NVMF and APGF consistently distinguish the fidelity of 2DGS, PGSR, and PGSR-D. Matched evaluations of GR00T, SmolVLA, and pi0.5 across 8 humanoid manipulation tasks show consistent ordering between reconstruction fidelity and real-sim performance agreement across pipelines. Further analysis of the evaluation workspaces shows that higher fidelity is associated with stronger real-sim agreement.

Xinyi Wang, Heng Hao, Wenjun Hu +5

Purdue University · Samsung SDS Research America

Simulation-Based Reinforcement Learning Evaluation Benchmarks

Ask PDF

Jul 7, 2026cs.RO

RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation

Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. In this work, we present RoboSnap, a real-to-sim framework that turns a single RGB image into a simulation-ready scene. The key idea is a layered design that separates the physics-critical interaction area from the surrounding visual context: collision-aware foreground assets are refined for stable robot interaction, while a 3D Gaussian splatting visual layer preserves faithful background appearance under novel views. Experiments on DROID scenes and real-robot tasks show that RoboSnap achieves reliable trajectory replay in the recovered scenes, supports task-specific synthetic data generation for policy training, and yields meaningful sim-real correlation for policy evaluation. To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID. Extensive experiments suggest that the value of real-to-sim methods lies not only in high-fidelity visual reconstruction, but in turning real environments into reusable infrastructure for robot learning and evaluation.

Shujie Zhang, Jingkun Yi, Weipeng Zhong +6

1Shanghai AI Laboratory · 3Zhejiang University · 2Shanghai Jiao Tong University +1

Sim-To-Real Reinforcement Learning Indoor Scene Generation Scalable Robot Learning Robocasa Robot Systems

Ask PDF

Jun 26, 2026cs.RO

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .

Nadun Ranawaka, Josiah Wong, Wei-Lin Pai +15

NVIDIA · Stanford University · The University of Texas at Austin +2

Sim-To-Real Reinforcement Learning Simulation-Based Reinforcement Learning Indoor Scene Generation Nvidia Isaac Sim Robot Policies Real-World+1

Ask PDF