YUBI-Real2Sim: 3DGS Scene Verification for Reliable Policy Evaluation

Hiroki Nishizawa; Takehiko Ohkawa; Nicola Scianca; Tatsuya Matsushima; Shigeo Morishima; Kei Ota

cs.ROOct 13, 2026

YUBI-Real2Sim: 3DGS Scene Verification for Reliable Policy Evaluation

Authors: Hiroki Nishizawa, Takehiko Ohkawa, Nicola Scianca, Tatsuya Matsushima, Shigeo Morishima, Kei Ota

Organizations: AI Robot Association (AIRoA) · Waseda University

Abstract

Evaluating VLA policies on real robots is costly, time-consuming, and difficult to reproduce, making real-to-sim simulation essential for scalable evaluation. While 3D Gaussian Splatting (3DGS) enables rapid real-to-sim scene generation, the reconstruction quality required for reliable policy evaluation remains unquantified. In particular, policies relying on wrist-mounted fisheye cameras (e.g., YUBI handheld gripper data) are heavily exposed to visual artifacts (holes, floaters) in wide-angle views, directly degrading task performance. To address this, we present YUBI-Real2Sim, a unified framework for 3DGS-based scene reconstruction and real-to-sim policy evaluation under dual fisheye wrist views. We develop an integrated pipeline combining 200-degree fisheye camera capture, 3DGS training, 3D foundation models for object modeling, and MuJoCo-based physical simulation, reproducing the observed real-world policy ordering in eight of nine scene-task comparisons. Using this framework, we construct a benchmark across two bimanual robot platforms, three scenes, three tasks, and dual pi0.5-based policies. This systematically measures real-to-sim rank agreement and evaluates policy sensitivity against counterfactual 3DGS artifacts (ceiling holes, floaters, warping) and capture disturbances. We further analyze whether novel-view quality scores can indicate policy-ranking reliability under a scene-held-out evaluation protocol. The results show task-dependent performance and limitations of quality-based reliability prediction, highlighting the need to assess reconstruction quality when using Real2Sim for policy evaluation. These results do not yet establish a generally reliable scene-acceptance rule. Project Page: https://yubi-real2sim.airoa.io.

Figures & tables

Figure 1: YUBI-Real2Sim analyzes and verifies 3DGS scenes for reliable policy-ranking evaluation. The quality of 3DGS can easily degrade depending on the camera views. We analyze how variations in 3DGS quality affect policy rankings and examine whether scene-quality scores can indicate policy-ranking reliability.

Figure 2: YUBI-Real2Sim pipeline. We capture the background with a single $200^{\circ}$ FOV camera, estimate camera poses with COLMAP, and densify the point cloud with OpenMVS to initialize Gaussian Wrapping for high-fidelity reconstruction. We generate object geometry and textures with TRELLIS 2.0 and approximate collision meshes with CoACD. The reconstructed scene and objects are integrated into MuJoCo simulation. For wrist-view rendering, we independently render the background Gaussians and the MuJoCo visual meshes, then composite their pixels using depth maps.

Figure 3: Simulated scenes, and the added disturbances for 3DGS or viewpoints. One column per scene, one band per condition. Blue : the clean 3DGS reconstructions the rollouts run in. Red : the same reconstructions carrying a counterfactual artifact — a ceiling hole in FR3 Flat, the $54^{\circ}$ and $90^{\circ}$ half-angles either side of the dashed rule, low floaters in FR3 Duo, and warping in TX-G2. Amber , the colour Fig. 1 gives capture: one reduced-coverage reconstruction per condition, each in the scene of its column. The panels show representative examples of the reconstruction conditions. We evaluated two task-specific VLA checkpoints for each of three tasks (Cup, Tape, and Pen) in all three scenes (FR3 Flat, FR3 Duo, and TX-G2).

Figure 4: Policy ranking in real and simulated evaluation. Black/gray bars show real success and blue bars show simulated success; light and dark bars denote

\pi_{1}

and

\pi_{2}

, respectively, as in Fig. 1 . Policy labels are fixed within each task across scenes and domains. Error bars are marginal 95% Wilson confidence intervals from 20 real and 80 simulated trials per policy. The observed ranking agrees in 8/9 scene–task pairs (88.9%); Flat Cup is reversed.

CF condition	Lost	Gained	Reversed	Tied	with real
	Significance		Ranking vs. Clean		Agreement
Hole 54 ∘	1	1	1	0	7/9
Hole 90 ∘	4	0	1	0	7/9
Floater low8	1	0	0	1	7/9
Floater mid8	1	1	1	0	7/9
Floater high8	1	0	1	0	7/9

Table I: Effect of each counterfactual (CF) condition on the nine scene–task policy comparisons. Significance: comparisons whose Fisher-test status (

p<0.05

) is lost or gained relative to Clean; the others keep their status. Ranking vs. Clean: comparisons whose simulated ranking is reversed or tied relative to Clean. Agreement with real: comparisons whose simulated ranking matches the real ranking. Clean agrees in 8/9 (Flat Cup is reversed) and no CF condition corrects it, so every reversal or tie is a lost agreement.

Metric/view	Cup	Tape	Pen	Total	All-three
NVS Test	7/15	8/15	15/15	30/45	13/15
NVS OBS	12/15	2/15	15/15	29/45	7/15
PSNR Test	8/15	8/15	15/15	31/45	9/15
SSIM Test	8/15	9/15	15/15	32/45	7/15
LPIPS Test	8/15	5/15	15/15	28/45	10/15

Table II: Scene-held-out reliability classification: correct/total decisions on 15 matched conditions per task. NVS: NVS-SQA; Total: pooled task decisions; All-three: reliability of all three task rankings.

Real-to-Sim Transfer

See all 14 papers

Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation
Sep 30, 2026Sanya Verma, Luca Cilio, Velissarios Christodoulou +43D Reconstruction Robotics Simulation
Kaedim
ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluation
Sep 20, 2026Xinyi Wang, Heng Hao, Wenjun Hu +53D Reconstruction Robotic Manipulation
Purdue University · Samsung SDS Research America
Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies
Oct 7, 2026Yihan Li, Yating Feng, Shengjiu Sun +7Sim-to-Real Transfer Robotic Manipulation
Sun Yat-sen University · The Hong Kong University of Science and Technology (Guangzhou)

Sep 30, 2026J/K move · Enter open · S save

Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation

Sanya Verma, Luca Cilio, Velissarios Christodoulou, Tyler Fermelis+3

Kaedim

3D Reconstruction Robotics Simulation Sim-to-Real Transfer Robotic Policy Evaluation

Open paper AskPDF arXiv

Simulated evaluation is increasingly used alongside real-world evaluation of robot policies because it is cheaper and easier to repeat; however, its value depends on how closely its outcomes track the real robot's. We test whether our reconstruction pipeline, combining metrically scaled object geometry, authored physical parameters and scene reconstruction, reduces disagreement between simulated and real robot scores relative to a default open-source recipe. We constructed two simulated versions of one bimanual robot cell: an authored reconstruction, using object geometry at estimated metric scale, projected textures, authored physics and our own scene splat; and a baseline, referred to as the default reconstruction, using the open-source recipe of a generative single-image mesh, engine-default physics and a Gaussian-splat scene. Both reconstructions use the same object photographs and scene video. Two policies ran five tasks each, giving ten task-policy pairs, which we call cells; each cell was run twenty times in each reconstruction with all other settings held fixed. Both reconstructions were scored against the same real trials, graded by a third-party evaluator. Pearson correlation between the ten simulated and real cell means is r = 0.90 for the authored reconstruction and 0.51 for the default. Mean score error is 6.97 percentage points for the authored reconstruction and 17.54 for the default, a reduction of 10.56 percentage points. These results show that improving the quality of the environment reconstruction through higher visual fidelity, authored physics and metric scale makes the simulation more faithful to the real world and narrows the sim-to-real gap. We release the harness, the per-trial scores, every reported run's configuration, and the assets and scenes of both reconstructions.

Abstract

Real-to-Sim Transfer

Explore similar work

Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation

ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluation

Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies