SCAPE: Scenario-Conditioned Simulation-Augmented Policy Evaluation

Dijie Zhu; Seunghun Oh; Ruopeng Huang; Zhiyu Huang; Jiaqi Ma; Chen Tang

cs.ROAug 19, 2026

SCAPE: Scenario-Conditioned Simulation-Augmented Policy Evaluation

Authors: Dijie Zhu, Seunghun Oh, Ruopeng Huang, Zhiyu Huang, Jiaqi Ma, Chen Tang

Organizations: University of California, Los Angeles · Seoul National University · University of Southern California · North Carolina State University

Abstract

Reliable performance evaluation is a central bottleneck for deploying robot-learning policies in real-world conditions. Real-world testing is faithful but costly and difficult to scale, whereas simulation-based testing scales easily but is inevitably biased by the sim-to-real gap. Existing simulation-augmented methods combine limited real-world rollouts with abundant simulation proxies, but focus on performance averaged over initial conditions and deployment settings. Such population-level averages obscure scenario-specific variation and provide limited guidance about when and where a policy can be safely deployed. We propose SCAPE, a scenario-conditioned simulation-augmented policy evaluation framework that predicts scenario-conditioned real-world policy performance using limited paired sim-and-real samples and large-scale simulation rollouts. SCAPE corrects sim-to-real bias in simulation labels before training the prediction model and calibrates prediction uncertainty through conformal prediction. We validate SCAPE on autonomous driving and quadruped velocity tracking. In sim-to-sim studies, SCAPE reduces scenario-level prediction error by 4.9%/34.7% (driving) and 14.5%/27.7% (quadruped) relative to scene-conditioned neural and aggregate statistical baselines on average. We further evaluate a velocity-tracking policy deployed on a physical Unitree Go2. SCAPE also improves testing sample efficiency, produces narrower calibrated prediction intervals, generalizes better to out-of-distribution scenarios, and enables fine-grained deployment strategies.

Figures & tables

Figure 1: Overview of Scape . Scape combines limited paired sim-and-real data with large-scale simulation data to predict scenario-conditioned target performance with calibrated uncertainty, which supports fine-grained deployment decisions by revealing scenario-specific variation and failures that aggregate metrics can hide.

Figure 2: Method overview of Scape . Scape corrects biased simulation labels using paired sim-and-real data, trains a scenario-conditioned performance predictor with corrected simulation-only data, and applies conformal calibration to quantify prediction uncertainty.

Figure 3: Paired sample construction in (a) nuPlan and (b) quadruped experiments. In nuPlan, each pair is constructed from the same driving scenario by measuring the target metric $R$ with closed-loop evaluation and the surrogate metric $S$ with open-loop evaluation. In quadruped, each pair matches a real-world rollout with a reconstructed-sim rollout under the same scene context, robot placement, and command.

Figure 4: Test loss vs. fraction of paired-label on autonomous driving and quadruped locomotion tasks. The paired-data fraction denotes the fraction of the full paired training set $\mathcal{D}_{p}$ used for training. The Urban Driver setting contains $83.6$ K paired training samples and $119.2$ K surrogate-only samples; the quadruped Sim2Sim setting contains $300$ paired training samples and $1{,}493$ surrogate-only samples. Panels (a)–(d) report autonomous driving Sim2Sim results, and panels (e)–(f) report quadruped Sim2Sim velocity-tracking results. Curves report mean $\pm$ standard error over 10 random paired-data seeds.

	nuPlan Urban Driver				Go2 Sim2Sim
Method	ADE	TTC $<1$ s	Drivable	Collision	Velocity	Yaw Rate
Scape	$\boldsymbol{40.07\!\pm\!0.26}$	$\boldsymbol{1.447\!\pm\!0.010}$	$\boldsymbol{1.118\!\pm\!0.004}$	$\boldsymbol{1.430\!\pm\!0.004}$	$\boldsymbol{10.84\!\pm\!0.48}$	$\boldsymbol{7.60\!\pm\!0.39}$
ppi-nn	$44.44\!\pm\!0.16$	$1.614\!\pm\!0.004$	$1.233\!\pm\!0.004$	$1.573\!\pm\!0.004$	$11.42\!\pm\!0.38$	$10.27\!\pm\!0.31$
rs-mix	$44.65\!\pm\!0.16$	$1.505\!\pm\!0.006$	$1.152\!\pm\!0.002$	$1.495\!\pm\!0.003$	$11.56\!\pm\!0.28$	$9.12\!\pm\!0.22$
r-only	$41.02\!\pm\!0.30$	$1.497\!\pm\!0.003$	$1.155\!\pm\!0.003$	$1.474\!\pm\!0.009$	$11.12\!\pm\!0.59$	$7.75\!\pm\!0.31$
mc	$50.27\!\pm\!0.00$	$2.000\!\pm\!0.000$	$2.000\!\pm\!0.000$	$2.000\!\pm\!0.000$	$12.64\!\pm\!0.88$	$11.95\!\pm\!0.31$

Table 1: Split conformal width at nominal coverage

1-\alpha=0.95

. Results for fraction of paired-label

1.0

are reported. For the nuPlan regression target ADE, width is the prediction-interval length

2\hat{q}

in meters; for the three nuPlan binary targets, width is the average conformal prediction-set size

|C(x)|\in[0,2]

. For quadruped Sim2Sim, width is the regression interval length reported as

\times 10^{-2}

. Cells are mean

\pm

standard error over 10 paired-data seeds. Bold marks the narrowest width per column.

Metric ( $\times 10^{-3}$ )	mc	cv	ppi	r_only	rs_mix	ppi-nn	Scape
$R_{\mathrm{vel}}$ test MAE	$16.55\!\pm\!0.96$	$17.07\!\pm\!0.86$	$26.51\!\pm\!1.52$	$13.29\!\pm\!0.76$	$16.92\!\pm\!0.99$	$19.16\!\pm\!1.79$	$\boldsymbol{11.79\!\pm\!0.70}$
$R_{\mathrm{yaw}}$ test MAE	$5.24\!\pm\!0.41$	$5.48\!\pm\!0.41$	$6.86\!\pm\!0.56$	$5.16\!\pm\!0.41$	$6.22\!\pm\!0.47$	$5.37\!\pm\!0.45$	$\boldsymbol{4.71\!\pm\!0.32}$

Table 2: Quadruped Sim2Real velocity tracking. Test MAE on the sim-to-real velocity tracking task. Cells are mean

\pm

standard error over 10 random paired-data seeds. Bold marks the lowest error per row.

Figure 5: Additional nuPlan results. Left: city-level OOD generalization results for UrbanDriver . Bars show mean $\pm$ standard error over 4 city holdouts. Right: UrbanDriver results at paired data fraction $1.0$ with varying surrogate-only fraction. Curves report mean $\pm$ standard error over 9 (3 paired-data seed $\times$ 3 surrogate-data seed) random seeds.

Figure 6: Evaluator guided planner selection accuracy and corresponding selected performance. Results for fraction of paired-label $1.0$ are reported. Learned evaluators are used to select the predicted best planner for each scenario. Bars show mean $\pm$ standard error over 10 paired-data seeds.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

e_{(1)}\leq\cdots\leq e_{(m)}.

Appendix

Algorithm 1 Scape

Planner	Split	Singapore	Boston	Las Vegas	Pittsburgh
Urban Driver Open-Loop	Train	19,504	20,145	23,833	20,123
	Val	3,555	3,574	4,547	3,791
	Test	6,993	6,347	7,737	7,188
	Calibration	264	254	355	276
	Surrogate	31,521	30,939	39,890	16,841
Vector Model	Train	19,504	20,145	23,833	20,123

Appendix

Table 3: Detailed city-level composition of the Scape nuPlan self-driving experiments dataset splits.

Figure 7: Paired real and reconstructed simulation scenes. The top row shows real-world scenes, and the bottom row shows their corresponding reconstructed simulation environments.

Sim2Sim
	scene1	scene2	scene3	scene4		Total
Paired ( $\mathcal{D}_{p}$ )	59	293	29	319		700
Surrogate ( $\mathcal{D}_{s}$ )	109	619	69	696		1,493
Sim2Real
	indoor	grass	dirt	dirt1	brick	Total
Paired ( $\mathcal{D}_{p}$ )	15	25	25	9	21	95

Appendix

Table 4: Detailed scene-level composition of the Scape quadruped velocity tracking dataset splits. For each setting, the paired pool (

\mathcal{D}_{p}

) is randomly shuffled and split into Train/Val/Test (

300/200/200

for Sim2Sim,

65/15/15

for Sim2Real) independently for each of the

10

random seeds, so we report the per-scene totals; surrogate samples (

\mathcal{D}_{s}

) are unsplit.

Type	Method	Definition
Statistical	MC	$\displaystyle\hat{\mu}_{\mathrm{MC}}=\frac{1}{n}\sum_{i}r_{i}$
Statistical	CV	$\displaystyle\hat{\mu}_{\mathrm{CV}}=\frac{1}{n}\sum_{i}(r_{i}-\hat{\beta}s_{i})+\frac{1}{\ell}\sum_{k}\hat{\beta}s_{k},\quad\hat{\beta}=\frac{\ell}{n+\ell}\frac{\widehat{\mathrm{Cov}}(s,r)}{\widehat{\mathrm{Var}}(s)}$
Statistical	PPI	$\displaystyle\hat{\mu}_{\mathrm{PPI}}=\frac{1}{\ell}\sum_{k}s_{k}+\frac{1}{n}\sum_{i}(r_{i}-s_{i})$
Neural	R-Only	$\displaystyle\mathcal{L}_{\mathrm{R}}=\frac{1}{n}\sum_{i}\mathcal{L}(f_{\theta}(x_{i}),r_{i})$
Neural	RS-Mix	$\displaystyle\mathcal{L}_{\mathrm{RS}}=\mathcal{L}_{\mathrm{R}}+\frac{1}{\ell}\sum_{k}\mathcal{L}(f_{\theta}(x_{k}),s_{k})$
Neural	PPI-NN	$\displaystyle\mathcal{L}_{\mathrm{PPI\text{-}NN}}=\frac{1}{\ell}\sum_{k}\mathcal{L}(f_{\theta}(x_{k}),s_{k})+\frac{1}{n}\sum_{i}\big[\mathcal{L}(f_{\theta}(x_{i}),r_{i})-\mathcal{L}(f_{\theta}(x_{i}),s_{i})\big]$

Appendix

Table 5: Baseline definitions. Statistical baselines estimate aggregate target performance, while neural baselines train scenario-level predictors

f_{\theta}(x)

Component	Scenario evaluator $f_{\theta}$	Correction model $g_{\phi}$
Input	$x\in\mathbb{R}^{256}$	$[x,s]\in\mathbb{R}^{260}$
Output	$\hat{r}=f_{\theta}(x)\in\mathbb{R}^{4}$	$\tilde{r}=g_{\phi}(x,s)\in\mathbb{R}^{4}$
Hidden layers	$[1024,512,256,128]$	$[1024,512,256,128]$
Activation	ReLU	ReLU
Dropout	$0.2$	$0.2$
Trainable parameters	$0.953$ M	$0.957$ M

Appendix

Table 6: MLP architecture and training hyperparameters (nuPlan). The scenario evaluator

f_{\theta}

is used by R-Only, RS-Mix, PPI-NN, and the final evaluator in Scape . The correction model

g_{\phi}

additionally takes the surrogate metric vector

s

as input and predicts corrected target metrics

\tilde{r}

. The four output dimensions correspond to ADE, TTC

<1

s, drivable-area compliance, and no ego at-fault collision.

Component	Scenario evaluator $f_{\theta}$	Correction model $g_{\phi}$
Input	$x\in\mathbb{R}^{d}$	$[x,s]\in\mathbb{R}^{d+2}$
Output	$\hat{r}=f_{\theta}(x)\in\mathbb{R}^{2}$	$\tilde{r}=g_{\phi}(x,s)\in\mathbb{R}^{2}$
Hidden layers	$[128,64,32]$	$[128,64,32]$
Activation	ReLU	ReLU
Dropout	$0.3$	$0.3$
Trainable parameters	$16.8$ K	$17.1$ K

Appendix

Table 7: MLP architecture and training hyperparameters (quadruped). The scenario evaluator

f_{\theta}

is used by R-Only, RS-Mix, PPI-NN, and the final evaluator in Scape . The correction model

g_{\phi}

additionally takes the surrogate metric vector

s

as input and predicts corrected target metrics

\tilde{r}

. The two output dimensions correspond to the per-step velocity tracking error

v

and yaw-rate tracking error

\omega

. Input dimensionality

d{=}51

(

44

-D

11{\times}4

ground-height crop

+

[v_{x},\omega_{z}]

command

+

5

-D scene one-hot) for the sim-to-real experiment, and

d{=}82

for the in-sim experiment.

Panel of Figure 8 — Figure 8: Additional scenario-level prediction results on nuPlan. Panels (a) and (b) report Vector and Simple Vector planner results, respectively. Test loss is reported as the paired-data fraction varies; curves show mean $\pm$ standard error over 10 random paired-data seeds.

Method	.1	.2	.3	.4	.5	.6	.7	.8	.9	1.0
Scape	$\boldsymbol{44.92\!\pm\!0.26}$	$\boldsymbol{43.23\!\pm\!0.23}$	$\boldsymbol{42.10\!\pm\!0.34}$	$\boldsymbol{41.69\!\pm\!0.22}$	$\boldsymbol{41.00\!\pm\!0.35}$	$\boldsymbol{40.64\!\pm\!0.26}$	$\boldsymbol{40.63\!\pm\!0.26}$	$\boldsymbol{40.35\!\pm\!0.26}$	$\boldsymbol{39.93\!\pm\!0.26}$	$\boldsymbol{40.07\!\pm\!0.26}$
PPI-NN	$45.21\!\pm\!0.26$	$45.11\!\pm\!0.21$	$44.91\!\pm\!0.32$	$44.76\!\pm\!0.18$	$44.60\!\pm\!0.09$	$44.25\!\pm\!0.16$	$44.55\!\pm\!0.18$	$44.44\!\pm\!0.14$	$44.17\!\pm\!0.21$	$44.44\!\pm\!0.16$
RS-Mix	$49.09\!\pm\!0.21$	$48.52\!\pm\!0.18$	$47.90\!\pm\!0.25$	$46.78\!\pm\!0.30$	$47.08\!\pm\!0.30$	$46.38\!\pm\!0.31$	$45.41\!\pm\!0.12$	$45.66\!\pm\!0.20$	$45.45\!\pm\!0.18$	$44.65\!\pm\!0.16$
R-Only	$46.11\!\pm\!0.22$	$44.73\!\pm\!0.35$	$43.82\!\pm\!0.18$	$43.68\!\pm\!0.31$	$42.80\!\pm\!0.30$	$42.27\!\pm\!0.22$	$42.00\!\pm\!0.22$	$41.98\!\pm\!0.21$	$40.99\!\pm\!0.16$	$41.02\!\pm\!0.30$
MC	$50.21\!\pm\!0.11$	$50.22\!\pm\!0.09$	$50.26\!\pm\!0.07$	$50.24\!\pm\!0.04$	$50.25\!\pm\!0.03$	$50.28\!\pm\!0.03$	$50.28\!\pm\!0.02$	$50.28\!\pm\!0.02$	$50.28\!\pm\!0.01$	$50.27\!\pm\!0.00$
CV	$50.37\!\pm\!0.04$	$50.32\!\pm\!0.03$	$50.31\!\pm\!0.02$	$50.29\!\pm\!0.02$	$50.29\!\pm\!0.02$	$50.29\!\pm\!0.01$	$50.29\!\pm\!0.01$	$50.29\!\pm\!0.01$	$50.29\!\pm\!0.00$	$50.29\!\pm\!0.00$

Appendix

Table 8: Full split conformal width sweep on nuPlan Urban Driver. Columns are paired-data fractions. Cells are mean

\pm

standard error over 10 paired-data seeds; bold marks the narrowest method per column. For ADE, width is interval length in meters; for binary targets, width is average prediction-set size. cov. reports empirical test coverage range across neural methods.

Method	.1	.2	.3	.4	.5	.6	.7	.8	.9	1.0
Scape	$48.78\!\pm\!0.47$	$\boldsymbol{46.67\!\pm\!0.44}$	$\boldsymbol{45.88\!\pm\!0.40}$	$\boldsymbol{44.82\!\pm\!0.25}$	$\boldsymbol{45.13\!\pm\!0.48}$	$\boldsymbol{43.64\!\pm\!0.35}$	$\boldsymbol{43.75\!\pm\!0.43}$	$\boldsymbol{43.77\!\pm\!0.41}$	$\boldsymbol{43.27\!\pm\!0.29}$	$\boldsymbol{43.43\!\pm\!0.29}$
PPI-NN	$51.20\!\pm\!0.29$	$51.54\!\pm\!0.20$	$50.72\!\pm\!0.22$	$50.44\!\pm\!0.26$	$50.29\!\pm\!0.21$	$50.06\!\pm\!0.30$	$49.96\!\pm\!0.18$	$49.88\!\pm\!0.27$	$50.33\!\pm\!0.24$	$49.88\!\pm\!0.14$
RS-Mix	$\boldsymbol{47.49\!\pm\!0.21}$	$46.91\!\pm\!0.24$	$46.11\!\pm\!0.20$	$46.29\!\pm\!0.22$	$46.13\!\pm\!0.18$	$45.59\!\pm\!0.24$	$45.16\!\pm\!0.20$	$45.22\!\pm\!0.27$	$44.98\!\pm\!0.24$	$44.98\!\pm\!0.27$
R-Only	$51.47\!\pm\!0.34$	$50.26\!\pm\!0.39$	$49.28\!\pm\!0.45$	$48.60\!\pm\!0.45$	$48.38\!\pm\!0.31$	$47.70\!\pm\!0.28$	$47.17\!\pm\!0.39$	$47.20\!\pm\!0.27$	$47.04\!\pm\!0.23$	$46.82\!\pm\!0.23$
MC	$56.23\!\pm\!0.09$	$56.25\!\pm\!0.09$	$56.30\!\pm\!0.07$	$56.32\!\pm\!0.05$	$56.31\!\pm\!0.03$	$56.32\!\pm\!0.03$	$56.31\!\pm\!0.02$	$56.30\!\pm\!0.01$	$56.26\!\pm\!0.01$	$56.26\!\pm\!0.00$
CV	$56.48\!\pm\!0.04$	$56.47\!\pm\!0.03$	$56.49\!\pm\!0.02$	$56.47\!\pm\!0.02$	$56.45\!\pm\!0.02$	$56.45\!\pm\!0.01$	$56.44\!\pm\!0.01$	$56.43\!\pm\!0.01$	$56.41\!\pm\!0.01$	$56.41\!\pm\!0.00$

Appendix

Table 9: Full split conformal width sweep on nuPlan Vector. Columns are paired-data fractions. Cells are mean

\pm

Method	.1	.2	.3	.4	.5	.6	.7	.8	.9	1.0
Scape	$67.63\!\pm\!0.51$	$\boldsymbol{66.47\!\pm\!0.33}$	$66.63\!\pm\!0.23$	$\boldsymbol{66.21\!\pm\!0.20}$	$\boldsymbol{65.63\!\pm\!0.18}$	$\boldsymbol{65.94\!\pm\!0.33}$	$\boldsymbol{65.67\!\pm\!0.39}$	$\boldsymbol{65.73\!\pm\!0.27}$	$\boldsymbol{65.26\!\pm\!0.27}$	$\boldsymbol{65.20\!\pm\!0.26}$
PPI-NN	$70.42\!\pm\!0.24$	$68.72\!\pm\!0.35$	$68.72\!\pm\!0.39$	$67.32\!\pm\!0.45$	$67.94\!\pm\!0.32$	$68.18\!\pm\!0.22$	$67.80\!\pm\!0.37$	$67.99\!\pm\!0.30$	$67.54\!\pm\!0.37$	$67.23\!\pm\!0.23$
RS-Mix	$\boldsymbol{66.75\!\pm\!0.24}$	$67.13\!\pm\!0.27$	$\boldsymbol{66.35\!\pm\!0.26}$	$66.59\!\pm\!0.28$	$66.80\!\pm\!0.39$	$66.17\!\pm\!0.24$	$66.36\!\pm\!0.45$	$66.38\!\pm\!0.18$	$66.78\!\pm\!0.36$	$65.82\!\pm\!0.34$
R-Only	$69.54\!\pm\!0.45$	$68.61\!\pm\!0.55$	$67.53\!\pm\!0.35$	$67.64\!\pm\!0.36$	$67.53\!\pm\!0.37$	$66.82\!\pm\!0.34$	$66.90\!\pm\!0.29$	$66.57\!\pm\!0.16$	$66.62\!\pm\!0.18$	$65.90\!\pm\!0.21$
MC	$105.42\!\pm\!0.16$	$105.26\!\pm\!0.10$	$105.25\!\pm\!0.08$	$105.24\!\pm\!0.07$	$105.28\!\pm\!0.06$	$105.27\!\pm\!0.04$	$105.27\!\pm\!0.03$	$105.22\!\pm\!0.02$	$105.19\!\pm\!0.01$	$105.18\!\pm\!0.00$
CV	$105.49\!\pm\!0.09$	$105.36\!\pm\!0.05$	$105.31\!\pm\!0.04$	$105.29\!\pm\!0.04$	$105.31\!\pm\!0.03$	$105.31\!\pm\!0.03$	$105.33\!\pm\!0.02$	$105.29\!\pm\!0.01$	$105.28\!\pm\!0.01$	$105.28\!\pm\!0.00$

Appendix

Table 10: Full split conformal width sweep on nuPlan Simple Vector. Columns are paired-data fractions. Cells are mean

\pm

Method	.1	.2	.3	.4	.5	.6	.7	.8	.9	1.0
Scape	$\boldsymbol{11.37\!\pm\!0.34}$	$\boldsymbol{11.05\!\pm\!0.48}$	$\boldsymbol{11.05\!\pm\!0.41}$	$\boldsymbol{11.04\!\pm\!0.48}$	$\boldsymbol{11.08\!\pm\!0.59}$	$\boldsymbol{10.98\!\pm\!0.46}$	$\boldsymbol{10.97\!\pm\!0.50}$	$\boldsymbol{11.02\!\pm\!0.58}$	$\boldsymbol{10.75\!\pm\!0.55}$	$\boldsymbol{10.84\!\pm\!0.48}$
PPI-NN	$12.42\!\pm\!0.83$	$12.57\!\pm\!1.27$	$11.16\!\pm\!0.44$	$11.75\!\pm\!1.00$	$11.26\!\pm\!0.45$	$12.01\!\pm\!0.51$	$11.39\!\pm\!0.57$	$11.55\!\pm\!0.39$	$11.14\!\pm\!0.34$	$11.42\!\pm\!0.38$
RS-Mix	$12.39\!\pm\!0.36$	$12.61\!\pm\!0.44$	$12.39\!\pm\!0.40$	$12.10\!\pm\!0.33$	$12.34\!\pm\!0.39$	$12.34\!\pm\!0.34$	$12.29\!\pm\!0.45$	$12.07\!\pm\!0.29$	$12.16\!\pm\!0.43$	$11.56\!\pm\!0.28$
R-Only	$12.05\!\pm\!0.42$	$12.11\!\pm\!0.47$	$11.44\!\pm\!0.47$	$11.57\!\pm\!0.49$	$11.53\!\pm\!0.47$	$11.27\!\pm\!0.40$	$11.06\!\pm\!0.52$	$11.45\!\pm\!0.47$	$10.86\!\pm\!0.48$	$11.12\!\pm\!0.59$
MC	$13.18\!\pm\!0.84$	$13.02\!\pm\!0.92$	$12.51\!\pm\!0.88$	$12.85\!\pm\!0.87$	$12.84\!\pm\!0.90$	$12.67\!\pm\!0.83$	$12.84\!\pm\!0.86$	$12.60\!\pm\!0.87$	$12.64\!\pm\!0.88$	$12.64\!\pm\!0.88$
CV	$12.82\!\pm\!0.85$	$12.57\!\pm\!0.87$	$12.46\!\pm\!0.87$	$12.57\!\pm\!0.85$	$12.51\!\pm\!0.85$	$12.55\!\pm\!0.81$	$12.57\!\pm\!0.87$	$12.48\!\pm\!0.85$	$12.51\!\pm\!0.87$	$12.50\!\pm\!0.88$

Appendix

Table 11: Full split conformal width sweep on Go2 Sim2Sim. Columns are paired-data fractions. Cells are mean

\pm

standard error over 10 paired-data seeds; bold marks the narrowest method per column. Width is regression interval length reported as

\times 10^{-2}

. cov. reports empirical test coverage range across neural methods.

Panel of Figure 9 — Figure 9: Additional city-level OOD generalization results on nuPlan. Panels (a) and (b) report results for the Vector and Simple Vector planners, respectively. Results compare full training, OOD training with surrogate-only data from the removed city, and OOD training without removed-city surrogate data. Results are averaged over four leave-one-city-out settings.

Panel of Figure 10 — Figure 10: Surrogate scaling and correction capacity on nuPlan Urban Driver. Paired-data and surrogate-only fractions are varied jointly for Scape , PPI-NN, and RS-Mix. Results are averaged over $3\times 3$ paired-data and surrogate-data seeds. Square markers denote the best surrogate-only fraction for each paired-data fraction.

Panel of Figure 11 — Figure 11: Additional evaluator-guided planner selection results on nuPlan. Results are shown across paired-data fractions $\rho_{p}\in\{0.1,\ldots,0.9\}$ , where $\rho_{p}$ denotes the fraction of the full paired training set $\mathcal{D}_{p}$ used for training. Bars report mean $\pm$ standard error over 10 paired-data seeds.

Explore similar work

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation
Jun 9, 2026Shuo Wang, Hanyuan Xu, Yingdong Hu +2Robotics Simulation Sim-to-Real Transfer
Tsinghua University · Shanghai Qi Zhi Institute
SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
Jun 26, 2026Nadun Ranawaka, Josiah Wong, Wei-Lin Pai +15Robotics Simulation Sim-to-Real Transfer
NVIDIA · Stanford University · The University of Texas at Austin +2
Sim-to-Real Aware End-to-End Learning Environment for Micromobility
Sep 24, 2026Shouma Amano, Takuya AzumiEnd-to-End Autonomous Driving Sim-to-Real Transfer
Saitama University · Academic Association (Graduate School of Science and Engineering) Saitama University

Jun 9, 2026, cs.ROJ/K move · Enter open · S save

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

Shuo Wang, Hanyuan Xu, Yingdong Hu, Fanqi Lin+1

Tsinghua University · Shanghai Qi Zhi Institute

Robotics Simulation Sim-to-Real Transfer Vision-Language-Action Models

Open paper AskPDF arXiv

Simulation has become an essential tool for evaluating and improving vision-language-action (VLA) policies, offering scalable, reproducible, and controllable alternatives to costly real-world robot evaluation. Recent simulation benchmarks have made substantial progress on realism and diversity, yet these platforms have not been widely adopted as reliable proxies for real-world policy evaluation. In this work, we investigate this issue through the lens of sim-and-real correlation. We conduct a systematic study across multiple simulation platforms, VLA policies, tasks, and perturbation factors, measuring whether simulated evaluation preserves real-world conclusions in terms of policy ranking consistency, performance correlation, and perturbation-wise failure patterns. This analysis allows us to characterize the limitations of existing simulators and identify what kinds of simulation signals are more aligned with real-world deployment. We further examine how users should exploit simulation for policy improvement, including when simulator-based finetuning is beneficial and how the amount of post-training data affects sim-and-real alignment. Overall, our work provides a unified framework for measuring, interpreting, and improving the usefulness of simulation for VLA policies, offering guidance both for simulator designers and for practitioners who use simulation as part of the policy development pipeline.

Abstract

Explore similar work

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Sim-to-Real Aware End-to-End Learning Environment for Micromobility