RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

Jenai Xuning Yang; Rishit Dagli; Alex Zook; Hugo Hadfield; Ankit Goyal; Stan Birchfield; Fabio Ramos; Jonathan Tremblay

cs.ROApr 10, 2026

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

Authors: Jenai Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, Jonathan Tremblay

Organizations: NVIDIA · University of Toronto · Work done during internship at NVIDIA · The University of Sydney

Abstract

The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a high-fidelity simulation environment. We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies. Project website: https://research.nvidia.com/labs/srl/projects/robolab/.

Figures & tables

Fig. 2 : Three approaches for robotic benchmarks. Left: To date, pure simulation based benchmarks have exhibited low visual quality, creating a large sim2real transfer gap. Middle: Real2sim benchmarks address this issue by using techniques to bring real-world visual texture into simulation. However, these environments are extremely costly with reported per-scene generation time of $\sim$ 1hr [ 9 ] . Right: Our approach achieves a high degree of realism with low overhead.

Fig. 3 : Task progression of a few tasks, illustrating errors encountered during policy rollout. Top row: Although the task is successfully completed, errors were encountered during execution: 1) The robot drops the milk jug too early, missing the bin. 2) the robot grasps an orange (wrong object) and puts it in the bin. Mid row: An extraneous object was reoriented before the actual intended object. Final row: Intended objects were attempted unsuccessfully, and the policy tended to two wrong other objects.

Fig. 4 : Example of language instructions in RoboLab-120.

Fig. 5 : Comparison of policy performance for bowl-in-bin manipulation. Rows represent distinct policies shown in chronological order (left to right). Successful execution involves grasping the central red bowl and depositing it into the gray bin on the right. Unsuccessful attempts are characterized by aimless arm trajectories and a lack of object interaction.

Fig. 6 : Example scene variations, lighting variations, and camera pose variations in RoboLab.

Overall Metrics

Difficulty (succ% / score)

Procedural (succ% / score)

Relational (succ% / score)

Visual (succ% / score)

Model

Succ% / Score (

\uparrow

)

SPARC (

\uparrow

)

Speed (

\uparrow

)

simple

moderate

complex

affordance

reorientation

stacking

conjunction

counting

spatial

color

semantics

size

\pi_{0.5}

[ 26 ]

28.0 / 0.43

-

8.34

\pm

6.7

5.4

\pm

1.6

29.7 / 0.39

31.5 / 0.51

13.5 / 0.44

10.0 / 0.27

28.3 / 0.48

35.0 / 0.57

43.8 / 0.55

65.7 / 0.75

21.0 / 0.36

23.8 / 0.42

23.2 / 0.39

30.0 / 0.36

\pi_{0}\text{-FAST}

[ 25 ]

15.5 / 0.27

-

9.63

\pm

5.4

4.6

\pm

1.8

20.2 / 0.27

13.3 / 0.29

2.9 / 0.22

2.5 / 0.13

3.3 / 0.10

15.0 / 0.28

27.5 / 0.38

44.3 / 0.59

15.9 / 0.30

6.2 / 0.17

10.5 / 0.22

10.0 / 0.18

GR00T N1.6 [ 22 ]

7.2 / 0.17

-

6.87

\pm

4.6

4.3

\pm

1.4

8.8 / 0.14

7.9 / 0.25

0.0 / 0.12

0.8 / 0.11

0.0 / 0.03

8.3 / 0.18

10.0 / 0.14

18.6 / 0.34

7.2 / 0.22

3.8 / 0.16

6.8 / 0.17

0.0 / 0.03

\pi_{0}

[ 4 ]

5.0 / 0.12

-

9.49

\pm

4.1

4.2

\pm

1.3

7.2 / 0.10

3.6 / 0.18

0.0 / 0.09

0.0 / 0.07

1.7 / 0.06

0.0 / 0.05

12.5 / 0.21

11.4 / 0.28

3.1 / 0.16

1.2 / 0.08

2.3 / 0.09

1.7 / 0.03

PaliGemma [ 3 ]

3.4 / 0.10

-

16.52

\pm

10.2

1.9

\pm

1.6

3.4 / 0.06

4.9 / 0.16

0.0 / 0.09

0.0 / 0.01

1.7 / 0.07

0.0 / 0.06

3.8 / 0.06

22.9 / 0.36

1.7 / 0.12

0.8 / 0.09

3.5 / 0.09

0.0 / 0.05

TABLE I : Overall performance of SOTA policies on RoboLab. While recent foundation models exhibit emerging capabilities across diverse task dimensions, overall performance remain limited. See Table IV and Table V for more granular breakdown of performance trajectory-quality results.

Fig. 7 : Examples of language ablation experiments. Top: Same scene and goal, but the instruction wording ranges from precise to increasingly vague. Middle: Same scene, but the instruction specifies different tasks to perform. Bottom: Same instruction, but the scene becomes progressively more complex.

Panel of Figure 8 — Fig. 8 : Effect of language, scene complexity, and task horizon on policy performance (success rate). Performance using success rate across complexity axes. For a robust policy, performance should relatively comparable as complexity increases. (a) Performance degrades as instructions become more abstract, indicating brittle task-level reasoning. (b) Performance degrades sharply as the number of objects in the scene increases, exposing systematic geometric biases in current models. (c) Performance degrades as the task horizon increases, indicating limited multi-step reasoning across all evaluated policies. Fig. 12 shows the same analysis with normalized scores for a more granular analysis.

Fig. 9 : Results of the sensitivity analysis using MNPE. Policies were highly sensitive to wrist-camera displacement from the nominal pose, indicating strong dependence on wrist-mounted camera calibration. Success also peaked for objects placed at approximately 0.5m from the robot, likely due to robot reachability.

Fig. 10 : Correlation between RoboLab-120 success rate and RoboArena Elo score across the four policies with both measurements available ( $\pi_{0.5}$ , $\pi_{0}\text{-FAST}$ , $\pi_{0}$ , PaliGemma). Rankings are preserved (Spearman $\rho=1.00$ ) and the scores are positively correlated (Pearson $r=0.68$ ), suggesting RoboLab-120 performance is a useful proxy for real-world policy quality.

	$\pi_{0.5}$		$\pi_{0}\text{-FAST}$		$\pi_{0}$
Variation	Succ.%	Time (s)	Succ.%	Time (s)	Succ.%	Time (s)
Lighting
Color	96.7	14.5 $\pm$ 7.9	93.3	17.9 $\pm$ 10.7	6.7	31.1 $\pm$ 4.3
Shadows	100.0	16.0 $\pm$ 6.0	90.0	12.4 $\pm$ 3.5	0.0	-
Dim	90.0	9.1 $\pm$ 2.1	70.0	13.1 $\pm$ 2.8	70.0	35.5 $\pm$ 9.7
Overexposed	100.0	13.9 $\pm$ 4.4	100.0	9.6 $\pm$ 1.7	0.0	-

TABLE II : Robustness to controlled environmental variations over two simple tasks ( BananaInBowl , BananaAndCubeInBowl ). PaliGemma is excluded as it fails to achieve meaningful results.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Pipeline (per 1 scene 1 task)	O/H
RobotArena $\infty$	VLM+2D bbox detector per frame $\rightarrow$ 2D‑to‑3D diffusion $\rightarrow$ 3D asset retrieval $\rightarrow$ diff. rendering	High
Polaris	RGB-D video $\rightarrow$ camera calib. $\rightarrow$ 3DGS $\rightarrow$ mesh extraction, semantic segmentation $\rightarrow$ physics	High (1hr)
LIBERO	Object+placement+task+goal (python) $\rightarrow$ BDDL	High
RoboLab	Drag-and-drop $\rightarrow$ text instructions	Low

Appendix

TABLE III : Comparison of task creation pipelines across benchmarks. O/H denotes overhead.

Panel of Figure 13 — Fig. 11 : Initial scene and final-frame viewports from two successful $\pi_{0.5}$ rollouts on CubesAndBlocksInBinTask (7 subtasks). While the task-horizon is long, the task itself is non-causal. The policy was able to make significant progress and completion on this task.

Task Categories	#	$\pi_{0.5}$			$\pi_{0}\text{-FAST}$			GR00T N1.6			$\pi_{0}$			PaliGemma
Task Categories	#	Succ %	Score	Score (fail)	Succ %	Score	Score (fail)	Succ %	Score	Score (fail)	Succ %	Score	Score (fail)	Succ %	Score	Score (fail)
Total	120	28.0%	0.43	0.21	15.5%	0.27	0.13	7.2%	0.17	0.11	5.0%	0.12	0.08	3.4%	0.10	0.07
simple	64	29.7%	0.39	0.13	20.2%	0.27	0.08	8.8%	0.14	0.05	7.2%	0.10	0.03	3.4%	0.06	0.03
moderate	39	31.5%	0.51	0.28	13.3%	0.29	0.18	7.9%	0.25	0.19	3.6%	0.18	0.15	4.9%	0.16	0.12
complex	17	13.5%	0.44	0.35	2.9%	0.22	0.20	0.0%	0.12	0.12	0.0%	0.09	0.09	0.0%	0.09	0.09
Procedural	22	21.8%	0.42	0.25	6.4%	0.16	0.11	2.7%	0.11	0.09	0.5%	0.07	0.06	0.5%	0.04	0.04

Appendix

TABLE IV : Capability metrics per task category on RoboLab. “Succ%” is the fraction of episodes that fully complete the task; “Score” is the mean per-episode normalized score awarded for partial subtask completion (Sec. III-C ); “Score (fail)” is the mean score of failed episodes only, which isolates partial progress on episodes that did not complete. Trajectory-quality metrics (SPARC, Speed) for the same categories are reported in Table V .

Task Categories	#	$\pi_{0.5}$		$\pi_{0}\text{-FAST}$		GR00T N1.6		$\pi_{0}$		PaliGemma
Task Categories	#	SPARC	Speed (cm/s)	SPARC	Speed (cm/s)	SPARC	Speed (cm/s)	SPARC	Speed (cm/s)	SPARC	Speed (cm/s)
Total	120	$-$ 8.34 ( $\pm$ 6.65)	5.4 ( $\pm$ 1.6)	$-$ 9.63 ( $\pm$ 5.40)	4.6 ( $\pm$ 1.8)	$-$ 6.87 ( $\pm$ 4.63)	4.3 ( $\pm$ 1.4)	$-$ 9.49 ( $\pm$ 4.12)	4.2 ( $\pm$ 1.3)	$-$ 16.52 ( $\pm$ 10.21)	1.9 ( $\pm$ 1.6)
simple	64	$-$ 7.38 ( $\pm$ 6.66)	5.4 ( $\pm$ 1.4)	$-$ 8.09 ( $\pm$ 4.41)	4.7 ( $\pm$ 1.7)	$-$ 6.37 ( $\pm$ 5.00)	4.5 ( $\pm$ 1.4)	$-$ 8.46 ( $\pm$ 4.37)	4.3 ( $\pm$ 1.4)	$-$ 15.33 ( $\pm$ 8.73)	2.0 ( $\pm$ 1.7)
moderate	39	$-$ 8.85 ( $\pm$ 7.21)	5.4 ( $\pm$ 1.8)	$-$ 10.07 ( $\pm$ 5.44)	4.7 ( $\pm$ 2.1)	$-$ 6.92 ( $\pm$ 4.21)	4.2 ( $\pm$ 1.3)	$-$ 10.13 ( $\pm$ 2.99)	4.1 ( $\pm$ 1.3)	$-$ 15.02 ( $\pm$ 7.52)	2.1 ( $\pm$ 1.7)
complex	17	$-$ 10.83 ( $\pm$ 3.89)	4.8 ( $\pm$ 1.3)	$-$ 14.38 ( $\pm$ 5.75)	4.1 ( $\pm$ 1.3)	$-$ 8.63 ( $\pm$ 3.52)	4.1 ( $\pm$ 1.4)	$-$ 11.91 ( $\pm$ 4.09)	3.8 ( $\pm$ 1.1)	$-$ 24.46 ( $\pm$ 15.68)	1.5 ( $\pm$ 0.8)
Procedural	22	$-$ 10.08 ( $\pm$ 4.93)	4.8 ( $\pm$ 1.4)	$-$ 11.06 ( $\pm$ 6.31)	4.3 ( $\pm$ 1.4)	$-$ 6.98 ( $\pm$ 5.16)	3.8 ( $\pm$ 1.3)	$-$ 10.58 ( $\pm$ 2.96)	4.1 ( $\pm$ 1.1)	$-$ 16.07 ( $\pm$ 7.69)	1.5 ( $\pm$ 0.8)

Appendix

TABLE V : Trajectory-quality metrics per task category on RoboLab. “SPARC” is the spectral arc length of the end-effector velocity profile (Sec. III-C )—smoother motions yield values closer to zero, jerkier motions are more negative. “Speed” is the mean end-effector speed in cm/s. Standard deviations across episodes are shown in gray. Capability metrics for the same categories are reported in Table IV .

Panel of Figure 16 — Fig. 12 : Score (partial credit) across complexity axes. Same complexity axes as Fig. 8 , but plotting score (which awards partial credit for failed episodes) instead of success rate. Shaded bands show SE. Score degrades more gracefully than strict success, indicating policies make meaningful progress even when they fail to fully complete the task.

Fig. 13 : ( Left ) We show one of our Gaussian Splat + Mesh scenes in RoboLab. This scene has a Gaussian splat background with a collision mesh for the splat estimated with 3DGRUT [ 20 , 31 ] , and a mesh foreground. All objects in the scene have spatially varying density, and thus mass is estimated with VoMP [ 5 ] . ( Right ) We show a VLA running a task in this scene.

Panel of Figure 18 — Fig. 14 : System prompt for Stage I (Semantic Planning). This prompt instructs the LLM to generate physically plausible scene layouts using structured predicates rather than raw coordinates.

Panel of Figure 19 — Fig. 15 : Continued System prompt for Stage I (Semantic Planning). This prompt instructs the LLM to generate physically plausible scene layouts using structured predicates rather than raw coordinates.

Panel of Figure 20 — Fig. 16 : User prompt template for Stage I (medium target count). The highlighted fields are populated at runtime (theme, target count, size warnings, catalog subsets, and diversity suggestions). Analogous strategy blocks are used for sparse (fewer than 10) and dense (15+) targets.

Panel of Figure 21 — Fig. 17 : Feedback block appended to the user prompt when spatial solving, physical placement, grammar checks, or intersection validation fails. The highlighted region is the dynamic diagnostic message.

Method	VQA ( $\uparrow$ )	Real. ( $\uparrow$ )	Func. ( $\uparrow$ )	Lay. ( $\uparrow$ )	Compl. ( $\uparrow$ )	Qual. ( $\uparrow$ )	# Obj ( $\uparrow$ )	GPT Pref. ( $\uparrow$ )
Baseline	0.398 ( $\pm$ 0.04)	6.889 ( $\pm$ 0.36)	6.221 ( $\pm$ 0.58)	6.166 ( $\pm$ 0.42)	4.687 ( $\pm$ 0.57)	5.991 ( $\pm$ 0.24)	13.750 ( $\pm$ 10.52)	18.000
Ours	0.554 ( $\pm$ 0.03)	8.755 ( $\pm$ 0.27)	8.951 ( $\pm$ 0.33)	7.919 ( $\pm$ 0.28)	8.207 ( $\pm$ 0.40)	8.458 ( $\pm$ 0.25)	26.870 ( $\pm$ 24.90)	82.000

Appendix

TABLE VI : Quantitative comparison for Scene Generation. We evaluate our against the baseline across diverse metrics measuring the visual realism (Real.), functionality (Func.), layout correctness (Lay.), Quality (Qual.), VQA score [ 17 ] , and GPT Preference.

Method	VQA ( $\uparrow$ )	Real. ( $\uparrow$ )	Func. ( $\uparrow$ )	Lay. ( $\uparrow$ )	Compl. ( $\uparrow$ )	Qual. ( $\uparrow$ )	# Obj ( $\uparrow$ )	GPT Pref. ( $\uparrow$ )
Easy ( $[0,5]$ objects)
Baseline	0.458 ( $\pm$ 0.02)	6.767 ( $\pm$ 0.36)	6.269 ( $\pm$ 0.57)	6.079 ( $\pm$ 0.48)	4.737 ( $\pm$ 0.61)	5.963 ( $\pm$ 0.26)	6.467 ( $\pm$ 0.52)	33.333
Ours	0.525 ( $\pm$ 0.05)	8.331 ( $\pm$ 0.24)	8.423 ( $\pm$ 0.27)	7.636 ( $\pm$ 0.21)	7.626 ( $\pm$ 0.25)	8.004 ( $\pm$ 0.09)	12.800 ( $\pm$ 11.54)	66.667
Medium ( $[6,15]$ objects)
Baseline	0.401 ( $\pm$ 0.02)	6.933 ( $\pm$ 0.37)	6.223 ( $\pm$ 0.59)	6.199 ( $\pm$ 0.41)	4.634 ( $\pm$ 0.56)	5.997 ( $\pm$ 0.22)	10.957 ( $\pm$ 2.11)	17.143
Ours	0.561 ( $\pm$ 0.02)	8.779 ( $\pm$ 0.18)	8.996 ( $\pm$ 0.23)	7.932 ( $\pm$ 0.24)	8.235 ( $\pm$ 0.29)	8.485 ( $\pm$ 0.13)	22.414 ( $\pm$ 19.15)	82.857

Appendix

TABLE VII : Quantitative comparison across Difficulty Splits. We evaluate our method against the baseline on Easy, Medium, and Hard splits. Our method consistently outperforms the baseline across all difficulty levels.

Theme	Method	VQA ( $\uparrow$ )	Real. ( $\uparrow$ )	Func. ( $\uparrow$ )	Lay. ( $\uparrow$ )	Compl. ( $\uparrow$ )	Qual. ( $\uparrow$ )	# Obj ( $\uparrow$ )	GPT Pref. ( $\uparrow$ )
Bathroom Counter	Baseline	0.405 ( $\pm$ 0.02)	6.901 ( $\pm$ 0.37)	6.196 ( $\pm$ 0.73)	6.260 ( $\pm$ 0.26)	4.805 ( $\pm$ 0.56)	6.041 ( $\pm$ 0.19)	15.00 ( $\pm$ 0.00)	10.00
Bathroom Counter	Ours	0.564 ( $\pm$ 0.02)	8.913 ( $\pm$ 0.16)	9.159 ( $\pm$ 0.22)	7.967 ( $\pm$ 0.28)	8.276 ( $\pm$ 0.33)	8.579 ( $\pm$ 0.15)	28.50 ( $\pm$ 18.47)	90.00
Classroom Supplies	Baseline	0.401 ( $\pm$ 0.02)	6.881 ( $\pm$ 0.31)	6.458 ( $\pm$ 0.50)	5.931 ( $\pm$ 0.41)	5.018 ( $\pm$ 0.55)	6.072 ( $\pm$ 0.17)	10.40 ( $\pm$ 1.26)	50.00
Classroom Supplies	Ours	0.562 ( $\pm$ 0.02)	8.790 ( $\pm$ 0.18)	8.990 ( $\pm$ 0.18)	7.842 ( $\pm$ 0.28)	8.406 ( $\pm$ 0.26)	8.507 ( $\pm$ 0.12)	32.90 ( $\pm$ 23.70)	50.00
Craft Station	Baseline	0.399 ( $\pm$ 0.02)	7.062 ( $\pm$ 0.49)	6.385 ( $\pm$ 0.56)	6.171 ( $\pm$ 0.39)	4.590 ( $\pm$ 0.60)	6.052 ( $\pm$ 0.24)	9.70 ( $\pm$ 0.48)	10.00
Craft Station	Ours	0.560 ( $\pm$ 0.03)	8.761 ( $\pm$ 0.19)	9.045 ( $\pm$ 0.28)	7.938 ( $\pm$ 0.22)	8.179 ( $\pm$ 0.33)	8.481 ( $\pm$ 0.06)	17.70 ( $\pm$ 13.61)	90.00

Appendix

TABLE VIII : Per-Scene Quantitative Analysis. We report the performance breakdown across 10 distinct scene themes. Our method demonstrates robust generalization, outperforming the baseline in nearly all metrics across diverse environments.

		LLM Judge						Coverage
Category	$N$	Alignment	Clarity	Feasibility	Match	Aligned%	Partial%	Object	Predicate
color	116	0.81	0.94	0.80	0.90	57	40	0.88	0.29
conjunction	116	0.97	0.98	1.00	0.98	91	9	0.88	0.29
counting	116	0.87	0.97	0.90	0.92	60	38	0.88	0.29
recognition	116	0.96	0.97	0.96	0.97	85	15	0.88	0.29
semantics	116	0.89	0.95	0.94	0.94	72	27	0.88	0.29

Appendix

TABLE IX : LLM-judged quality metrics for

812

automatically generated manipulation tasks across

59

scenes and

7

competency axes.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 114 papers

RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
Jul 5, 2026Tianxing Chen, Yue Chen, Zixuan Li +41Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
X2Real: an eXtensive simulation benchmark for real-world generalist policies
Sep 23, 2026Lian Ruan, Jade Yang, Sherphylan Gao +23Robotic Policy Evaluation Robot Manipulation Benchmarks
X Square Robot
RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots
Mar 4, 2026Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri +1Robot Manipulation Benchmarks Mobile Manipulation
The University of Texas at Austin · NVIDIA Research

Jul 5, 2026J/K move · Enter open · S save

RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang+40

Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation Robotics Simulation Generalization in Robotic Manipulation Robotic Policy Evaluation Robot Manipulation Benchmarks+1

Open paper AskPDF arXiv

Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.

Abstract

Robotic Policy Evaluation

Explore similar work

RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

X2Real: an eXtensive simulation benchmark for real-world generalist policies

RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots