The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a high-fidelity simulation environment. We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies. Project website: https://research.nvidia.com/labs/srl/projects/robolab/.
Figures & tables
Fig. 2 : Three approaches for robotic benchmarks. Left: To date, pure simulation based benchmarks have exhibited low visual quality, creating a large sim2real transfer gap. Middle: Real2sim benchmarks address this issue by using techniques to bring real-world visual texture into simulation. However, these environments are extremely costly with reported per-scene generation time of ∼ 1hr [ 9 ] . Right: Our approach achieves a high degree of realism with low overhead.
Fig. 3 : Task progression of a few tasks, illustrating errors encountered during policy rollout. Top row: Although the task is successfully completed, errors were encountered during execution: 1) The robot drops the milk jug too early, missing the bin. 2) the robot grasps an orange (wrong object) and puts it in the bin. Mid row: An extraneous object was reoriented before the actual intended object. Final row: Intended objects were attempted unsuccessfully, and the policy tended to two wrong other objects.
Fig. 4 : Example of language instructions in RoboLab-120.
Fig. 5 : Comparison of policy performance for bowl-in-bin manipulation. Rows represent distinct policies shown in chronological order (left to right). Successful execution involves grasping the central red bowl and depositing it into the gray bin on the right. Unsuccessful attempts are characterized by aimless arm trajectories and a lack of object interaction.
Fig. 6 : Example scene variations, lighting variations, and camera pose variations in RoboLab.
Overall Metrics
Difficulty (succ% / score)
Procedural (succ% / score)
Relational (succ% / score)
Visual (succ% / score)
Model
Succ% / Score ( ↑ )
SPARC ( ↑ )
Speed ( ↑ )
simple
moderate
complex
affordance
reorientation
stacking
conjunction
counting
spatial
color
semantics
size
π0.5 [ 26 ]
28.0 / 0.43
− 8.34 ± 6.7
5.4 ± 1.6
29.7 / 0.39
31.5 / 0.51
13.5 / 0.44
10.0 / 0.27
28.3 / 0.48
35.0 / 0.57
43.8 / 0.55
65.7 / 0.75
21.0 / 0.36
23.8 / 0.42
23.2 / 0.39
30.0 / 0.36
π0-FAST [ 25 ]
15.5 / 0.27
− 9.63 ± 5.4
4.6 ± 1.8
20.2 / 0.27
13.3 / 0.29
2.9 / 0.22
2.5 / 0.13
3.3 / 0.10
15.0 / 0.28
27.5 / 0.38
44.3 / 0.59
15.9 / 0.30
6.2 / 0.17
10.5 / 0.22
10.0 / 0.18
GR00T N1.6 [ 22 ]
7.2 / 0.17
− 6.87 ± 4.6
4.3 ± 1.4
8.8 / 0.14
7.9 / 0.25
0.0 / 0.12
0.8 / 0.11
0.0 / 0.03
8.3 / 0.18
10.0 / 0.14
18.6 / 0.34
7.2 / 0.22
3.8 / 0.16
6.8 / 0.17
0.0 / 0.03
π0 [ 4 ]
5.0 / 0.12
− 9.49 ± 4.1
4.2 ± 1.3
7.2 / 0.10
3.6 / 0.18
0.0 / 0.09
0.0 / 0.07
1.7 / 0.06
0.0 / 0.05
12.5 / 0.21
11.4 / 0.28
3.1 / 0.16
1.2 / 0.08
2.3 / 0.09
1.7 / 0.03
PaliGemma [ 3 ]
3.4 / 0.10
− 16.52 ± 10.2
1.9 ± 1.6
3.4 / 0.06
4.9 / 0.16
0.0 / 0.09
0.0 / 0.01
1.7 / 0.07
0.0 / 0.06
3.8 / 0.06
22.9 / 0.36
1.7 / 0.12
0.8 / 0.09
3.5 / 0.09
0.0 / 0.05
TABLE I : Overall performance of SOTA policies on RoboLab. While recent foundation models exhibit emerging capabilities across diverse task dimensions, overall performance remain limited. See Table IV and Table V for more granular breakdown of performance trajectory-quality results.
Fig. 7 : Examples of language ablation experiments. Top: Same scene and goal, but the instruction wording ranges from precise to increasingly vague. Middle: Same scene, but the instruction specifies different tasks to perform. Bottom: Same instruction, but the scene becomes progressively more complex.
Fig. 8 : Effect of language, scene complexity, and task horizon on policy performance (success rate). Performance using success rate across complexity axes. For a robust policy, performance should relatively comparable as complexity increases. (a) Performance degrades as instructions become more abstract, indicating brittle task-level reasoning. (b) Performance degrades sharply as the number of objects in the scene increases, exposing systematic geometric biases in current models. (c) Performance degrades as the task horizon increases, indicating limited multi-step reasoning across all evaluated policies. Fig. 12 shows the same analysis with normalized scores for a more granular analysis.
Fig. 9 : Results of the sensitivity analysis using MNPE. Policies were highly sensitive to wrist-camera displacement from the nominal pose, indicating strong dependence on wrist-mounted camera calibration. Success also peaked for objects placed at approximately 0.5m from the robot, likely due to robot reachability.
Fig. 10 : Correlation between RoboLab-120 success rate and RoboArena Elo score across the four policies with both measurements available ( π0.5 , π0-FAST , π0 , PaliGemma). Rankings are preserved (Spearman ρ=1.00 ) and the scores are positively correlated (Pearson r=0.68 ), suggesting RoboLab-120 performance is a useful proxy for real-world policy quality.
π0.5
π0-FAST
π0
Variation
Succ.%
Time (s)
Succ.%
Time (s)
Succ.%
Time (s)
Lighting
Color
96.7
14.5 ± 7.9
93.3
17.9 ± 10.7
6.7
31.1 ± 4.3
Shadows
100.0
16.0 ± 6.0
90.0
12.4 ± 3.5
0.0
-
Dim
90.0
9.1 ± 2.1
70.0
13.1 ± 2.8
70.0
35.5 ± 9.7
Overexposed
100.0
13.9 ± 4.4
100.0
9.6 ± 1.7
0.0
-
TABLE II : Robustness to controlled environmental variations over two simple tasks ( BananaInBowl , BananaAndCubeInBowl ). PaliGemma is excluded as it fails to achieve meaningful results.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Pipeline (per 1 scene 1 task)
O/H
RobotArena ∞
VLM+2D bbox detector per frame → 2D‑to‑3D diffusion → 3D asset retrieval → diff. rendering
High
Polaris
RGB-D video → camera calib. → 3DGS → mesh extraction, semantic segmentation → physics
High (1hr)
LIBERO
Object+placement+task+goal (python) → BDDL
High
RoboLab
Drag-and-drop → text instructions
Low
Appendix
TABLE III : Comparison of task creation pipelines across benchmarks. O/H denotes overhead.
Fig. 11 : Initial scene and final-frame viewports from two successful π0.5 rollouts on CubesAndBlocksInBinTask (7 subtasks). While the task-horizon is long, the task itself is non-causal. The policy was able to make significant progress and completion on this task.
Task Categories
#
π0.5
π0-FAST
GR00T N1.6
π0
PaliGemma
Succ %
Score
Score (fail)
Succ %
Score
Score (fail)
Succ %
Score
Score (fail)
Succ %
Score
Score (fail)
Succ %
Score
Score (fail)
Total
120
28.0%
0.43
0.21
15.5%
0.27
0.13
7.2%
0.17
0.11
5.0%
0.12
0.08
3.4%
0.10
0.07
simple
64
29.7%
0.39
0.13
20.2%
0.27
0.08
8.8%
0.14
0.05
7.2%
0.10
0.03
3.4%
0.06
0.03
moderate
39
31.5%
0.51
0.28
13.3%
0.29
0.18
7.9%
0.25
0.19
3.6%
0.18
0.15
4.9%
0.16
0.12
complex
17
13.5%
0.44
0.35
2.9%
0.22
0.20
0.0%
0.12
0.12
0.0%
0.09
0.09
0.0%
0.09
0.09
Procedural
22
21.8%
0.42
0.25
6.4%
0.16
0.11
2.7%
0.11
0.09
0.5%
0.07
0.06
0.5%
0.04
0.04
Appendix
TABLE IV : Capability metrics per task category on RoboLab. “Succ%” is the fraction of episodes that fully complete the task; “Score” is the mean per-episode normalized score awarded for partial subtask completion (Sec. III-C ); “Score (fail)” is the mean score of failed episodes only, which isolates partial progress on episodes that did not complete. Trajectory-quality metrics (SPARC, Speed) for the same categories are reported in Table V .
Task Categories
#
π0.5
π0-FAST
GR00T N1.6
π0
PaliGemma
SPARC
Speed (cm/s)
SPARC
Speed (cm/s)
SPARC
Speed (cm/s)
SPARC
Speed (cm/s)
SPARC
Speed (cm/s)
Total
120
− 8.34 ( ± 6.65)
5.4 ( ± 1.6)
− 9.63 ( ± 5.40)
4.6 ( ± 1.8)
− 6.87 ( ± 4.63)
4.3 ( ± 1.4)
− 9.49 ( ± 4.12)
4.2 ( ± 1.3)
− 16.52 ( ± 10.21)
1.9 ( ± 1.6)
simple
64
− 7.38 ( ± 6.66)
5.4 ( ± 1.4)
− 8.09 ( ± 4.41)
4.7 ( ± 1.7)
− 6.37 ( ± 5.00)
4.5 ( ± 1.4)
− 8.46 ( ± 4.37)
4.3 ( ± 1.4)
− 15.33 ( ± 8.73)
2.0 ( ± 1.7)
moderate
39
− 8.85 ( ± 7.21)
5.4 ( ± 1.8)
− 10.07 ( ± 5.44)
4.7 ( ± 2.1)
− 6.92 ( ± 4.21)
4.2 ( ± 1.3)
− 10.13 ( ± 2.99)
4.1 ( ± 1.3)
− 15.02 ( ± 7.52)
2.1 ( ± 1.7)
complex
17
− 10.83 ( ± 3.89)
4.8 ( ± 1.3)
− 14.38 ( ± 5.75)
4.1 ( ± 1.3)
− 8.63 ( ± 3.52)
4.1 ( ± 1.4)
− 11.91 ( ± 4.09)
3.8 ( ± 1.1)
− 24.46 ( ± 15.68)
1.5 ( ± 0.8)
Procedural
22
− 10.08 ( ± 4.93)
4.8 ( ± 1.4)
− 11.06 ( ± 6.31)
4.3 ( ± 1.4)
− 6.98 ( ± 5.16)
3.8 ( ± 1.3)
− 10.58 ( ± 2.96)
4.1 ( ± 1.1)
− 16.07 ( ± 7.69)
1.5 ( ± 0.8)
Appendix
TABLE V : Trajectory-quality metrics per task category on RoboLab. “SPARC” is the spectral arc length of the end-effector velocity profile (Sec. III-C )—smoother motions yield values closer to zero, jerkier motions are more negative. “Speed” is the mean end-effector speed in cm/s. Standard deviations across episodes are shown in gray. Capability metrics for the same categories are reported in Table IV .
Fig. 12 : Score (partial credit) across complexity axes. Same complexity axes as Fig. 8 , but plotting score (which awards partial credit for failed episodes) instead of success rate. Shaded bands show SE. Score degrades more gracefully than strict success, indicating policies make meaningful progress even when they fail to fully complete the task.
Fig. 13 : ( Left ) We show one of our Gaussian Splat + Mesh scenes in RoboLab. This scene has a Gaussian splat background with a collision mesh for the splat estimated with 3DGRUT [ 20 , 31 ] , and a mesh foreground. All objects in the scene have spatially varying density, and thus mass is estimated with VoMP [ 5 ] . ( Right ) We show a VLA running a task in this scene.
Fig. 14 : System prompt for Stage I (Semantic Planning). This prompt instructs the LLM to generate physically plausible scene layouts using structured predicates rather than raw coordinates.
Fig. 15 : Continued System prompt for Stage I (Semantic Planning). This prompt instructs the LLM to generate physically plausible scene layouts using structured predicates rather than raw coordinates.
Fig. 16 : User prompt template for Stage I (medium target count). The highlighted fields are populated at runtime (theme, target count, size warnings, catalog subsets, and diversity suggestions). Analogous strategy blocks are used for sparse (fewer than 10) and dense (15+) targets.
Fig. 17 : Feedback block appended to the user prompt when spatial solving, physical placement, grammar checks, or intersection validation fails. The highlighted region is the dynamic diagnostic message.
Method
VQA ( ↑ )
Real. ( ↑ )
Func. ( ↑ )
Lay. ( ↑ )
Compl. ( ↑ )
Qual. ( ↑ )
# Obj ( ↑ )
GPT Pref. ( ↑ )
Baseline
0.398 ( ± 0.04)
6.889 ( ± 0.36)
6.221 ( ± 0.58)
6.166 ( ± 0.42)
4.687 ( ± 0.57)
5.991 ( ± 0.24)
13.750 ( ± 10.52)
18.000
Ours
0.554 ( ± 0.03)
8.755 ( ± 0.27)
8.951 ( ± 0.33)
7.919 ( ± 0.28)
8.207 ( ± 0.40)
8.458 ( ± 0.25)
26.870 ( ± 24.90)
82.000
Appendix
TABLE VI : Quantitative comparison for Scene Generation. We evaluate our against the baseline across diverse metrics measuring the visual realism (Real.), functionality (Func.), layout correctness (Lay.), Quality (Qual.), VQA score [ 17 ] , and GPT Preference.
Method
VQA ( ↑ )
Real. ( ↑ )
Func. ( ↑ )
Lay. ( ↑ )
Compl. ( ↑ )
Qual. ( ↑ )
# Obj ( ↑ )
GPT Pref. ( ↑ )
Easy ( [0,5] objects)
Baseline
0.458 ( ± 0.02)
6.767 ( ± 0.36)
6.269 ( ± 0.57)
6.079 ( ± 0.48)
4.737 ( ± 0.61)
5.963 ( ± 0.26)
6.467 ( ± 0.52)
33.333
Ours
0.525 ( ± 0.05)
8.331 ( ± 0.24)
8.423 ( ± 0.27)
7.636 ( ± 0.21)
7.626 ( ± 0.25)
8.004 ( ± 0.09)
12.800 ( ± 11.54)
66.667
Medium ( [6,15] objects)
Baseline
0.401 ( ± 0.02)
6.933 ( ± 0.37)
6.223 ( ± 0.59)
6.199 ( ± 0.41)
4.634 ( ± 0.56)
5.997 ( ± 0.22)
10.957 ( ± 2.11)
17.143
Ours
0.561 ( ± 0.02)
8.779 ( ± 0.18)
8.996 ( ± 0.23)
7.932 ( ± 0.24)
8.235 ( ± 0.29)
8.485 ( ± 0.13)
22.414 ( ± 19.15)
82.857
Appendix
TABLE VII : Quantitative comparison across Difficulty Splits. We evaluate our method against the baseline on Easy, Medium, and Hard splits. Our method consistently outperforms the baseline across all difficulty levels.
Theme
Method
VQA ( ↑ )
Real. ( ↑ )
Func. ( ↑ )
Lay. ( ↑ )
Compl. ( ↑ )
Qual. ( ↑ )
# Obj ( ↑ )
GPT Pref. ( ↑ )
Bathroom Counter
Baseline
0.405 ( ± 0.02)
6.901 ( ± 0.37)
6.196 ( ± 0.73)
6.260 ( ± 0.26)
4.805 ( ± 0.56)
6.041 ( ± 0.19)
15.00 ( ± 0.00)
10.00
Ours
0.564 ( ± 0.02)
8.913 ( ± 0.16)
9.159 ( ± 0.22)
7.967 ( ± 0.28)
8.276 ( ± 0.33)
8.579 ( ± 0.15)
28.50 ( ± 18.47)
90.00
Classroom Supplies
Baseline
0.401 ( ± 0.02)
6.881 ( ± 0.31)
6.458 ( ± 0.50)
5.931 ( ± 0.41)
5.018 ( ± 0.55)
6.072 ( ± 0.17)
10.40 ( ± 1.26)
50.00
Ours
0.562 ( ± 0.02)
8.790 ( ± 0.18)
8.990 ( ± 0.18)
7.842 ( ± 0.28)
8.406 ( ± 0.26)
8.507 ( ± 0.12)
32.90 ( ± 23.70)
50.00
Craft Station
Baseline
0.399 ( ± 0.02)
7.062 ( ± 0.49)
6.385 ( ± 0.56)
6.171 ( ± 0.39)
4.590 ( ± 0.60)
6.052 ( ± 0.24)
9.70 ( ± 0.48)
10.00
Ours
0.560 ( ± 0.03)
8.761 ( ± 0.19)
9.045 ( ± 0.28)
7.938 ( ± 0.22)
8.179 ( ± 0.33)
8.481 ( ± 0.06)
17.70 ( ± 13.61)
90.00
Appendix
TABLE VIII : Per-Scene Quantitative Analysis. We report the performance breakdown across 10 distinct scene themes. Our method demonstrates robust generalization, outperforming the baseline in nearly all metrics across diverse environments.
LLM Judge
Coverage
Category
N
Alignment
Clarity
Feasibility
Match
Aligned%
Partial%
Object
Predicate
color
116
0.81
0.94
0.80
0.90
57
40
0.88
0.29
conjunction
116
0.97
0.98
1.00
0.98
91
9
0.88
0.29
counting
116
0.87
0.97
0.90
0.92
60
38
0.88
0.29
recognition
116
0.96
0.97
0.96
0.97
85
15
0.88
0.29
semantics
116
0.89
0.95
0.94
0.94
72
27
0.88
0.29
Appendix
TABLE IX : LLM-judged quality metrics for 812 automatically generated manipulation tasks across 59 scenes and 7 competency axes.
Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.
Generalist robot manipulation policies have developed rapidly, yet their reliable evaluation remains challenging due to fundamental flaws in existing simulation benchmarks: prominent sim-to-real gaps, narrow task coverage, and unfair evaluation caused by ambiguous training-test pipelines. Prior works only partially resolve these issues and lack simultaneous faithfulness, diversity, and fairness, while static benchmark designs fail to sustain long-term policy development. We present X2Real, an evolvable simulation benchmark for faithfully evaluating the real-world performance of robotic manipulation policies based on Nvidia Isaac Lab-Arena. Following three core principles (faithfulness, diversity, and fairness), X2Real calibrates simulation visual and physical properties to align with real hardware, achieving a 0.84 linear correlation between simulated and real-robot evaluation results. It features a comprehensive taxonomy with 10 capability dimensions and 44 hierarchical long-horizon tasks, covering basic manipulation skills and advanced capacities such as visual grounding, language understanding, and bimanual control. We further adopt multi-axis domain randomization and strictly disjoint training-evaluation pipelines to mitigate benchmark exploitation and ensure credible evaluation. Powered by a custom physical domain-specific language, the Mana simulation ecosystem supports modular task design and iterative performance analysis, alongside a nearly 300-hour annotated simulation trajectory dataset. X2Real offers a faithful, diverse, and fair evolving evaluation infrastructure, effectively bridging the sim-to-real evaluation gap and supporting the advancement of generalist robotic manipulation policies.
Recent advances in robot learning have accelerated progress toward generalist robots that can perform everyday tasks in human environments. Yet it remains difficult to gauge how close we are to this vision. The field lacks a reproducible, large-scale benchmark for systematic evaluation. To fill this gap, we present RoboCasa365, a comprehensive simulation benchmark for household mobile manipulation. Built on the RoboCasa platform, RoboCasa365 introduces 365 everyday tasks across 2,500 diverse kitchen environments, with over 600 hours of human demonstration data and over 1600 hours of synthetically generated demonstration data -- making it one of the most diverse and large-scale resources for studying generalist policies. RoboCasa365 is designed to support systematic evaluations for different problem settings, including multi-task learning, robot foundation model training, and lifelong learning. We conduct extensive experiments on this benchmark with state-of-the-art methods and analyze the impacts of task diversity, dataset scale, and environment variation on generalization. Our results provide new insights into what factors most strongly affect the performance of generalist robots and inform strategies for future progress in the field.