X2Real: an eXtensive simulation benchmark for real-world generalist policies
Authors: Lian Ruan, Jade Yang, Sherphylan Gao, Felix Gao, Kyson Liang, Galen Liu, Ligo Wu, Lane Jin, Guu Gu, Bevan Xie, Cloud Yan, Zongzi Yuan, Kino Luo, Emma Chen, Shuwen Chen, Yang Ping, Miles Guo, Rain Sun, Kayden Zhang, Alex Du, Ruihai Wu, Liang Hao, Zhaoshuo Li, Roy Gan, Hao Wang, Qian Wang
Generalist robot manipulation policies have developed rapidly, yet their reliable evaluation remains challenging due to fundamental flaws in existing simulation benchmarks: prominent sim-to-real gaps, narrow task coverage, and unfair evaluation caused by ambiguous training-test pipelines. Prior works only partially resolve these issues and lack simultaneous faithfulness, diversity, and fairness, while static benchmark designs fail to sustain long-term policy development. We present X2Real, an evolvable simulation benchmark for faithfully evaluating the real-world performance of robotic manipulation policies based on Nvidia Isaac Lab-Arena. Following three core principles (faithfulness, diversity, and fairness), X2Real calibrates simulation visual and physical properties to align with real hardware, achieving a 0.84 linear correlation between simulated and real-robot evaluation results. It features a comprehensive taxonomy with 10 capability dimensions and 44 hierarchical long-horizon tasks, covering basic manipulation skills and advanced capacities such as visual grounding, language understanding, and bimanual control. We further adopt multi-axis domain randomization and strictly disjoint training-evaluation pipelines to mitigate benchmark exploitation and ensure credible evaluation. Powered by a custom physical domain-specific language, the Mana simulation ecosystem supports modular task design and iterative performance analysis, alongside a nearly 300-hour annotated simulation trajectory dataset. X2Real offers a faithful, diverse, and fair evolving evaluation infrastructure, effectively bridging the sim-to-real evaluation gap and supporting the advancement of generalist robotic manipulation policies.
Figures & tables
Figure 1: X2Real Task Taxonomy. X2Real features a comprehensive task taxonomy spanning 10 core capability dimensions and 44 hierarchical long-horizon tasks. It leverages full multi - axis domain randomization with strictly disjoint training - evaluation pipelines to remove benchmark biases and enhance real - to - sim generalization fidelity.
Figure 2: Mana System Overview. Physical DSL enables agents to drive task design and rollout analysis in closed loops, from semantic scene, subtask, and robot composition to environment compilation, policy rollout, and trajectory evaluation.
Figure 3: Domain Randomization Example. The lighting condition, table material, task assets can be randomized for the Classify by Color task.
Figure 4: X2Real Embodiments. X2Real includes mobile, single-arm, dual-arm robots with replaceable grippers.
Figure 5: X2Real Dataset. we collect nearly 300 hours high-quality demonstration data on different embodiments. The image area of each task is proportional to its data length.
Figure 6: Frame-level Labeling. The Fruit in Microwave task contains 5 subtasks. Our system can automatically label corresponding frame duration during data collection.
Figure 7: Policy Space Overview . Policy Space coordinates observation mapping, action adaptation, and the episode lifecycle. Within each policy service, a model adapter converts mapped observations into model-specific inputs for policy inference. Privileged simulator state is accessible only to the evaluator.
Figure 8: Task-oriented tabletop asset spectrum. Representative packages span 44 rigid categories and nine articulated interaction groups. Rigid examples are arranged in four visual groups for readability; the collection taxonomy contains ten semantic families.
Figure 9: Dependency-ordered simulation-ready asset preparation. Seven stages transform sourced or generated content into a benchmark release package; failed candidates return for targeted repair and dependent-record revalidation.
Figure 10: Coverage-directed AIG3D asset synthesis. Text or image evidence conditions detailed reconstruction, simulation-oriented retopology, and PBR regeneration on the final mesh; four examples show the resulting long-tail geometry and appearance.
Figure 11: Semantic scene program and compilation. A semantic scene program can be edited visually and compiled against selected geometry into a concrete scene.
Figure 12: Runtime variation compilation and sampling. The compiler consolidates feasible alternatives into a reset-time plan; examples show coupled layouts and environment-lighting variation.
Figure 13: State-aligned replay augmentation. Recorded motion defines a swept envelope for static-context recomposition while preserving the state-time stream; panels (b) and (c) show source and recomposed views from the same recorded instant.
Figure 14: Executable Task DAG. (a) The Task DAG represents only the necessary partial-order dependencies and does not impose an order on independent nodes. (b) At runtime, a node becomes active only after all of its predecessors have completed, and its completion state is latched when its success predicate is first satisfied. (c) Node-completion events record the node ID and first completion frame, supporting task-success determination, progress measurement, and failure attribution. The success and failure records illustrate two alternative episode outcomes rather than simultaneous results from the same execution.
Figure 15: DAG-guided Teleoperation Labeling. The first completion frames of the Task DAG nodes divide a Classify by Shape demonstration into trajectory segments with atomic-task labels.
Figure 16: DAG-guided Data Synthesis. The system selects an active node, instantiates its skill sequence to generate sparse targets, and plans and executes the resulting trajectory. The node success predicate independently verifies the outcome. On success, the system latches the node state, saves the trajectory segment and scene checkpoint, and updates the Task DAG; on failure, it restores the latest valid checkpoint and resamples the target or motion path. Once all sink nodes have completed, the stage-wise verified segments form a complete demonstration.
Figure 17: Physics-render scheduling in Mana. (a) sync/sync serializes physics and rendering. (b) sync/async overlaps R(Sn+1) with main-thread work. (c) async/async overlaps R(Sn) with physics advancing to Sn+1 ; camera observations use Sn , while state observations and evaluation use Sn+1 .
Figure 18: Sim2real Trajectory Alignment. Representative real and simulated end-effector trajectories on a held-out test trajectory.
Figure 19: Sim2real Tasks. We create digital twins (bottom row) of 8 tasks from ManipArena ( 8 ) (top row).
Figure 20: Sim2real Correlation. Wall-oss-0.5 trained on zero simulation data achieves high sim2real linear correlation in success rate and progress score.
Figure 21: Sim2real Trajectory Analysis. The policy trained on real data can overcome sim2real gap through multiple attempts in general manipulation tasks, but deteriorate in precision tasks due to the high failure penalty.
Task
wall-oss-preview
π0.5
DreamZero
Wall-OSS-0.5
SR (%)
Score
SR (%)
Score
SR (%)
Score
SR (%)
Score
Reasoning
Generalization
atom_objects_beside_block
92.50/75.00
94.50/84.75
50.00/7.32
70.25/53.30
5.00/2.50
35.80/47.00
15.00/7.50
50.00/44.00
atom_bottle_on_plate
70.00/55.00
84.50/78.50
22.50/15.00
58.50/52.00
25.00/5.00
47.80/67.50
0.00/0.00
41.25/32.50
atom_fruit_in_basket
97.50/87.50
97.50/96.50
65.00/67.50
67.00/86.50
65.00/17.50
56.00/80.50
17.50/20.00
21.50/34.25
Table 1: X2Real evaluation results. Each cell reports ID/OOD.
Figure 22: Coding Agent as Policy. Task-level OOD progress-score differences (top) and success-rate differences (bottom), computed as GPT-6-Astra minus wall-oss-preview. Tasks are ordered by decreasing score difference, with the same order in both panels. Blue denotes Reasoning tasks and pink denotes Manipulation tasks. Score differences use the 100-point scale; success-rate differences are in percentage points.
Task
wall-oss-preview
π0.5
SR (%)
Score
SR (%)
Score
move_bottle_to_bin
35/7.5
67/45
37.5/0
70.5/25
move_fruit_to_basket
85/2.5
93.5/45
15/0
38/30.5
put_bottle_on_woodshelf
55/5
72/42.5
35/2.5
50/19
average
58.33/5.00
77.50/44.17
29.17/0.83
52.83/24.83
Table 2: Mobile Operation Evaluation. Each cell reports ID/OOD.
Figure 23: DAG-based Mobile Operation Analysis. ID and OOD stage progression for the three mobile tasks. Pink (upper) and blue (lower) ribbons represent wall-oss-preview and π0.5 , respectively. Each flow starts with 40 episodes, and all ribbon widths use a shared linear count scale. Colored flows indicate completion of successive DAG stages; gray branches count episodes whose first incomplete stage at termination is the corresponding column. Numbers denote episode counts; zero-count branches are omitted.
Data source
Instruction
Checkpoint
Mean layers
Max layer
Automatic synthesis
Full-task
55k
3.75
6
Teleoperation
Full-task
50k
1.90
4
Automatic synthesis
Stagewise
40k
4.40
10
Teleoperation
Stagewise
50k
2.05
4
Table 3: Closed-loop performance of automatic synthesis and teleoperation under full-task and stagewise instruction settings. Each row reports one representative checkpoint evaluated over 20 episodes.
Figure 24: Layer-wise completion rates across 20 closed-loop episodes. Blue and pink denote automatic synthesis and teleoperation, respectively. Solid lines denote stagewise instructions, whereas dashed lines denote a single full-task instruction. Each curve extends to the first layer with a zero completion rate.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Collection
Count
Collection role
Coverage and major sources
Indexed rigid catalog
96,480
Source records indexed for candidate retrieval
Tabletop objects, furniture, fixtures, beds, and major appliances.
Tabletop rigid release
12,485
Packages prepared and accepted for tabletop task roles
44 leaf categories and 10 semantic families. Major sources: Objaverse/CanoVerse (7,946), Omni6DPose (2,090), and BlenderKit (786).
Articulated release
670
Packages prepared and accepted for articulated task roles
Table 4: Asset collection scope and provenance. Counts use package-level units, so geometrically similar content from different sources may remain distinct packages. AIG3D additions are included in the tabletop rigid row, while deformable assets form a separate collection outside the rigid and articulated totals.
Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.
Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets. Given the high cost of real-world data collection, a practical alternative is to generate synthetic data through simulation. However, the resulting synthetic data often exhibits a significant gap from real-world distributions. While many prior studies have proposed algorithms to bridge the Sim-to-Real discrepancy, there remains a lack of principled research that grounds these methods in real-world manipulation tasks, particularly their performance on generalist policies such as Vision-Language-Action (VLA) models. In this study, we empirically examine the primary determinants of Sim-to-Real generalization across four dimensions: multi-level domain randomization, photorealistic rendering, physics-realistic modeling, and reinforcement learning updates. To support this study, we design a comprehensive evaluation protocol to quantify the real-world performance of manipulation tasks. The protocol accounts for key variations in background, lighting, distractors, object types, and spatial features. Through experiments involving over 10k real-world trials, we derive critical insights into Sim-to-Real transfer. To inform and advance future studies, we release both the robotic platforms and the evaluation protocol for public access to facilitate independent verification, thereby establishing a realistic and standardized benchmark for robotic manipulation policies.
Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang +4
School of Data Science, The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a high-fidelity simulation environment. We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies. Project website: https://research.nvidia.com/labs/srl/projects/robolab/.
Jenai Xuning Yang, Rishit Dagli, Alex Zook +5
NVIDIA · University of Toronto · Work done during internship at NVIDIA +1