cs.ROSep 23, 2026
SaveX2Real: an eXtensive simulation benchmark for real-world generalist policies
Organizations: X Square Robot
Abstract
Generalist robot manipulation policies have developed rapidly, yet their reliable evaluation remains challenging due to fundamental flaws in existing simulation benchmarks: prominent sim-to-real gaps, narrow task coverage, and unfair evaluation caused by ambiguous training-test pipelines. Prior works only partially resolve these issues and lack simultaneous faithfulness, diversity, and fairness, while static benchmark designs fail to sustain long-term policy development. We present X2Real, an evolvable simulation benchmark for faithfully evaluating the real-world performance of robotic manipulation policies based on Nvidia Isaac Lab-Arena. Following three core principles (faithfulness, diversity, and fairness), X2Real calibrates simulation visual and physical properties to align with real hardware, achieving a 0.84 linear correlation between simulated and real-robot evaluation results. It features a comprehensive taxonomy with 10 capability dimensions and 44 hierarchical long-horizon tasks, covering basic manipulation skills and advanced capacities such as visual grounding, language understanding, and bimanual control. We further adopt multi-axis domain randomization and strictly disjoint training-evaluation pipelines to mitigate benchmark exploitation and ensure credible evaluation. Powered by a custom physical domain-specific language, the Mana simulation ecosystem supports modular task design and iterative performance analysis, alongside a nearly 300-hour annotated simulation trajectory dataset. X2Real offers a faithful, diverse, and fair evolving evaluation infrastructure, effectively bridging the sim-to-real evaluation gap and supporting the advancement of generalist robotic manipulation policies.
Figures & tables
Figure 1: X2Real Task Taxonomy. X2Real features a comprehensive task taxonomy spanning 10 core capability dimensions and 44 hierarchical long-horizon tasks. It leverages full multi - axis domain randomization with strictly disjoint training - evaluation pipelines to remove benchmark biases and enhance real - to - sim generalization fidelity.
Figure 2: Mana System Overview. Physical DSL enables agents to drive task design and rollout analysis in closed loops, from semantic scene, subtask, and robot composition to environment compilation, policy rollout, and trajectory evaluation.
Figure 3: Domain Randomization Example. The lighting condition, table material, task assets can be randomized for the Classify by Color task.
Figure 4: X2Real Embodiments. X2Real includes mobile, single-arm, dual-arm robots with replaceable grippers.
Figure 5: X2Real Dataset. we collect nearly 300 hours high-quality demonstration data on different embodiments. The image area of each task is proportional to its data length.
Figure 6: Frame-level Labeling. The Fruit in Microwave task contains 5 subtasks. Our system can automatically label corresponding frame duration during data collection.
Figure 7: Policy Space Overview . Policy Space coordinates observation mapping, action adaptation, and the episode lifecycle. Within each policy service, a model adapter converts mapped observations into model-specific inputs for policy inference. Privileged simulator state is accessible only to the evaluator.
Figure 8: Task-oriented tabletop asset spectrum. Representative packages span 44 rigid categories and nine articulated interaction groups. Rigid examples are arranged in four visual groups for readability; the collection taxonomy contains ten semantic families.
Figure 9: Dependency-ordered simulation-ready asset preparation. Seven stages transform sourced or generated content into a benchmark release package; failed candidates return for targeted repair and dependent-record revalidation.
Figure 10: Coverage-directed AIG3D asset synthesis. Text or image evidence conditions detailed reconstruction, simulation-oriented retopology, and PBR regeneration on the final mesh; four examples show the resulting long-tail geometry and appearance.
Figure 11: Semantic scene program and compilation. A semantic scene program can be edited visually and compiled against selected geometry into a concrete scene.
Figure 12: Runtime variation compilation and sampling. The compiler consolidates feasible alternatives into a reset-time plan; examples show coupled layouts and environment-lighting variation.
Figure 13: State-aligned replay augmentation. Recorded motion defines a swept envelope for static-context recomposition while preserving the state-time stream; panels (b) and (c) show source and recomposed views from the same recorded instant.
Figure 14: Executable Task DAG. (a) The Task DAG represents only the necessary partial-order dependencies and does not impose an order on independent nodes. (b) At runtime, a node becomes active only after all of its predecessors have completed, and its completion state is latched when its success predicate is first satisfied. (c) Node-completion events record the node ID and first completion frame, supporting task-success determination, progress measurement, and failure attribution. The success and failure records illustrate two alternative episode outcomes rather than simultaneous results from the same execution.
Figure 15: DAG-guided Teleoperation Labeling. The first completion frames of the Task DAG nodes divide a Classify by Shape demonstration into trajectory segments with atomic-task labels.
Figure 16: DAG-guided Data Synthesis. The system selects an active node, instantiates its skill sequence to generate sparse targets, and plans and executes the resulting trajectory. The node success predicate independently verifies the outcome. On success, the system latches the node state, saves the trajectory segment and scene checkpoint, and updates the Task DAG; on failure, it restores the latest valid checkpoint and resamples the target or motion path. Once all sink nodes have completed, the stage-wise verified segments form a complete demonstration.
Figure 17: Physics-render scheduling in Mana. (a) sync/sync serializes physics and rendering. (b) sync/async overlaps with main-thread work. (c) async/async overlaps with physics advancing to ; camera observations use , while state observations and evaluation use .
Figure 18: Sim2real Trajectory Alignment. Representative real and simulated end-effector trajectories on a held-out test trajectory.
Figure 19: Sim2real Tasks. We create digital twins (bottom row) of 8 tasks from ManipArena ( 8 ) (top row).
Figure 20: Sim2real Correlation. Wall-oss-0.5 trained on zero simulation data achieves high sim2real linear correlation in success rate and progress score.
Figure 21: Sim2real Trajectory Analysis. The policy trained on real data can overcome sim2real gap through multiple attempts in general manipulation tasks, but deteriorate in precision tasks due to the high failure penalty.
| Task | wall-oss-preview | DreamZero | Wall-OSS-0.5 | |||||
| SR (%) | Score | SR (%) | Score | SR (%) | Score | SR (%) | Score | |
| Reasoning | ||||||||
| Generalization | ||||||||
| atom_objects_beside_block | 92.50/75.00 | 94.50/84.75 | 50.00/7.32 | 70.25/53.30 | 5.00/2.50 | 35.80/47.00 | 15.00/7.50 | 50.00/44.00 |
| atom_bottle_on_plate | 70.00/55.00 | 84.50/78.50 | 22.50/15.00 | 58.50/52.00 | 25.00/5.00 | 47.80/67.50 | 0.00/0.00 | 41.25/32.50 |
| atom_fruit_in_basket | 97.50/87.50 | 97.50/96.50 | 65.00/67.50 | 67.00/86.50 | 65.00/17.50 | 56.00/80.50 | 17.50/20.00 | 21.50/34.25 |
Table 1: X2Real evaluation results. Each cell reports ID/OOD.
Figure 22: Coding Agent as Policy. Task-level OOD progress-score differences (top) and success-rate differences (bottom), computed as GPT-6-Astra minus wall-oss-preview. Tasks are ordered by decreasing score difference, with the same order in both panels. Blue denotes Reasoning tasks and pink denotes Manipulation tasks. Score differences use the 100-point scale; success-rate differences are in percentage points.
| Task | wall-oss-preview | |||
| SR (%) | Score | SR (%) | Score | |
| move_bottle_to_bin | 35/7.5 | 67/45 | 37.5/0 | 70.5/25 |
| move_fruit_to_basket | 85/2.5 | 93.5/45 | 15/0 | 38/30.5 |
| put_bottle_on_woodshelf | 55/5 | 72/42.5 | 35/2.5 | 50/19 |
| average | 58.33/5.00 | 77.50/44.17 | 29.17/0.83 | 52.83/24.83 |
Table 2: Mobile Operation Evaluation. Each cell reports ID/OOD.
Figure 23: DAG-based Mobile Operation Analysis. ID and OOD stage progression for the three mobile tasks. Pink (upper) and blue (lower) ribbons represent wall-oss-preview and , respectively. Each flow starts with 40 episodes, and all ribbon widths use a shared linear count scale. Colored flows indicate completion of successive DAG stages; gray branches count episodes whose first incomplete stage at termination is the corresponding column. Numbers denote episode counts; zero-count branches are omitted.
| Data source | Instruction | Checkpoint | Mean layers | Max layer |
| Automatic synthesis | Full-task | 55k | 3.75 | 6 |
| Teleoperation | Full-task | 50k | 1.90 | 4 |
| Automatic synthesis | Stagewise | 40k | 4.40 | 10 |
| Teleoperation | Stagewise | 50k | 2.05 | 4 |
Table 3: Closed-loop performance of automatic synthesis and teleoperation under full-task and stagewise instruction settings. Each row reports one representative checkpoint evaluated over 20 episodes.
Figure 24: Layer-wise completion rates across 20 closed-loop episodes. Blue and pink denote automatic synthesis and teleoperation, respectively. Solid lines denote stagewise instructions, whereas dashed lines denote a single full-task instruction. Each curve extends to the first layer with a zero completion rate.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Collection | Count | Collection role | Coverage and major sources |
| Indexed rigid catalog | 96,480 | Source records indexed for candidate retrieval | Tabletop objects, furniture, fixtures, beds, and major appliances. |
| Tabletop rigid release | 12,485 | Packages prepared and accepted for tabletop task roles | 44 leaf categories and 10 semantic families. Major sources: Objaverse/CanoVerse (7,946), Omni6DPose (2,090), and BlenderKit (786). |
| Articulated release | 670 | Packages prepared and accepted for articulated task roles | 9 interaction groups. HSSD-derived (607), task-specific additions (61), and ReplicaCAD (2). |
Table 4: Asset collection scope and provenance. Counts use package-level units, so geometrically similar content from different sources may remain distinct packages. AIG3D additions are included in the tabletop rigid row, while deformable assets form a separate collection outside the rigid and articulated totals.
More in this topic
Robotic Policy Evaluation
Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.
See all 114 papers