ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Authors: Yiting Chen, Kenneth Kimble, Edward H. Adelson, Tamim Asfour, Podshara Chanrungmaneekul, Sachin Chitta, Yash Chitambar, Ziyang Chen, Ken Goldberg, Danica Kragic, Hui Li, Xiang Li, Yunzhu Li, Aaron Prather, Nancy Pollard, Maximo A. Roa-Garzon, Robert Seney, Shuo Sha, Shihefeng Wang, Yu Xiang, Kaifeng Zhang, Yuke Zhu, Kaiyu Hang
Organizations: Rice University · U.S. National Institute of Standards and Technology · Massachusetts Institute of Technology · Karlsruhe Institute of Technology · Autodesk Research · University of California, Berkeley · KTH Royal Institute of Technology · Tsinghua University · Columbia University · ASTM International · Carnegie Mellon University · German Aerospace Center (DLR) · University of Texas at Dallas · University of Texas at Austin · NVIDIA Research
Dexterous manipulation enables robots to purposefully alter the physical world, transforming them from passive observers into active agents in unstructured environments. This capability is the cornerstone of physical artificial intelligence. Despite decades of advances in hardware, perception, control, and learning, progress toward general manipulation systems remains fragmented due to the absence of widely adopted standard benchmarks. The central challenge lies in reconciling the variability of the real world with the reproducibility and authenticity required for rigorous scientific evaluation. To address this, we introduce ManipulationNet, a global infrastructure that hosts real-world benchmark tasks for robotic manipulation. ManipulationNet delivers reproducible task setups through standardized hardware kits, and enables distributed performance evaluation via a unified software client that delivers real-time task instructions and collects benchmarking results. As a persistent and scalable infrastructure, ManipulationNet organizes benchmark tasks into two complementary tracks: 1) the Physical Skills Track, which evaluates low-level physical interaction skills, and 2) the Embodied Reasoning Track, which tests high-level reasoning and multimodal grounding abilities. This design fosters the systematic growth of an interconnected network of real-world abilities and skills, paving the path toward general robotic manipulation. By enabling comparable manipulation research in the real world at scale, this infrastructure establishes a sustainable foundation for measuring long-term scientific progress and identifying capabilities ready for real-world deployment.
Figures & tables
Figure 1: The “impossible trinity” of the existing robotic manipulation benchmarking effort. Prior initiatives can be broadly grouped into three categories: standardized object sets with task protocols, real-world competitions, and simulation-based benchmarks. Each category, however, addresses at most two of the required aspects in theory, and none successfully balances realism, accessibility, and authenticity to function as a large-scale, real-world manipulation benchmark.
Figure 2: The overall structure of how ManipulationNet operates. Standard object sets are centrally designed and distributed to ensure reproducible task setups. Upon registration, each research group executes the selected task with their customized robotic systems and reports performance through the mnet-client at any time and from any location. All submitted results are centrally evaluated using unified metrics and validated by a human, providing trustworthy and comparable performance assessments across the globe.
Figure 3: Overview of the general protocol that applies to all hosted tasks. Once the research group receives the physical object set, they can reproducibly configure task setups locally with their customized robotic system. To formally benchmark the system performance, the only additional requirement is connecting an external camera to the mnet-client and launching it prior to the task execution. Performance data are then transmitted to the mnet-server , where they are evaluated by official judges according to unified metrics, ensuring comparability across submissions. Under this protocol, manipulation performance is collected in a decentralized manner through distributed mnet-client , while final verification is conducted centrally.
Figure 4: The design of the peg-in-hole assembly and cable management benchmarks. A. The board design comprises five distinct shapes, encompassing both symmetric and asymmetric forms. For each shape, four clearance levels were implemented, with tolerances of 0.02 mm, 0.1 mm, 1 mm, and 3 mm. The assembly board is fabricated from transparent acrylic, with a manufacturing tolerance within 20 microns, to introduce perceptual and physical challenges, while the pegs were constructed from stainless steel to ensure robustness and durability. B. Four different types of fixtures, including the Y-clip, Round Peg, U-clip, and the C-clip, are involved to flexibly define a wide range of cable routing configurations. Since cable routing does not require high precision, the board and the fixture are distributed through 3D printing files.
Figure 5: The overview of the Grasping in Clutter task. (A) An AprilTag is used to establish the world coordinate frame in the real environment, enabling the server to render the corresponding projected scene layout. (B) After the mnet-client receives this projected scene, a human operator places all objects according to the projection mask to construct the standardized grasping testbed. (C) The benchmark includes three difficulty levels: easy scenes with sparsely arranged objects, medium scenes with objects positioned in close proximity, and hard scenes featuring stacked objects.
Figure 6: The overview of the tabletop manipulation and the block arrangement benchmarks. A. The projected scene layout for the language-conditioned tabletop manipulation through mnet-client ; B. The example language instructions for the projected scene layout; C. The object set for the block arrangement task. In this task, the robot is required to plan and execute manipulation actions to arrange the blocks according to instructions received from the mnet-server . D. An example task with a language prompt. E. An example task with a visual prompt, where the default requirement is to replicate an observed real-world arrangement. F. An example of a visual-language prompt without visual occlusion. G. An example of a visual-language prompt that incorporates visual occlusion, requiring physics and spatial reasoning about support structures to maintain the layout while also satisfying the color constraints.
Figure 7: The summary of the baseline results for each benchmark. The score for each task is normalized as a percentage of its maximum possible score to provide an intuitive measure of progress. Statistics presented in this figure is preliminary results obtained during the test period of ManipulationNet. Data will be updated as we receive more submissions.
Figure 8: Detailed protocol of the server-client performance submission. A. The overview of the software structure of the mnet-client and the mnet-server . B. The overview of the detailed activities for each performance submission.
Achieving generalizable robotic manipulation remains a central challenge in embodied intelligence. Despite rapid advances in model architectures and learning algorithms, progress is often limited by the scarcity and narrow diversity of real-world data. The RoboSynChallenge competition introduces a unified benchmark to evaluate and advance the generalizability of manipulation policies across a spectrum of tasks, environments, and difficulty levels. To alleviate the shortage of realistic data, the challenge integrates large-scale synthetic data generation with standardized real-world robotic evaluation. Participants are encouraged to leverage synthesized state-action trials to improve general-purpose policy learning, while final assessments are conducted exclusively on unseen real-world manipulation environments. Baseline implementations, including Transformer-, Diffusion-, Vision-Language-Action, and World-Action-Model-based policies, are provided to ensure reproducibility and comparability. By coupling scalable simulation-based training with rigorous real-world validation, RoboSynChallenge aims to foster the development of broadly capable, data-efficient, and adaptable manipulation systems, thereby paving the way toward truly general robotic intelligence.
Runyi Zhao, Ruixin Wu, Chengkun Li +15
Shenzhen Loop Area Institute (SLAI) · The Chinese University of Hong Kong, Shenzhen · DexForce +8
Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.
Vision-Language-Action (VLA) models and world-action models have emerged as central paradigms for general-purpose robotic intelligence, yet their empirical progress remains constrained by the absence of evaluation protocols that are both physically realistic and diagnostically controlled. Simulator-centric benchmarks provide scale and reproducibility, but cannot fully capture the reality gap induced by perception noise, contact dynamics, latency, calibration error, and hardware constraints. Conversely, real-robot evaluations are often fragmented across platforms, scenes, objects, and scoring rules, making fair comparison and failure attribution difficult. We introduce ManipArena, a standardized real-robot evaluation framework for studying manipulation generalization under matched physical conditions. ManipArena comprises 20 tasks, 10,812 expert trajectories, 13.5M frames, and approximately 188 robot hours across tabletop and mobile manipulation. The framework combines schema-defined task variation, stratified in-domain, visualshift, and semantic-OOD trials, subtask-level partial-credit scoring, three-level language annotations, low-level motor signals, and paired real-to-sim environments reconstructed from physical scenes. Using ManipArena, we evaluate seven tabletop configurations spanning VLA and world-action-model policies. The results show that real-robot conclusions depend not only on architecture, but also on model provenance, fine-tuning regime, data sampling, and annotation granularity. ManipArena thus provides a reproducible and interpretable foundation for diagnosing capability boundaries and failure modes in embodied generalization.
Yu Sun, Meng Cao, Yang Ping +24
Sun Yat-sen University · X Square Robot · MBZUAI +2