ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Authors: Yiting Chen, Kenneth Kimble, Edward H. Adelson, Tamim Asfour, Podshara Chanrungmaneekul, Sachin Chitta, Yash Chitambar, Ziyang Chen, Ken Goldberg, Danica Kragic, Hui Li, Xiang Li, Yunzhu Li, Aaron Prather, Nancy Pollard, Maximo A. Roa-Garzon, Robert Seney, Shuo Sha, Shihefeng Wang, Yu Xiang, Kaifeng Zhang, Yuke Zhu, Kaiyu Hang
Organizations: Rice University · U.S. National Institute of Standards and Technology · Massachusetts Institute of Technology · Karlsruhe Institute of Technology · Autodesk Research · University of California, Berkeley · KTH Royal Institute of Technology · Tsinghua University · Columbia University · ASTM International · Carnegie Mellon University · German Aerospace Center (DLR) · University of Texas at Dallas · University of Texas at Austin · NVIDIA Research
Dexterous manipulation enables robots to purposefully alter the physical world, transforming them from passive observers into active agents in unstructured environments. This capability is the cornerstone of physical artificial intelligence. Despite decades of advances in hardware, perception, control, and learning, progress toward general manipulation systems remains fragmented due to the absence of widely adopted standard benchmarks. The central challenge lies in reconciling the variability of the real world with the reproducibility and authenticity required for rigorous scientific evaluation. To address this, we introduce ManipulationNet, a global infrastructure that hosts real-world benchmark tasks for robotic manipulation. ManipulationNet delivers reproducible task setups through standardized hardware kits, and enables distributed performance evaluation via a unified software client that delivers real-time task instructions and collects benchmarking results. As a persistent and scalable infrastructure, ManipulationNet organizes benchmark tasks into two complementary tracks: 1) the Physical Skills Track, which evaluates low-level physical interaction skills, and 2) the Embodied Reasoning Track, which tests high-level reasoning and multimodal grounding abilities. This design fosters the systematic growth of an interconnected network of real-world abilities and skills, paving the path toward general robotic manipulation. By enabling comparable manipulation research in the real world at scale, this infrastructure establishes a sustainable foundation for measuring long-term scientific progress and identifying capabilities ready for real-world deployment.
Figures & tables
Figure 1: The “impossible trinity” of the existing robotic manipulation benchmarking effort. Prior initiatives can be broadly grouped into three categories: standardized object sets with task protocols, real-world competitions, and simulation-based benchmarks. Each category, however, addresses at most two of the required aspects in theory, and none successfully balances realism, accessibility, and authenticity to function as a large-scale, real-world manipulation benchmark.
Figure 2: The overall structure of how ManipulationNet operates. Standard object sets are centrally designed and distributed to ensure reproducible task setups. Upon registration, each research group executes the selected task with their customized robotic systems and reports performance through the mnet-client at any time and from any location. All submitted results are centrally evaluated using unified metrics and validated by a human, providing trustworthy and comparable performance assessments across the globe.
Figure 3: Overview of the general protocol that applies to all hosted tasks. Once the research group receives the physical object set, they can reproducibly configure task setups locally with their customized robotic system. To formally benchmark the system performance, the only additional requirement is connecting an external camera to the mnet-client and launching it prior to the task execution. Performance data are then transmitted to the mnet-server , where they are evaluated by official judges according to unified metrics, ensuring comparability across submissions. Under this protocol, manipulation performance is collected in a decentralized manner through distributed mnet-client , while final verification is conducted centrally.
Figure 4: The design of the peg-in-hole assembly and cable management benchmarks. A. The board design comprises five distinct shapes, encompassing both symmetric and asymmetric forms. For each shape, four clearance levels were implemented, with tolerances of 0.02 mm, 0.1 mm, 1 mm, and 3 mm. The assembly board is fabricated from transparent acrylic, with a manufacturing tolerance within 20 microns, to introduce perceptual and physical challenges, while the pegs were constructed from stainless steel to ensure robustness and durability. B. Four different types of fixtures, including the Y-clip, Round Peg, U-clip, and the C-clip, are involved to flexibly define a wide range of cable routing configurations. Since cable routing does not require high precision, the board and the fixture are distributed through 3D printing files.
Figure 5: The overview of the Grasping in Clutter task. (A) An AprilTag is used to establish the world coordinate frame in the real environment, enabling the server to render the corresponding projected scene layout. (B) After the mnet-client receives this projected scene, a human operator places all objects according to the projection mask to construct the standardized grasping testbed. (C) The benchmark includes three difficulty levels: easy scenes with sparsely arranged objects, medium scenes with objects positioned in close proximity, and hard scenes featuring stacked objects.
Figure 6: The overview of the tabletop manipulation and the block arrangement benchmarks. A. The projected scene layout for the language-conditioned tabletop manipulation through mnet-client ; B. The example language instructions for the projected scene layout; C. The object set for the block arrangement task. In this task, the robot is required to plan and execute manipulation actions to arrange the blocks according to instructions received from the mnet-server . D. An example task with a language prompt. E. An example task with a visual prompt, where the default requirement is to replicate an observed real-world arrangement. F. An example of a visual-language prompt without visual occlusion. G. An example of a visual-language prompt that incorporates visual occlusion, requiring physics and spatial reasoning about support structures to maintain the layout while also satisfying the color constraints.
Figure 7: The summary of the baseline results for each benchmark. The score for each task is normalized as a percentage of its maximum possible score to provide an intuitive measure of progress. Statistics presented in this figure is preliminary results obtained during the test period of ManipulationNet. Data will be updated as we receive more submissions.
Figure 8: Detailed protocol of the server-client performance submission. A. The overview of the software structure of the mnet-client and the mnet-server . B. The overview of the detailed activities for each performance submission.