RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
Authors: Tan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh, Gang Yang, Jiamo Hu, Jianuo Jiang, Haoquan Fang, Zhongzheng Ren, Zhongyi Luo, Ian Chong, Aaron Tung, Santiago Pravisani, Minghui Lin, Yao Mo, Ishneet Singh, Benjamin Tee, Harold Soh, Siddhartha Srinivasa, Dieter Fox, David Hsu, Haoming Song, Maxime Ellerbach, Thomas Wolf, Pepijn Kooijmans, Ruijie He, Michael Cho, Jiafei Duan
Organizations: National University of Singapore · University of Washington · Stanford University · University of North Carolina at Chapel Hill · Allen Institute for AI · BitRobot · University of Oxford · Hugging Face · Singapore Embodied AI Centre
General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete.
Figures & tables
Figure 1: RoboColosseum overview. RoboColosseum connects distributed evaluation sites through shared robot interfaces, policy-serving infrastructure, and two complementary evaluation tracks. In the Fine-tune Track , policies receive the same 100 demonstrations per task and are evaluated on predefined tasks using common success and progress criteria. In the Open Track , policies are evaluated without additional task-specific adaptation through blinded, pairwise comparisons under matched local conditions, while tasks and environments vary across comparisons. The results shown for both tracks are based on evaluations we conducted from 17 September to 28 October 2026. The project spans SO-101, Franka in the DROID configuration, Bimanual-YAM, and Unitree G1, with participation determined by track and model compatibility. The lower panels illustrate track-specific evaluation outputs and examples of physical evaluation settings. The global network depicts the broader project vision; the scope of completed experiments is reported separately.
Figure 2: Robot platforms and hardware configurations. From left to right: Bimanual-YAM, SO-101, Franka in the DROID configuration, and Unitree G1, shown from the front and rear. Annotations identify cameras, grippers, computing equipment, and supporting hardware. These platforms span bimanual manipulation, low-cost and research-grade single-arm manipulation, and humanoid manipulation with locomotion. Observation and action specifications are standardized within each embodiment to support compatible policies across evaluation sites.
Embodiment
Action space
Cameras
Dataset
SO-101
6-D joint position
1 external + 1 wrist
MolmoAct2-SO101 ( Fang et al., 2026 )
DROID (Franka)
8-D joint position
2 external + 1 wrist
DROID dataset ( Khazatsky et al., 2024 )
Bimanual-YAM
14-D joint position
1 top + 2 wrist
MolmoAct2-YAM ( Fang et al., 2026 )
Table 1: Embodiments in RoboColosseum . Each embodiment has a fixed observation and action specification shared by all policies and sites. Open Track checkpoints are trained on the listed embodiment dataset; Fine-tune Track checkpoints use only the task demonstrations ( Section 3.2.1 ). Action dimensions include one gripper dimension per arm: 5 arm joints for SO-101, 7 for the Franka, and 6 per arm for Bimanual-YAM.
Figure 3: Open Track DCR ratings across embodiments. Points show posterior mean DCR ratings on an Elo-style scale, with 95% credible intervals; n is the number of evaluations involving each system.
Figure 4: Task progress of frontier AI models versus VLA policies. Comparison of normalized task progress across 901 Open Track evaluations, each involving a frontier AI model and a VLA policy on the same task. (a) Percentage of comparisons in which the agent makes more progress, the VLA makes more progress, or both make equal progress. Numbers inside the bars show comparison counts. (b) Mean difference in task progress (agent minus VLA, in percentage points), with 95% bootstrap confidence intervals. Positive values favor the agent.
Figure 5: Open Track performance by manipulation category. (a) Relative performance: category-specific, progress-based BTD expected comparison scores, i.e., the probability of winning plus half the probability of tying against a system of average ability, where a comparison is won by the system with higher normalized progress and 50% indicates average relative performance; bold values have 95% intervals that exclude 50%. (b) Mean normalized task progress and (c) success rate of each system’s own executions, both in percent. Values are followed by ± half the width of the 95% interval (bootstrap for (a) and (b), Wilson for (c)). n , shown in (a), is the number of evaluations involving each system and is the same in all three panels. Hatched cells have insufficient support ( n<10 or no evaluations). Relative scores in (a) compare systems under matched conditions, whereas the absolute values in (b) and (c) also reflect differences in the embodiments and tasks on which each system was evaluated.
Figure 6: Physical task performance across human-rated task difficulty. Open Track policy executions grouped by the human-rated difficulty of the task, with each pairwise evaluation contributing one execution per compared system. (a) Success rate, with 95% Wilson confidence intervals. (b) Mean normalized progress among unsuccessful executions, with 95% bootstrap confidence intervals. n is the number of executions in each difficulty range.
Task
GR00T N1.7
π0.5
MolmoAct2
G0.5
LingBot-VLA v2
PS
SR
PS
SR
PS
SR
PS
SR
PS
SR
SO-101
stack_bowls
97.0 ± 3.0
96.0 ± 3.9
94.0 ± 4.2
92.0 ± 5.4
88.0 ± 5.6
84.0 ± 7.3
51.0 ± 8.0
36.0 ± 9.6
62.0 ± 7.4
40.0 ± 9.8
remove_screwdriver
90.0 ± 4.1
80.0 ± 8.0
93.0 ± 3.4
84.0 ± 7.3
69.0 ± 5.5
40.0 ± 9.8
72.0 ± 6.7
52.0 ± 10.0
67.0 ± 7.2
52.0 ± 10.0
upright_bottle
98.0 ± 2.0
96.0 ± 3.9
86.0 ± 6.1
80.0 ± 8.0
84.0 ± 6.8
80.0 ± 8.0
66.0 ± 6.6
36.0 ± 9.6
84.0 ± 5.4
64.0 ± 9.6
stack_cubes
81.0 ± 6.2
68.0 ± 9.3
90.0 ± 5.6
84.0 ± 7.3
69.0 ± 7.0
52.0 ± 10.0
42.0 ± 5.7
16.0 ± 7.3
20.0 ± 2.0
0.0 ± 0.0
Table 2: Per-task results on the Fine-tune track. Cells report normalized progress score (PS, %) and full-success rate (SR, %) as mean ± standard error (SE), using 25 trial slots per policy and task. Unexecuted trials following a safety stop are scored zero. Embodiment averages weight the five tasks equally. Best and second-best means for each metric are marked in bold and underline ; ties share ranks at the displayed precision. † Evaluation stopped for safety reasons: LingBot-VLA v2 on YAM drawer after 1 trial, and GR00T N1.7 on YAM microwave after 9 trials.
Figure 7: Task progress reachability. Percentage of trials satisfying at least a given fraction of task criteria, averaged over the five tasks of each embodiment with equal weight. The 100% point is the full success rate. † LingBot-VLA v2 on FT-YAM-02 was stopped after n=1 trial, and GR00T N1.7 on FT-YAM-03 was stopped after n=9 trials, for safety.
Figure 8: Policy-specific failure stages after fine-tuning. We show two tasks whose progress criteria form an ordered sequence: SO-101 cube_drawer and DROID/Franka close_laptop . Each bar divides a policy’s 25 evaluation trials into full-task successes (green) and failures, shaded by the first criterion the trial did not meet (darker red indicates an earlier failure). Numbers give trial counts; segments of a single trial are unlabeled. Policies trained on the same demonstrations exhibit markedly different failure locations, revealing task- and policy-specific execution bottlenecks that are hidden by success rate alone.
Figure 9: Cross-track ranking consistency by embodiment. Open Track and Fine-tune Track rankings of the five VLA policies on (a) SO-101, (b) DROID/Franka, and (c) Bimanual-YAM. For each embodiment, the Open Track rank is computed from the DCR rating, while the Fine-tuning rank is computed from the mean progress score (PS) across its five tasks. Rank 1 denotes the best-performing model, and the dashed diagonal indicates identical rankings across the two tracks. Each panel reports Spearman’s rank correlation and the number of model pairs with the same relative ordering in both tracks.
Figure 10: Ranking consistency. a. Cross-embodiment ranking consistency based on Fine-tune Track progress score (PS) across SO-101, DROID, Bimanual-YAM, and Unitree G1. Rank 1 denotes the best-performing model on an embodiment. The substantial changes in model ordering across embodiments, together with Kendall’s coefficient of concordance W=0.24 , indicate weak agreement between embodiment-specific rankings. b. Consistency of DCR rankings across independent evaluation-site subsets. For each embodiment, evaluation sites are repeatedly divided into two disjoint subsets, and DCR is fit independently on each half. Points show the mean fraction of model pairs whose relative ordering agrees between the two rankings, and error bars show the 2.5th–97.5th percentile across valid random splits. The annotated ρ is the mean Spearman rank correlation between the two rankings. Higher values indicate that the leaderboard is less sensitive to which subset of evaluation sites contributes the data.
Figure 11: Ranking efficiency and stability under limited evaluation budgets. a. Active matchmaking improves ranking efficiency. Ranking agreement with the full-data DCR leaderboard is shown as a function of the number of physical A/B evaluations. Active matchmaking prioritizes uncertain model pairs, while uniform sampling selects pairs without uncertainty-based allocation. Lines show the mean agreement over 200 independent runs, and shaded regions span the 2.5th–97.5th percentiles. b. Effect of evaluation scale on Open Track ranking stability. At each evaluation budget, 200 random subsets containing all eight systems are sampled, DCR is fit using only those evaluations, and the resulting ranking is compared with the DCR ranking obtained from the full 5,699-evaluation dataset. The line shows the mean agreement across subsets, the shaded region spans the 2.5th–97.5th percentiles, and dashed lines mark 90% and 95% mean ranking agreement.
Setting
Value
Initialization
Base pretrained checkpoint (not the Open Track checkpoint)
Training data
100 task demonstrations only
Training recipe
Closest example configuration in the official repository
Learning-rate scaling
Linear in effective batch size, relative to the recipe’s batch size
Evaluated checkpoint
Final epoch
State and action space
Absolute joint positions
Table 3: Fine-tuning settings shared by all models.
Table 4: Per-model fine-tuning hyperparameters. Peak learning rates are after linear scaling from the recipe’s batch size to the effective batch size. “Upstream default” denotes the value in the cited recipe, left unchanged.
Figure 12: Per-task progress reachability on SO-101. Percentage of the 25 trials per model satisfying at least a given fraction of the task’s K progress criteria.
Figure 13: Per-task progress reachability on DROID. Percentage of the 25 trials per model satisfying at least a given fraction of the task’s K progress criteria.
Figure 14: Per-task progress reachability on Bimanual-YAM. Percentage of completed trials satisfying at least a given fraction of the task’s K progress criteria. Hollow markers denote evaluations stopped early for safety, with the number of completed trials n given in the panel title; all other curves use 25 trials.
Figure 15: Per-task progress reachability on Unitree G1. Percentage of the 25 trials per model satisfying at least a given fraction of the task’s K progress criteria.
Evaluation of robotic manipulation systems has largely relied on fixed benchmarks authored by a small number of experts, where task instances, constraints, and success criteria are predefined and difficult to extend. This paradigm limits who can shape evaluation and obscures how policies respond to user-authored variations in task intent, constraints, and notions of success. We argue that evaluating modern manipulation policies requires reframing evaluation as a language-driven process over structured physical domains. We present RoboPlayground, a framework that enables users to author executable manipulation tasks using natural language within a structured physical domain. Natural language instructions are compiled into reproducible task specifications with explicit asset definitions, initialization distributions, and success predicates. Each instruction defines a structured family of related tasks, enabling controlled semantic and behavioral variation while preserving executability and comparability. We instantiate RoboPlayground in a structured block manipulation domain and evaluate it along three axes. A user study shows that the language-driven interface is easier to use and imposes lower cognitive workload than programming-based and code-assist baselines. Evaluating learned policies on language-defined task families reveals generalization failures that are not apparent under fixed benchmark evaluations. Finally, we show that task diversity scales with contributor diversity rather than task count alone, enabling evaluation spaces to grow continuously through crowd-authored contributions. Project Page: https://roboplayground.github.io
Yi Ru Wang, Carter Ung, Evan Gubarev +3
University of Washington, Seattle, WA, USA · Allen Institute for Artificial Intelligence, Seattle, WA, USA
Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, so current practice is to use narrow test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy's performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We conduct 2331 real-world evaluations across 3 tasks with 3 factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing.
Vision-Language-Action (VLA) models demonstrate promising generalization in robotic manipulation, driven by advances in large-scale vision and language pre-training. This progress can be misleading. Despite the zero-shot perception and language capabilities of VLAs, their overall task performance often degrades under distribution shifts, revealing gaps in how these systems translate high-level understanding into robust behavior. To systematically study this gap, we introduce Colosseum V2, a large-scale simulation benchmark for evaluating VLA generalization in robot learning across diverse conditions. The benchmark comprises 28 tasks spanning 13 task categories and two robot morphologies, covering a wide range of manipulation primitives and long-horizon behaviors. Built on the ManiSkill simulator, Colosseum V2 enables fast, GPU-parallelized evaluation and supports both in-domain and out-of-domain testing at scale. We evaluate state-of-the-art methods, including Action Chunking Transformers (ACT) and Pi0.5, and reveal limitations in both base performance and generalization. We demonstrate strong correlations between simulation and real-world metrics that support the ecological validity of the benchmark. By standardizing tasks, metrics, and evaluation protocols within a unified benchmark, Colosseum V2 enables reproducible and fair comparisons, reduced evaluation overhead, and accelerated progress toward general-purpose robot policies.
Jeremy Morgan, Hyeonho Oh, Prajwal Vijay +7
Department of Computer Science, University of Southern California, Los Angeles, CA, USA · Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India · Fu Foundation School of Engineering and Applied Science, Columbia University, New York, NY, USA