cs.RONov 2, 2026
SaveRoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
Organizations: National University of Singapore · University of Washington · Stanford University · University of North Carolina at Chapel Hill ·
Abstract
General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete.
Figures & tables
Figure 1: RoboColosseum overview. RoboColosseum connects distributed evaluation sites through shared robot interfaces, policy-serving infrastructure, and two complementary evaluation tracks. In the Fine-tune Track , policies receive the same 100 demonstrations per task and are evaluated on predefined tasks using common success and progress criteria. In the Open Track , policies are evaluated without additional task-specific adaptation through blinded, pairwise comparisons under matched local conditions, while tasks and environments vary across comparisons. The results shown for both tracks are based on evaluations we conducted from 17 September to 28 October 2026. The project spans SO-101, Franka in the DROID configuration, Bimanual-YAM, and Unitree G1, with participation determined by track and model compatibility. The lower panels illustrate track-specific evaluation outputs and examples of physical evaluation settings. The global network depicts the broader project vision; the scope of completed experiments is reported separately.
Figure 2: Robot platforms and hardware configurations. From left to right: Bimanual-YAM, SO-101, Franka in the DROID configuration, and Unitree G1, shown from the front and rear. Annotations identify cameras, grippers, computing equipment, and supporting hardware. These platforms span bimanual manipulation, low-cost and research-grade single-arm manipulation, and humanoid manipulation with locomotion. Observation and action specifications are standardized within each embodiment to support compatible policies across evaluation sites.
| Embodiment | Action space | Cameras | Dataset |
| SO-101 | 6-D joint position | 1 external + 1 wrist | MolmoAct2-SO101 ( Fang et al., 2026 ) |
| DROID (Franka) | 8-D joint position | 2 external + 1 wrist | DROID dataset ( Khazatsky et al., 2024 ) |
| Bimanual-YAM | 14-D joint position | 1 top + 2 wrist | MolmoAct2-YAM ( Fang et al., 2026 ) |
Table 1: Embodiments in RoboColosseum . Each embodiment has a fixed observation and action specification shared by all policies and sites. Open Track checkpoints are trained on the listed embodiment dataset; Fine-tune Track checkpoints use only the task demonstrations ( Section 3.2.1 ). Action dimensions include one gripper dimension per arm: 5 arm joints for SO-101, 7 for the Franka, and 6 per arm for Bimanual-YAM.
Figure 3: Open Track DCR ratings across embodiments. Points show posterior mean DCR ratings on an Elo-style scale, with 95% credible intervals; is the number of evaluations involving each system.
Figure 4: Task progress of frontier AI models versus VLA policies. Comparison of normalized task progress across 901 Open Track evaluations, each involving a frontier AI model and a VLA policy on the same task. (a) Percentage of comparisons in which the agent makes more progress, the VLA makes more progress, or both make equal progress. Numbers inside the bars show comparison counts. (b) Mean difference in task progress (agent minus VLA, in percentage points), with 95% bootstrap confidence intervals. Positive values favor the agent.
Figure 5: Open Track performance by manipulation category. (a) Relative performance: category-specific, progress-based BTD expected comparison scores, i.e., the probability of winning plus half the probability of tying against a system of average ability, where a comparison is won by the system with higher normalized progress and 50% indicates average relative performance; bold values have 95% intervals that exclude 50%. (b) Mean normalized task progress and (c) success rate of each system’s own executions, both in percent. Values are followed by half the width of the 95% interval (bootstrap for (a) and (b), Wilson for (c)). , shown in (a), is the number of evaluations involving each system and is the same in all three panels. Hatched cells have insufficient support ( or no evaluations). Relative scores in (a) compare systems under matched conditions, whereas the absolute values in (b) and (c) also reflect differences in the embodiments and tasks on which each system was evaluated.
Figure 6: Physical task performance across human-rated task difficulty. Open Track policy executions grouped by the human-rated difficulty of the task, with each pairwise evaluation contributing one execution per compared system. (a) Success rate, with 95% Wilson confidence intervals. (b) Mean normalized progress among unsuccessful executions, with 95% bootstrap confidence intervals. is the number of executions in each difficulty range.
| Task | GR00T N1.7 | MolmoAct2 | G0.5 | LingBot-VLA v2 | ||||||
| PS | SR | PS | SR | PS | SR | PS | SR | PS | SR | |
| SO-101 | ||||||||||
| stack_bowls | 97.0 3.0 | 96.0 3.9 | 94.0 4.2 | 92.0 5.4 | 88.0 5.6 | 84.0 7.3 | 51.0 8.0 | 36.0 9.6 | 62.0 7.4 | 40.0 9.8 |
| remove_screwdriver | 90.0 4.1 | 80.0 8.0 | 93.0 3.4 | 84.0 7.3 | 69.0 5.5 | 40.0 9.8 | 72.0 6.7 | 52.0 10.0 | 67.0 7.2 | 52.0 10.0 |
| upright_bottle | 98.0 2.0 | 96.0 3.9 | 86.0 6.1 | 80.0 8.0 | 84.0 6.8 | 80.0 8.0 | 66.0 6.6 | 36.0 9.6 | 84.0 5.4 | 64.0 9.6 |
| stack_cubes | 81.0 6.2 | 68.0 9.3 | 90.0 5.6 | 84.0 7.3 | 69.0 7.0 | 52.0 10.0 | 42.0 5.7 | 16.0 7.3 | 20.0 2.0 | 0.0 0.0 |
Table 2: Per-task results on the Fine-tune track. Cells report normalized progress score (PS, %) and full-success rate (SR, %) as mean standard error (SE), using 25 trial slots per policy and task. Unexecuted trials following a safety stop are scored zero. Embodiment averages weight the five tasks equally. Best and second-best means for each metric are marked in bold and underline ; ties share ranks at the displayed precision. † Evaluation stopped for safety reasons: LingBot-VLA v2 on YAM drawer after 1 trial, and GR00T N1.7 on YAM microwave after 9 trials.
Figure 7: Task progress reachability. Percentage of trials satisfying at least a given fraction of task criteria, averaged over the five tasks of each embodiment with equal weight. The 100% point is the full success rate. † LingBot-VLA v2 on FT-YAM-02 was stopped after trial, and GR00T N1.7 on FT-YAM-03 was stopped after trials, for safety.
Figure 8: Policy-specific failure stages after fine-tuning. We show two tasks whose progress criteria form an ordered sequence: SO-101 cube_drawer and DROID/Franka close_laptop . Each bar divides a policy’s 25 evaluation trials into full-task successes (green) and failures, shaded by the first criterion the trial did not meet (darker red indicates an earlier failure). Numbers give trial counts; segments of a single trial are unlabeled. Policies trained on the same demonstrations exhibit markedly different failure locations, revealing task- and policy-specific execution bottlenecks that are hidden by success rate alone.
Figure 9: Cross-track ranking consistency by embodiment. Open Track and Fine-tune Track rankings of the five VLA policies on (a) SO-101, (b) DROID/Franka, and (c) Bimanual-YAM. For each embodiment, the Open Track rank is computed from the DCR rating, while the Fine-tuning rank is computed from the mean progress score (PS) across its five tasks. Rank 1 denotes the best-performing model, and the dashed diagonal indicates identical rankings across the two tracks. Each panel reports Spearman’s rank correlation and the number of model pairs with the same relative ordering in both tracks.
Figure 10: Ranking consistency. a. Cross-embodiment ranking consistency based on Fine-tune Track progress score (PS) across SO-101, DROID, Bimanual-YAM, and Unitree G1. Rank 1 denotes the best-performing model on an embodiment. The substantial changes in model ordering across embodiments, together with Kendall’s coefficient of concordance , indicate weak agreement between embodiment-specific rankings. b. Consistency of DCR rankings across independent evaluation-site subsets. For each embodiment, evaluation sites are repeatedly divided into two disjoint subsets, and DCR is fit independently on each half. Points show the mean fraction of model pairs whose relative ordering agrees between the two rankings, and error bars show the 2.5th–97.5th percentile across valid random splits. The annotated is the mean Spearman rank correlation between the two rankings. Higher values indicate that the leaderboard is less sensitive to which subset of evaluation sites contributes the data.
Figure 11: Ranking efficiency and stability under limited evaluation budgets. a. Active matchmaking improves ranking efficiency. Ranking agreement with the full-data DCR leaderboard is shown as a function of the number of physical A/B evaluations. Active matchmaking prioritizes uncertain model pairs, while uniform sampling selects pairs without uncertainty-based allocation. Lines show the mean agreement over 200 independent runs, and shaded regions span the 2.5th–97.5th percentiles. b. Effect of evaluation scale on Open Track ranking stability. At each evaluation budget, 200 random subsets containing all eight systems are sampled, DCR is fit using only those evaluations, and the resulting ranking is compared with the DCR ranking obtained from the full 5,699-evaluation dataset. The line shows the mean agreement across subsets, the shaded region spans the 2.5th–97.5th percentiles, and dashed lines mark 90% and 95% mean ranking agreement.
| Setting | Value |
| Initialization | Base pretrained checkpoint (not the Open Track checkpoint) |
| Training data | 100 task demonstrations only |
| Training recipe | Closest example configuration in the official repository |
| Learning-rate scaling | Linear in effective batch size, relative to the recipe’s batch size |
| Evaluated checkpoint | Final epoch |
| State and action space | Absolute joint positions |
Table 3: Fine-tuning settings shared by all models.
Table 4: Per-model fine-tuning hyperparameters. Peak learning rates are after linear scaling from the recipe’s batch size to the effective batch size. “Upstream default” denotes the value in the cited recipe, left unchanged.
Figure 12: Per-task progress reachability on SO-101. Percentage of the 25 trials per model satisfying at least a given fraction of the task’s progress criteria.
Figure 13: Per-task progress reachability on DROID. Percentage of the 25 trials per model satisfying at least a given fraction of the task’s progress criteria.
Figure 14: Per-task progress reachability on Bimanual-YAM. Percentage of completed trials satisfying at least a given fraction of the task’s progress criteria. Hollow markers denote evaluations stopped early for safety, with the number of completed trials given in the panel title; all other curves use 25 trials.
Figure 15: Per-task progress reachability on Unitree G1. Percentage of the 25 trials per model satisfying at least a given fraction of the task’s progress criteria.
| Claude Opus 5.5 | GPT-6 Astra | Grok 4.7 | |
| API name | claude-opus-5-5 | gpt-6-astra | grok-4.7 |
| Provider API | Anthropic Messages | OpenAI Responses | xAI Chat Completions |
| Access dates | Sep–Nov 2026 | Sep–Nov 2026 | Sep–Nov 2026 |
Table 5: Agents evaluated in the Open Track .
More in this topic
Robotic Policy Evaluation
Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.
See all 139 papers