RoboTeam: Benchmarking Vision-Based Multi-Agent Control of Many Robot Arms in Real Time
Organizations: The Hong Kong University of Science and Technology · Microsoft
Abstract
Multimodal agents can now control robot arms by calling tools and writing code. Existing benchmarks fall short in three ways: most test one or two arms on short tasks, many pause the world while the agent thinks, and many give it ground-truth object positions. We present RoboTeam, a simulated benchmark where agents control 4 to 16 robot arms in real time from camera images alone. Its 36 tasks are long jobs that the arms must do together, such as passing timbers or building a picture from hundreds of cubes. We start from the most flexible agent system, the adaptive system, where one agent decides when to fork delegates. With it, gpt-6.1-sol and opus-5.5 complete 28 and 27 tasks, and four other models at most 3. But at a high price: a typical task takes gpt-6.1-sol 3.5 hours and 218 million input tokens, and a third of its steps fail at least once and must be retried. We then compare it with three fixed systems: single-agent, hierarchical, and decentralized. All four complete a similar number of tasks but differ in time and tokens. The hierarchical and decentralized systems are faster but use up to 1.9 times the tokens, and the single agent uses fewer tokens but takes 1.6 times as long, so the adaptive system lies between them. Even so, no model or system is yet fast, cheap, and reliable at once. RoboTeam measures these limits where earlier benchmarks cannot: long multi-arm jobs, camera images only, and a world that never pauses.
Figures & tables
| Max | Multi- | Real- | Vision | Run length | |
| Benchmark | arms | agent | time | only | |
| RoboColiseum [ 15 ] | 2 | ✓ | 33 min | ||
| RobotWorld [ 46 ] | 2 | 26 min | |||
| EmbodiedBench [ 45 ] | 1 | 1 min | |||
| PARTNR [ 4 ] | 1 | ✓ | 36 min | ||
| RoCoBench [ 27 ] | 3 | 10 min |
| Task | At the end | End of the run | What is wrong |
| Transport, 8 arms, Level 1 | 23 of 24 items | declared done, 3.7 h | One cube is 5.3 mm off center and 4 mm outside its target square |
| Transport, 12 arms, Level 1 | 30 of 36 items | declared done, 11.9 h | Two cubes lie on a side face, and four cylinders are 5.5 to 8.1 mm off center |
| Mosaic, 6 arms, Level 2 | 160 of 162 cubes | declared done, 5.2 h | Two cubes are 10 and 11 mm off center, against a limit of 6 mm |
| Mosaic, 8 arms, Level 1 | 256 of 288 cubes | 24 h limit | 26 cells empty, 4 cubes tilted in their pockets, 1 of the wrong color, 1 still held |
| Mosaic, 8 arms, Level 2 | 241 of 288 cubes | 24 h limit | 40 cells empty, 7 cubes off center or rotated |
| Mosaic, 8 arms, Level 3 | 272 of 288 cubes | 24 h limit | 11 cells empty, 5 cubes 6.1 to 7.1 mm off center |
| Run | Task | At the end | End of the run | What is wrong |
|---|---|---|---|---|
| 2 | Transport, 12 arms | 5 of 36 items | stopped by agents, 0.9 h | A cube slipped from a gripper to the floor, out of reach. Three targets need this kind of cube, and there are only three |
| 2 | Mosaic, 8 arms | 259 of 288 cubes | 24 h limit | 177 cubes placed in the first 2 hours, then 4 to 8 per hour. 14 cells empty, 15 cubes up to 9 mm off center |
| 2 | Relay, 16 arms | 2 of 4 timbers | declared done | Two timbers lie outside the goal rectangle |
| 2 | Dial, 8 arms | 0 of 4 dials | stopped by agents, 15 h | A module fell from the table at 0.4 h. Using the other modules as hooks to reach it, the agents dropped all four and never tried a code |
| 3 | Mosaic, 6 arms | 159 of 162 cubes | declared done | Three cubes are 9.2 to 9.7 mm off center |
| 3 | Relay, 16 arms | 3 of 4 timbers | stopped by agents, 6.8 h | While correcting the position of one timber, an arm pushed it off the table |
| Tasks | Time per task | Total | Input | Output | Messages | Tool calls | Atomic failure | Failed grasps | ||
|---|---|---|---|---|---|---|---|---|---|---|
| completed | (h) | time (h) | (B) | (M) | Cached | per task | per task | rate | of all grasps | |
| Reference ( gpt-6.1-sol , Codex, single agent, ground-truth positions) | ||||||||||
| Oracle | 36 | 1.45 (0.25–6.70) | 172.2 | 3.94 | 16.5 | 98.6% | 0 (0–0) | 924 (373–1,919) | 1% | 13/3,830 (0.3%) |
| Agent systems ( gpt-6.1-sol , Codex) | ||||||||||
| Adaptive | 28 | 3.48 (1.44–7.90) | 258.0 | 27.01 | 65.7 | 97.9% | 184 (59–706) | 1,576 (808–6,612) | 34% (18–62%) | 1,205/5,244 (23.0%) |
| Hierarchical | 29 | 3.09 (1.25–7.41) | 203.9 | 51.68 | 153.2 | 97.6% | 471 (102–1,803) | 3,746 (1,428–12,996) | 37% (19–58%) | 1,047/4,468 (23.4%) |
| Ability | Note of the agents |
|---|---|
| Position to millimeters | “more than twice the 6mm placement tolerance” (Mosaic) |
| Separating objects | “Adjacent same-color cubes can merge at 128x80” (Mosaic) |
| Reading symbols | “tiny glyphs and three-label planar-pose ambiguity” (Dial) |
| Depth from views | “roughly one-pixel center change moved inferred disk depth by 110mm” (Dial) |
| Fine seating | “Low resolution does not directly resolve blade-depth tolerances” (Transport) |
| Tasks | Only | Only this | ||
| completed | Adaptive | condition | ||
| Oracle | 36 | 0 | 8 | 0.055 |
| Hierarchical | 29 (24–33) | 5 | 6 | 1 |
| Decentralized | 29 (24–33) | 4 | 5 | 1 |
| Single-agent | 27 (22–32) | 4 | 3 | 1 |
| Robot tool only | 13 (8–18) | 15 | 0 | 0.001 |
| Small | Medium | Large | |
|---|---|---|---|
| Adaptive | 1.4 h (11) | 4.4 h (10) | 9.2 h (7) |
| Hierarchical | 1.2 h (12) | 3.9 h (10) | 5.5 h (7) |
| Decentralized | 1.4 h (12) | 4.4 h (12) | 7.1 h (5) |
| Single-agent | 2.2 h (11) | 11.7 h (10) | 22.0 h (6) |
| gpt-6.1-sol | opus-5.5 | |
| Runs with entries | 36 | 31 |
| Entries per run | 11 | 6 |
| Revisions per run | 11 | 0 |
| Words per run | 4,130 | 1,395 |
| Words per hour of work | 1,097 | 515 |
| All entries | 541 | 433 |
| Operation | What it does |
|---|---|
| observe | Returns RGB images from the named cameras and the state of the arms |
| move | Moves the tool of an arm along a straight line to a position and orientation, or moves its joints to given angles |
| gripper | Opens, closes, or sets the opening of a gripper. In its grasp mode, it closes on an object of an expected width with a chosen force |
| action_batch | Sends several commands for several arms in one call |
| check_motion | Screens planned moves of several arms for overlap before they are sent |
| job | Reports the status of an earlier command |
| All | At arms | Overviews | Close-ups | |
|---|---|---|---|---|
| Transport | 16–42 | 8–24 | 4 | 4–14 |
| Mosaic | 14–24 | 8–16 | 4 | 2–4 |
| Relay | 23–49 | 16–32 | 4 | 3–13 |
| Dial | 29–69 | 8–32 | 4 | 17–33 |
Robot Manipulation Benchmarks
Robot manipulation benchmarks measure how reliably robots perceive, plan, and execute physical tasks across objects, settings, and embodiments.
See all 364 papers