MESA: An Evaluation Framework for Compositional, Semantic, and Spatial Generalization in Robotics
Organizations: Georgia Institute of Technology · Georgia Tech Research Institute
Abstract
Despite growing interest in foundation models for robotic manipulation, systematic evaluation of their capabilities remains elusive. Existing benchmarks are either narrow and saturated, offering little signal on SOTA methods, broadly scoped, providing diversity at the cost of controlled experimentation, or require training on large, messy open-source datasets. We introduce MESA, an evaluation framework designed for precise, reproducible measurement of language-conditioned policy generalization. MESA provides five evaluation suites that isolate in-distribution performance and four distinct axes of generalization: spatial configuration, object instance, object category, and subtask composition, enabling researchers to diagnose where and why policies fail, not just whether they fail. To populate these suites at scale, we develop MESA-Gen, a pipeline for task and demonstration generation built by improving MimicGen. We instantiate the framework on two platforms and release both: the single-arm MESA suite, and BiMESA, the first large-scale benchmark for bimanual generalization. We evaluate seven baseline methods spanning diffusion policies, VLM-based approaches, and pretrained VLAs, finding that the strongest model achieves only 52% on MESA and 49% on BiMESA, providing meaningful signal while leaving ample room for progress. MESA rankings are broadly consistent with RoboArena scores, suggesting that MESA captures method-level differences relevant beyond simulation. Project website: https://pairlab.github.io/MESA.
Figures & tables
| Benchmark | Num. Tasks | Obj. / Insts | MT | Action Space | Train Data | OOD | Bim. |
| RLBench [ 28 ] | 100 | 28 / 28 | ✓ | Keyframe | Scripted | ✗ | ✗ |
| PerAct2 [ 29 ] | 13 | 15 / 15 | ✓ | Keyframe | Scripted | ✗ | ✓ |
| COLOSSEUM [ 30 ] | 20 | 28 / 28 | ✗ | Keyframe | Scripted | ✓ | ✗ |
| CALVIN [ 31 ] | 34 | 5 / 30 | ✓ | Low-level | Human | ✗ | ✗ |
| MetaWorld [ 32 ] | 50 | 38 / 38 | ✗ | Low-level | None | ✗ | ✗ |
| LIBERO [ 17 ] | 130 | 51 / 75 | ✓ | Low-level | Human | ✗ | ✗ |
| Setting | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol † | ||
| MESA-ID | 1.6 | – | 30.7 | 25.0 | 43.1 | 46.5 | 62.8 | – |
| MESA-Spatial | 1.0 | – | 35.0 | 23.2 | 42.2 | 53.6 | 68.6 | – |
| MESA-Instance | 1.9 | – | 22.7 | 22.7 | 34.0 | 39.1 | 55.4 | – |
| MESA-Composite | 0.3 | – | 17.7 | 15.0 | 27.5 | 28.9 | 46.7 | – |
| MESA-Category | 0.7 | – | 7.7 | 14.1 | 11.6 | 8.4 | 26.4 | – |
| MESA Avg. | 1.1 | – | 22.8 | 20.0 | 31.7 | 35.3 | 52.0 | – |
| Setting | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| MESA-ID | 18.9 | – | 80.0 | 71.4 | 79.7 | 83.7 | 87.6 | – |
| MESA-Spatial | 11.4 | – | 77.4 | 53.7 | 73.0 | 78.6 | 84.5 | – |
| MESA-Instance | 23.5 | – | 66.6 | 60.5 | 66.5 | 69.0 | 76.9 | – |
| MESA-Composite | 6.4 | – | 65.1 | 56.2 | 65.5 | 66.3 | 81.5 | – |
| MESA-Category | 12.1 | – | 32.6 | 42.5 | 33.1 | 28.0 | 44.1 | – |
| MESA Avg. | 14.4 | – | 64.3 | 56.9 | 63.6 | 65.1 | 74.9 | – |
| ID | Spatial | Instance | Composite | Category | |
| DP | 1.6 | 1.0 | 1.9 | 0.3 | 0.7 |
| DP+Aux | 2.3 | 2.4 | 2.2 | 0.6 | 0.3 |
| 43.1 | 42.2 | 34.0 | 27.5 | 11.6 | |
| +Aux | 43.4 | 54.4 | 33.2 | 28.8 | 8.7 |
| 62.8 | 68.6 | 55.4 | 46.7 | 26.4 | |
| +Aux | 59.0 | 69.2 | 53.6 | 45.6 | 23.0 |
| PG-Bin | PG-FM | -FAST | |||
| RoboArena | 771 | 1527 | 1449 | 1564 | 1594 |
| MESA | – | 22.8 | 31.7 | 35.3 | 52.0 |
| BiMESA | 18.3 | 30.4 | 29.6 | 39.4 | 48.7 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| MESA-ID | DP | PG-FM | GR00T-N1.6 | FAST | ||
|---|---|---|---|---|---|---|
| put the apple on the tray | 2.0 | 54.0 | 62.0 | 66.0 | 64.0 | 66.0 |
| put the avocado in the basket | 0.0 | 58.0 | 18.0 | 50.0 | 70.0 | 88.0 |
| put the bagel on the cutting board | 8.0 | 26.0 | 36.0 | 48.0 | 36.0 | 86.0 |
| put the bar in the bottom drawer of the cabinet and close it | 0.0 | 20.0 | 12.0 | 22.0 | 30.0 | 46.0 |
| put the bar soap in the top drawer of the cabinet and close it | 0.0 | 24.0 | 20.0 | 24.0 | 58.0 | 60.0 |
| put the beer in the slide cabinet and close it | 0.0 | 6.0 | 6.0 | 10.0 | 10.0 | 34.0 |
| MESA-Spatial | DP | PG-FM | GR00T-N1.6 | FAST | ||
|---|---|---|---|---|---|---|
| open the slide cabinet and put the tomato in it | 2.0 | 34.0 | 30.0 | 58.0 | 54.0 | 74.0 |
| open the slide cabinet and put the water bottle in it | 0.0 | 12.0 | 6.0 | 16.0 | 28.0 | 46.0 |
| put the rolling pin in the right side of the dish drainer | 0.0 | 4.0 | 6.0 | 8.0 | 32.0 | 38.0 |
| put the rolling pin in the pan | 4.0 | 12.0 | 36.0 | 26.0 | 46.0 | 62.0 |
| put the rolling pin on the tray | 4.0 | 28.0 | 22.0 | 28.0 | 54.0 | 72.0 |
| put the tomato on the cutting board | 0.0 | 54.0 | 28.0 | 58.0 | 72.0 | 78.0 |
| MESA-Instance | DP | PG-FM | GR00T-N1.6 | FAST | ||
|---|---|---|---|---|---|---|
| put the apple on the tray | 10.0 | 38.0 | 46.0 | 48.0 | 56.0 | 64.0 |
| put the beer in the slide cabinet and close it | 0.0 | 6.0 | 4.0 | 10.0 | 6.0 | 14.0 |
| put the bell pepper in the box and close it | 0.0 | 10.0 | 4.0 | 10.0 | 14.0 | 30.0 |
| put the cheese on the plate | 0.0 | 4.0 | 2.0 | 2.0 | 12.0 | 28.0 |
| put the fish in the pan | 4.0 | 32.0 | 54.0 | 60.0 | 76.0 | 80.0 |
| put the jam in the basket | 0.0 | 50.0 | 24.0 | 52.0 | 44.0 | 78.0 |
| MESA-Composite | DP | PG-FM | GR00T-N1.6 | FAST | ||
|---|---|---|---|---|---|---|
| put the apple on the plate | 0.0 | 66.0 | 54.0 | 70.0 | 64.0 | 84.0 |
| put the broccoli and mushroom on the cutting board | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 4.0 |
| put the carrot in the pan | 2.0 | 30.0 | 42.0 | 58.0 | 60.0 | 74.0 |
| put the cheese in the basket | 0.0 | 28.0 | 6.0 | 36.0 | 62.0 | 62.0 |
| put the cheese and mug on the tray | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 2.0 |
| put the cup in the left side of the dish drainer | 4.0 | 22.0 | 34.0 | 24.0 | 28.0 | 34.0 |
| MESA-Category | DP | PG-FM | GR00T-N1.6 | FAST | ||
|---|---|---|---|---|---|---|
| put the beet in the pot | 0.0 | 0.0 | 12.0 | 8.0 | 10.0 | 38.0 |
| put the brussel sprout in the pan | 0.0 | 2.0 | 6.0 | 8.0 | 4.0 | 8.0 |
| put the chili pepper in the bowl | 0.0 | 20.0 | 38.0 | 22.0 | 4.0 | 26.0 |
| put the cupcake in the basket | 2.0 | 14.0 | 28.0 | 14.0 | 8.0 | 26.0 |
| put the ginger on the cutting board | 0.0 | 10.0 | 12.0 | 0.0 | 26.0 | 20.0 |
| put the grapes in the basket | 2.0 | 12.0 | 16.0 | 24.0 | 8.0 | 28.0 |
| BiMESA-ID | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1- Sol | ||
|---|---|---|---|---|---|---|---|---|
| put the apple in the bowl (left arm) | 16.0 | 76.0 | 76.0 | 86.0 | 78.0 | 94.0 | 98.0 | 80.0 |
| put the apple in the box and close it | 0.0 | 0.0 | 8.0 | 16.0 | 16.0 | 50.0 | 46.0 | 10.0 |
| put the apple on the tray (right arm) | 28.0 | 48.0 | 74.0 | 78.0 | 66.0 | 88.0 | 92.0 | 70.0 |
| put the beer on the plate (left arm) | 26.0 | 70.0 | 70.0 | 76.0 | 56.0 | 88.0 | 98.0 | 60.0 |
| put the beer in the slide cabinet and close it | 0.0 | 0.0 | 0.0 | 6.0 | 10.0 | 22.0 | 20.0 | 0.0 |
| put the bell pepper in the basket (right arm) | 4.0 | 16.0 | 62.0 | 46.0 | 62.0 | 76.0 | 80.0 | 40.0 |
| BiMESA-Spatial | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1- Sol | ||
|---|---|---|---|---|---|---|---|---|
| put the apple in the bowl (right arm) | 24.0 | 56.0 | 76.0 | 84.0 | 76.0 | 88.0 | 92.0 | 50.0 |
| put the apple on the tray (left arm) | 24.0 | 78.0 | 72.0 | 82.0 | 58.0 | 86.0 | 80.0 | 70.0 |
| put the beer on the plate (right arm) | 22.0 | 36.0 | 64.0 | 72.0 | 42.0 | 80.0 | 88.0 | 50.0 |
| put the beer in the slide cabinet and close it (left arm, right fixture) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 6.0 | 0.0 |
| put the bell pepper in the basket (pick L, place R) | 20.0 | 32.0 | 52.0 | 44.0 | 56.0 | 60.0 | 80.0 | 10.0 |
| put the bell pepper on the tray (left arm) | 36.0 | 58.0 | 76.0 | 54.0 | 60.0 | 80.0 | 70.0 | 70.0 |
| BiMESA-Instance | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1- Sol | ||
|---|---|---|---|---|---|---|---|---|
| put the apple in the bowl | 12.0 | 56.0 | 54.0 | 62.0 | 42.0 | 64.0 | 72.0 | 40.0 |
| put the apple in the box and close it | 0.0 | 0.0 | 4.0 | 14.0 | 6.0 | 28.0 | 52.0 | 0.0 |
| put the apple on the tray | 18.0 | 30.0 | 20.0 | 62.0 | 22.0 | 54.0 | 42.0 | 80.0 |
| put the beer on the plate | 12.0 | 2.0 | 20.0 | 52.0 | 4.0 | 40.0 | 54.0 | 60.0 |
| put the bell pepper in the basket | 10.0 | 6.0 | 28.0 | 36.0 | 14.0 | 32.0 | 34.0 | 20.0 |
| put the bell pepper on the tray | 6.0 | 16.0 | 38.0 | 38.0 | 26.0 | 34.0 | 48.0 | 80.0 |
| BiMESA-Composite | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1- Sol | ||
|---|---|---|---|---|---|---|---|---|
| put the apple on the plate | 32.0 | 62.0 | 76.0 | 78.0 | 74.0 | 76.0 | 90.0 | 90.0 |
| put the broccoli and the mushroom on the cutting board | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 30.0 |
| put the carrot in the pan | 2.0 | 4.0 | 4.0 | 14.0 | 2.0 | 2.0 | 32.0 | 100.0 |
| put the fish and the garlic in the pan | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 2.0 | 2.0 | 20.0 |
| put the lemon on the cutting board | 28.0 | 54.0 | 84.0 | 70.0 | 72.0 | 90.0 | 94.0 | 100.0 |
| put the lemon and the lime in the basket | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 2.0 | 2.0 | 10.0 |
| BiMESA-Category | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1- Sol | ||
|---|---|---|---|---|---|---|---|---|
| put the beet in the pot | 2.0 | 0.0 | 4.0 | 16.0 | 2.0 | 2.0 | 8.0 | 30.0 |
| put the brussel sprout in the pan | 20.0 | 14.0 | 18.0 | 42.0 | 6.0 | 24.0 | 14.0 | 100.0 |
| put the chili pepper in the bowl | 0.0 | 0.0 | 0.0 | 8.0 | 0.0 | 2.0 | 2.0 | 40.0 |
| put the cupcake in the basket | 2.0 | 4.0 | 26.0 | 32.0 | 10.0 | 20.0 | 26.0 | 10.0 |
| put the ginger on the cutting board | 6.0 | 20.0 | 14.0 | 24.0 | 8.0 | 28.0 | 10.0 | 90.0 |
| put the grapes in the basket | 2.0 | 0.0 | 6.0 | 24.0 | 0.0 | 16.0 | 12.0 | 20.0 |
Robot Manipulation Benchmarks
Robot manipulation benchmarks measure how reliably robots perceive, plan, and execute physical tasks across objects, settings, and embodiments.
See all 363 papers