Coordinating two players in a cooking task requires a vision-language agent to track changing scene states, sequence recipe steps, and allocate shared resources under a time budget. We introduce an Overcooked-inspired benchmark for evaluating these capabilities in Unreal Engine environments constructed using coding agents. A single vision-language model controls both players from top-down visual observations, selecting high-level actions while the engine executes navigation. The benchmark contains 20 maps in five categories. These categories support evaluation of centralized coordination under equipment contention, changing terrain, and handoff dependencies. A prop legend and pot ingredient-count indicators provide explicit visual cues, while grid-based destinations support spatial grounding. The evaluation compares four VLMs on all 20 maps, measuring task score, order completion, delivery precision, action acceptance rate, and model API expenditure. A memory ablation on the same 20 maps examines the effect of a shared, self-authored notebook while retaining six-response action history in both conditions. The strongest evaluated model, gpt-6-astra, achieves 26.2% mean order completion. Notebook memory has mixed score effects across models, with positive mean differences for Claude and Qwen and negative differences for GPT and Gemini.
Figures & tables
Figure 1: Benchmark pipeline. Left: coding-agent kitchen construction, manual refinement, and actor-contract verification. Center: joint control of two players from shared multimodal inputs, illustrated by onion-soup preparation. Right: timed-order scoring and evaluation of score, order completion, delivery precision, action acceptance, and API expenditure.
Zone
Elapsed waiting time w (steps)
Base tip
Green
0≤w≤1,350
8
Yellow
1,350<w≤2,100
5
Red
2,100<w≤2,550
3
Last call
2,550<w<3,000
0
Table 1: Base tips by elapsed waiting time for a non-expired order. The thresholds correspond to 45%, 70%, and 85% of the 3,000-step waiting limit.
Figure 2: Input components retained in both memory conditions. Left: instructions, live-state JSON, legal operations, six-response action history, and execution feedback. Right: a top-down kitchen image with grid labels and pot indicators, plus a prop legend. The additional notebook used in the memory-enabled condition is not shown.
Controller
Score ↑
Completion (%) ↑
Precision (%) ↑
Acceptance (%) ↑
USD/episode ↓
qwen3.8-27b
9.1
8.8
91.7 ( n=6 )
39.5
0.27
gemini-3.5-flash-lite
11.9
11.2
81.0 ( n=7 )
32.1
0.30
claude-opus-5
8.2
8.8
100.0 ( n=5 )
52.7
4.61
gpt-6-astra
25.6
26.2
93.8 ( n=16 )
64.6
5.18
Table 2: Main benchmark performance, averaged with equal weight across 20 maps, with one episode per model and map. Arrows give the preferred direction. Delivery precision excludes maps with no deliveries; n gives the number of maps entering that mean. Bold values mark the best reported mean in each column.
Figure 3: Mean order completion by map category. Each column averages the four maps in that category with equal weight.
Figure 4: Effect of shared memory, as paired differences (with memory minus without) on the 20 ablation maps. (a) Score difference: grey points are per-map differences, the marker is the equal-weight mean with a 95% percentile bootstrap interval over resampled maps. (b) API cost per episode under each condition, log scale. Intervals resample maps rather than repeated runs, since each model-map-condition result is a single episode; they summarize sensitivity of the mean to resampling the observed maps, not the distribution of individual map effects or run-to-run variability. The constructed maps are not a random sample from a specified population, so these intervals do not establish generalization beyond this suite.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Map category
Source level
Main
Memory
Basic
1-1
Yes
Yes
1-3
Yes
Yes
1-4
Yes
Yes
1-5
Yes
Yes
Shared equipment
3-1
Yes
Yes
6-4
Yes
Yes
Appendix
Table 3: Source-level correspondence and evaluation membership. Main denotes the full 20-map main evaluation; Memory denotes the 20-map memory ablation.
Agent systems powered by multimodal large language models (MLLMs) have advanced rapidly in recent years, yet existing embodied-agent benchmarks still lack fine-grained diagnostics for multi-agent coordination. Most benchmarks either focus on single-agent task completion or summarize multi-agent behavior with overall task success rates, which can obscure coordination failures such as duplicated work, violations of ordering constraints, resource contention, and desynchronized handoffs. In this paper, we introduce CoCoBench, a construct-level benchmark for evaluating multi-agent embodied coordination in executable household tasks. CoCoBench contains 897 oracle-validated instances organized around four recurring coordination constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination. In addition to task success rate, CoCoBench provides construct-level scores that measure whether agents coordinate effectively. We evaluate 11 leading MLLMs across different coordination modes, observation inputs, and numbers of agents. The results show that coordination ability is highly construct-specific: strong overall performance does not imply balanced competence across different coordination types. These findings point to new directions for designing targeted model architectures and improving multi-agent coordination ability.
AI agents need to plan to achieve complex goals that involve orchestrating perception, sub-goal decomposition, and execution. These plans consist of ordered steps structured according to a Temporal Execution Order (TEO, a directed acyclic graph that ensures each step executes only after its preconditions are satisfied. Existing research on foundational models' understanding of temporal execution is limited to automatically derived annotations, approximations of the TEO as a linear chain, or text-only inputs. To address this gap, we introduce MATEO (MultimodAl Temporal Execution Order), a benchmark designed to assess and improve the temporal reasoning abilities of Large Vision Language Models (LVLMs) required for real-world planning. We acquire a high-quality professional multimodal recipe corpus, authored through a standardized editorial process that decomposes instructions into discrete steps, each paired with corresponding images. We collect TEO annotations as graphs by designing and using a scalable crowdsourcing pipeline. Using MATEO, we evaluate six state-of-the-art LVLMs across model scales, varying language context, multimodal input structure, and fine-tuning strategies.
Gabriel Roccabruna, Olha Khomyn, Giuseppe Riccardi
Signals and Interactive Systems Lab, University of Trento, Italy
Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pipeline requires resolving incompatible dependencies, matching underspecified evaluation protocols, and reverse-engineering undocumented preprocessing. This burden scales with the number of models and benchmarks, making comprehensive evaluation impractical for most teams. We present vla-eval, an open-source evaluation harness that eliminates this per-benchmark cost by decoupling model inference from benchmark execution through a WebSocket+msgpack protocol with Docker-based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four-method interface; the full cross-evaluation matrix works automatically. The framework supports 14 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves up to 47x wall-clock speedup, completing 2,000 LIBERO episodes in ~18 minutes. To validate the framework, we reproduce published scores across six VLA codebases and three benchmarks, documenting previously undocumented pitfalls. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available at https://github.com/allenai/vla-evaluation-harness and https://allenai.github.io/vla-evaluation-harness/leaderboard.
Suhwan Choi, Yunsung Lee, Yubeen Park +4
Allen Institute for AI (AI2) · Seoul National University