CRAFT: Grounded Multi-Agent Coordination Under Partial Information

Abhijnan Nath; Hannah VanderHoeven; Nikhil Krishnaswamy

cs.CLMar 26, 2026

CRAFT: Grounded Multi-Agent Coordination Under Partial Information

Authors: Abhijnan Nath, Hannah VanderHoeven, Nikhil Krishnaswamy

Organizations: Situated Grounding and Natural Language (SIGNAL) Lab∗ Department of Computer Science, Colorado State University Fort Collins, CO 80523 USA

Abstract

We introduce CRAFT, a multi-agent benchmark for evaluating pragmatic communication in large language models under strict partial information. In this setting, multiple agents with complementary but incomplete views must coordinate through natural language to construct a shared 3D structure that no single agent can fully observe. We formalize this problem as a multi-sender Bounded Pragmatic Speaker problem and provide a diagnostic framework that decomposes failures into spatial grounding, belief modeling and pragmatic communication errors, including a taxonomy of behavioral failure profiles in both frontier and open-weight models. Across a diverse set of models, including 8 open-weight and 7 frontier including reasoning models, we find that stronger reasoning ability does not reliably translate to better coordination: smaller open-weight models often match or outperform frontier systems, and improved individual communication does not guarantee successful collaboration. These results suggest that multi-agent coordination remains a fundamentally unsolved challenge for current language models. Our code can be found at https://github.com/csu-signal/CRAFT

Figures & tables

Figure 1: CRAFT overview. A structure generator creates a target 3D object and 3 private 2D views for Directors, enforcing information asymmetry. Each turn, Directors produce instructions from their partial views, and a Builder executes them in the CRAFT environment, which logs task progress and evaluates communication for spatial grounding , mind modeling , and pragmatic sufficiency .

Figure 2: Director perspective views for a 25 block structure with the “camera” at the “bottom” of the Full Grid view. D1 (left wall), D2 (far wall), and D3 (right wall) each represent a fixed 2D projection across all vertical layers. The Full Grid minimap shows ground-truth stack heights.

Model	Prog. $\uparrow$	Comp. $\uparrow$	Pos. Acc. $\uparrow$	IoU $\uparrow$	Fail $\downarrow$	REMOVE $\downarrow$	Gap $\downarrow$
Gemini-3-Flash	$\mathbf{0.675}_{{\pm 0.052}}$	$\mathbf{0.716}_{{\pm 0.045}}$	$\mathbf{0.594}_{{\pm 0.065}}$	$\mathbf{0.817}_{{\pm 0.034}}$	$0.625_{{\pm 0.125}}$	$\mathbf{0.196}$	$\mathbf{0.018}$
GPT-4o	$0.588_{{\pm 0.051}}$	$0.633_{{\pm 0.048}}$	$0.500_{{\pm 0.061}}$	$0.753_{{\pm 0.038}}$	$0.421_{{\pm 0.116}}$	$0.280$	$0.056$
GPT-4o-Mini	$0.333_{{\pm 0.041}}$	$0.383_{{\pm 0.040}}$	$\underline{0.233}_{{\pm 0.047}}$	$0.531_{{\pm 0.041}}$	$0.550_{{\pm 0.114}}$	$0.432$	$0.254$
GPT-4.1-Mini	$0.312_{{\pm 0.053}}$	$0.352_{{\pm 0.054}}$	$0.233_{{\pm 0.056}}$	$0.481_{{\pm 0.053}}$	$0.500_{{\pm 0.115}}$	$0.463$	${0.388}$
Claude-Sonnet-4.6	$0.285_{{\pm 0.036}}$	$0.332_{{\pm 0.038}}$	$0.189_{{\pm 0.039}}$	$0.479_{{\pm 0.041}}$	$0.350_{{\pm 0.109}}$	$0.395$	$0.265$
Gemini-2.5-Flash	$\underline{0.257}_{{\pm 0.030}}$	$0.279_{{\pm 0.033}}$	$0.206_{{\pm 0.033}}$	$0.428_{{\pm 0.039}}$	$\mathbf{0.300}_{{\pm 0.105}}$	$0.467$	$0.402$

Table 1: Task performance in CRAFT across frontier models (top) and open-weight models (bottom) across 20 games after turn

t=20

. The Builder receives up to 5 “legal” moves per turn and selects among them based solely on Director instructions. Prog. , Comp. , Pos. Acc. , and IoU are continuous metrics; Failed is a binary per-turn outcome. Subscripted values show Mean

{}_{{\pm\text{SEM}}}

. REMOVE and Gap are aggregated over all turns. Bold = best within group. Underline = worst across all models.

Figure 3: Failure taxonomy over all turns across 15 director models.

Figure 4: Per turn oracle-prescribed vs. attempted remove rate, averaged across all 20 structures (shading = remove gap , or difference between fraction of turns where Directors instruct a remove action and fraction where oracle prescribes one; green : model under-removes, orange : model over-removes). Subplot titles show mean gap and final-turn task progress.

Figure 5: LLM grader scores across three evaluation dimensions— spatial grounding (left), mind modeling (center), and pragmatic sufficiency (right)—broken down by question and model group. Error bars denote standard error of the mean across all structure–turn–director observations per model from 3 independent runs.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Director perspective views for structure_016 , a complex-tier structure with 25 total blocks. D1 (left column, $j{=}0$ ) sees a large yellow domino spanning $(0,0)$ – $(1,0)$ at L2, a small red at $(0,0)$ and large yellow domino spanning $(1,0)$ – $(2,0)$ at L1, and small orange, red, green blocks at L0. D2 (top row, $i{=}0$ ) sees a large orange domino spanning $(0,0)$ – $(0,1)$ with small red at $(0,2)$ at L0, a large red domino spanning $(0,0)$ – $(0,1)$ with small red at $(0,2)$ at L1, and small yellow, red, blue at L2. D3 (right column, $j{=}2$ ) sees small blue, red, orange at L2, small red, yellow, green at L1, and a large red domino spanning $(0,2)$ – $(1,2)$ with small red at $(2,2)$ at L0. The shared position $(0,0)$ is visible to both D1 and D2, serving as the sole grounding anchor between views. The Full Grid minimap shows all seven required positions at height 3; the optional positions $(1,1)$ and $(2,1)$ have heights 1 and 2 respectively, both topped by green blocks. No Director can reconstruct the full structure unilaterally: D1 and D3 observe vertical depth along their respective walls but cannot see interior or opposite-wall positions, while D2 has exclusive visibility into interior positions but cannot observe vertical structure below the topmost block. Notably, the large orange domino spanning $(0,0)$ – $(0,1)$ at L0 is seen as a full domino by D2 (both cells visible) but appears as a small block to D1, which can only see $(0,0)$ and has no visibility into $(0,1)$ . Similarly, the large red domino spanning $(0,2)$ – $(1,2)$ at L0 is seen as a full domino by D3 but appears small to D2, which sees $(0,2)$ but not $(1,2)$ . These cases illustrate how the same physical block can have different apparent sizes depending on the observing Director’s projection, a direct consequence of the partial observability design.

Archetype	Description	D1	D2	D3	Total
Assertive	Confident and direct; forms hypotheses quickly and shares them, updates when others provide compelling evidence.	75	60	30	165
Cautious	Methodical and verification-focused; synthesizes others’ observations before adding interpretation.	60	90	75	225
Observant	Notices patterns and anomalies; flags inconsistencies and connects information across directors.	15	30	90	135
Skeptical	Questions assumptions including its own; probes claims to ensure group correctness, comfortable with uncertainty.	45	75	75	195
Synthesizer	Integrates all directors’ observations into a coherent picture; reconciles contradictions and drives shared understanding.	105	45	30	180
Total		300	300	300	900

Appendix

Table 2: Director personality archetypes used in CRAFT experiments. Each archetype shapes the Director’s internal reasoning style and public communication tone. Assignments are deterministic per (structure_index, run, director_id) ensuring consistency across all model evaluations.

Type	Model	Correct	Layer	Span	Other	Wrong-Color	Wrong-Pos
Base	DeepSeek-Lite	.711	.086	.086	—	.089	.026
	Gemma-9b	.738	.120	.069	—	.066	.006
	Llama-8b	.774	.119	.036	.003	.062	.006
	Mistral-7b	.759	.094	.085	—	.053	.009
	Qwen-7b	.817	.070	.034	—	.061	.018
	Qwen-14b	.616	.209	.088	.025	.051	.011

Appendix

Table 3: Failure taxonomy for base and frontier models across CRAFT evaluation runs on 20 target structures. Each cell shows the fraction of oracle-available turns; Correct is the full oracle match rate. Engine errors are diagnosed from the game-engine response after the Builder agent makes a move; positional, color, and span errors reflect oracle-level mismatches.

Figure 7: Turn-level outcome rates per model sorted by oracle adherence. Oracle available (orange) is uniformly high; the gap between oracle available and oracle followed (green) directly corresponds to the failure fractions in Table 3 . No-oracle turns (purple) are rare across all models ( $<15\%$ ).

Time (s)	$\Delta t$ (s)	Speaker	Utterance	Role
570.0	-29.6	Builder	This green?	Clarification query
571.1	-28.5	Director	Yes.	Confirmation
571.3	-28.3	Director	Yeah, stick that on top of the blue.	Instruction
572.7	-26.9	Builder	Right here?	Clarification query
572.9	-26.7	Director	No, no, no, no.	Correction
574.1	-25.5	Director	The yellow that we have, for me it is showing green.	Perceptual mismatch

Appendix

Table 4: Dialogue segment illustrating ambiguity resolution, perceptual misalignment, and multi-agent grounding prior to moving the small red block ( RS ) at layer 1.

\Delta t

is measured relative to the action at

t=599.6

s from Group 3.

Model	Turn	Director instruction	Failure & cause
F1 — Wrong block (base speaker failure)
DeepSeek-Lite	T8	D1 & D3 (identical) : “place a large orange block spanning (0,1) and (0,2)”	Builder places os at layer 2; oracle needs bs . Both directors describe target state not current state; redundant identical messages add zero information.
DeepSeek-Lite	T1	D2 & D3: “large orange block spanning middle and right of my bottom layer”	Builder places ol at (1,2) $\rightarrow$ (2,2) ; oracle needs ol at (0,0) span (0,1) . Frame-of-reference ambiguity: “middle and right” resolves to wrong global coordinates.
F2 — Correction spiral (ToM listener failure)
Qwen-32B	T13	D1: “remove the large orange block from the middle-left of my bottom layer”; D2: “remove the orange block from my bottom left corner”	Cannot remove layer 0 at (1,0) — ol sits at layer 1 not layer 0; both directors specify “bottom layer” without checking current stack depth. Oracle recommends place gs @ (0,0) layer 2 .
Qwen-32B	T14–15	D1 & D3: same remove instruction as T13	Identical error, identical board state. Three consecutive turns consumed; no director issues prerequisite remove (1,0) layer 1 needed to unblock target.

Appendix

Table 5: Representative turn-level failures organized by BPS failure mode, as identified in Sec. 3.1 . The Qwen-32b deadlock (Fig. 22 ) across turns 10–15 illustrates how a single wrong early placement traps directors in an irrecoverable correction spiral when no agent tracks the repair plan across turns.

Figure 8: Three turns of zero progress in Qwen-32b ( structure_001 , T10–T14): D1 and D2 repeatedly instruct removal from the wrong layer while the oracle recommends forward placement, and the Builder follows Director consensus over oracle, consuming three turns without any board state change (full trace in Table 11 ).

Panel of Figure 9 — Figure 9: Director prompt used in the experiments, section I. Note that personality and archetypes are populated from Table 2 .

Panel of Figure 10 — Figure 10: Director prompt used in the experiments, section II.

Panel of Figure 11 — Figure 11: Director prompt used in the experiments, section III.

Panel of Figure 12 — Figure 12: Builder prompt used in the experiments, section I.

Panel of Figure 13 — Figure 13: Builder prompt used in the experiments, section II.

Panel of Figure 14 — Figure 14: Builder prompt used in the experiments, section III.

Panel of Figure 15 — Figure 15: Builder’s More Exploration Tool Call Prompt. Note that although CRAFT provides this facility, this is not explored in our current benchmark, in favor of oracle moves in the Builder’s observation space in order to restrict the action space of the Builder for controlled experiments.

Panel of Figure 16 — Figure 16: Spatial Grounding (SG) judge. Evaluated once per turn across 20 structures and 20 overall turns over the collective Director messages.

Panel of Figure 17 — Figure 17: Mind Model (MM) Judge. Evaluated once per turn across 20 structures and 20 overall turns over the collective Director messages. Note that oracle correct moves are not provided to the MM judge during evaluation.

Panel of Figure 18 — Figure 18: Pragmatic Sufficiency (PS) judge. Evaluated once per turn over the collective Director messages. Six binary questions targeting location specificity, block type precision, rational sufficiency, spatial anchoring, Builder understanding, and failure attribution.

Judge	Q	Exact	W. $\kappa$	Rank $\rho$	$p$
SG	SG1	0.665	0.265	0.821	0.000
	SG2	0.547	0.146	0.697	0.004
	SG3	0.473	0.260	0.516	0.049
	SG4	0.510	0.139	0.427	0.113
	SG5	0.565	0.314	0.649	0.009
	SG6	0.545	0.271	0.825	0.000

Appendix

Table 6: GPT-4o-mini vs GPT-4o agreement on a stratified subsample selected as 2 turns per model

\times

structure for SG and MM (15 models

\times

20 structures

\times

2 turns

=600

director-turns each, yielding

4{,}200

and

4{,}800

question-answer pairs) and 1 turn per model

\times

structure for PS (

n=295

turns,

1{,}770

pairs). Exact agreement and linearly weighted

\kappa

measure absolute agreement; Rank

\rho

reports Spearman correlation of per-model mean scores between judges (

n=15

models). Bold rows indicate questions with significant rank preservation (

p{<}0.05

). GPT-4o applies stricter correctness thresholds throughout, producing lower Yes rates; the preserved model rankings confirm that relative comparisons reported in the paper are robust to this calibration difference.

Model	SG $\pm$ SEM	MM $\pm$ SEM	PS $\pm$ SEM	Progress
Frontier Models
Gemini-3-Flash	$0.669\pm 0.010$	$0.237\pm 0.009$	$0.356\pm 0.015$	$0.675$
GPT-4o	$0.792\pm 0.006$	$0.706\pm 0.004$	$0.492\pm 0.010$	$0.588$
GPT-4o-Mini	$0.741\pm 0.005$	$0.632\pm 0.005$	$0.457\pm 0.006$	$0.333$
GPT-4.1-Mini	$0.937\pm 0.003$	$0.787\pm 0.003$	$0.481\pm 0.005$	$0.312$
Claude-Sonnet-4.6	$0.910\pm 0.006$	$0.775\pm 0.006$	$0.444\pm 0.006$	$0.285$

Appendix

Table 7: LLM grader scores (mean

\pm

SEM) for spatial grounding (SG), mind modeling (MM), and pragmatic sufficiency (PS), alongside task progress. Frontier models achieve higher SG and MM scores but lower progress than open-weight models. All scores are averaged over three independent grader runs .

Panel of Figure 19 — Figure 19: Per-model LLM grader scores across all judge questions for spatial grounding (top), mind modeling (second), and pragmatic sufficiency split by condition—oracle followed (third) and oracle rejected (bottom). Models are sorted by task progress descending along the x-axis. PS scores for oracle-followed turns exclude PS6 (Failure attribution), which is not applicable when the builder executes a correct move. All scores are averaged across independent grader runs ( $n{=}3$ ).

Feature	Pearson $r$	$p$	Spearman $\rho$	$p$
Behavioral
Remove gap	$-0.780^{***}$	0.001	$-0.785^{***}$	0.001
Message calibration (MM)
Novel information (MM1)	$-0.619^{*}$	0.014	$-0.652^{**}$	0.008
Overall MM score	$-0.603^{*}$	0.017	$-0.613^{*}$	0.015
Unique perspective (MM5)	$-0.575^{*}$	0.025	$-0.536^{*}$	0.039

Appendix

Table 8: Pearson and Spearman correlations of Director communication features with task progress at turn 20 (

n=15

models). Features are ordered by absolute Pearson

r

. Judge question codes in parentheses refer to the evaluation framework in Sec. E . Significance:

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

Model	$\lambda_{D1}$	$\lambda_{D2}$	$\lambda_{D3}$	$\lambda$ -std	Filtered	Total	Oracle	Orc%	Deg%
Frontier Models
Claude-Sonnet-4.6	0.336	0.363	0.302	0.0250	35	400	176	44.0%	11.8%
Gemini-2.5-Flash	0.340	0.351	0.308	0.0182	48	400	164	41.0%	15.7%
Gemini-3-Flash	0.344	0.360	0.296	0.0270	84	385	269	69.9%	21.3%
Gemini-3.1-Flash-Lite-Preview	0.323	0.347	0.331	0.0100	28	400	158	39.5%	10.1%
GPT-4.1-Mini	0.334	0.320	0.346	0.0106	26	400	171	42.8%	8.9%

Appendix

Table 9: Empirical

\lambda_{i}

weights and associated diagnostics.

\lambda_{D1}

\lambda_{D2}

\lambda_{D3}

denote the empirical credit share attributed to each Director by the Builder on oracle-correct turns, normalized to sum to one. Oracle is the number of turns where the Builder executed a verified oracle-correct move. Orc% is the fraction of total turns corresponding to oracle-correct moves. Deg% denotes the percentage of degenerate credits removed during filtering. Total represents the total number of actual turns evaluated (termination condition) across 20 test structures.

Figure 20: Oracle-prescribed vs. attempted remove rate per turn for all base open-weight models.

Figure 21: Oracle-prescribed vs. attempted remove rate per turn for all frontier and proprietary models.

Model	Orc_Remove $\downarrow$	Att_Remove	Gap
Open-weight models
DeepSeek-Lite	0.433	0.148	$-$ 0.275
Mistral-7b	0.238	0.124	$-$ 0.124
Qwen-7b	0.238	0.205	$-$ 0.116
Gemma-9b	0.211	0.122	$-$ 0.084
Llama-8b	0.232	0.277	$-$ 0.057

Appendix

Table 10: Remove action rates per model. Oracle remove rate reflects what the CRAFT game engine considers necessary given the current board state; attempted remove rate reflects what Directors instructed the Builder to do. Negative gap indicates under-removal (Builder removes less than oracle prescribes); positive gap indicates over-removal. Bold = best within group. Underline = largest absolute gap. Turn-level Spearman correlations use full oracle adherence as outcome (

n=5,382

Figure 22: Three turns, zero progress: a CRAFT correction spiral (Qwen-32b, structure_001 , T10–T14) (see Table 11 for the full turn-by-turn trace). D1 (left wall) sees a large orange domino at layer 1 spanning (0,0)--(1,0) sitting directly on top of a small yellow at (0,0) layer 0 and a large blue domino spanning (1,0)--(2,0) at layer 0. The orange domino is therefore the top block at both positions and is directly accessible for removal — yet D1 instructs “remove the large orange block from the bottom layer,” targeting layer 0 which is occupied by the blue domino. D2 (far wall) independently sees orange small blocks at layer 1 on (0,0) and (0,1) but similarly instructs removal from “bottom left corner,” again specifying layer 0. D3 (right wall) sees only a single orange block at (2,2) layer 0 and has no visibility into either problem position, remaining entirely unaware of the “deadlock”. The full grid minimap shows five occupied positions with maximum height 3 at (0,1) ; the orange blocks at (0,0) and (1,0) each sit at height 2 as the topmost block, confirming they are accessible. Despite this, neither D1 nor D2 corrects the layer specification across turns 10, 13, and 14—the Builder’s oracle recommends forward placement moves throughout ( place gs @(0,0) L2 , place rl @(1,0) L2 ) but the Builder follows the Director consensus and attempts the same failed remove three times, consuming turns without any board state change.

Turn	Oracle (top move)	Director messages & outcome
T10	place gs @(0,0) L2	D1: “remove the large orange block from the middle-left of my bottom layer ” $\cdot$ D2: “remove orange from bottom left corner” $\cdot$ Builder follows D1 $\rightarrow$ Cannot remove layer 0 at (1,0)
T11	place gs @(0,0) L2	D2: same remove instruction $\cdot$ D3: “confirm small orange in bottom right” $\cdot$ Builder follows D3 $\rightarrow$ remove os @(0,2) L0 $\checkmark$ progress=0.355
T12	place gs @(0,0) L2	D1: same remove instruction $\cdot$ D3: “add small orange to bottom right” $\cdot$ Builder follows D3 $\rightarrow$ place os @(2,2) L0 $\checkmark$ progress=0.386
T13	place gs @(0,0) L2	D1: “get rid of large orange from bottom layer middle-left” $\cdot$ D2: “remove orange from bottom left corner ” $\cdot$ D3: unaware of deadlock, confirms own wall $\cdot$ Builder follows D1+D2 consensus $\rightarrow$ Cannot remove layer 0 at (1,0)
T14	place gs @(0,0) L2	D1: identical instruction as T13 $\cdot$ D3: “ focus on removing the large orange from D1’s bottom layer ” (endorses wrong instruction) $\cdot$ Builder follows D1+D3 consensus $\rightarrow$ Cannot remove layer 0 at (1,0)

Appendix

Table 11: Turn-by-turn trace of the Qwen-32b correction spiral on structure_001 . Despite the oracle recommending forward placement moves at every turn, D1 and D2 repeatedly issue remove instructions targeting the wrong layer, while D3 remains unaware of the deadlock. The Builder escapes the spiral on T11 and T12 by following D3 instead, but returns to the failed pattern when D1 and D2 reach consensus on T13–14.

Explore similar work

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Jun 6, 2026Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker +7Multi-Agent Coordination LLM Agent Evaluation
University of Edinburgh · University of Oxford · University College London
Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue
May 13, 2026Vardhan Dongre, Dilek Hakkani-TürMulti-Agent Coordination AI Agent Benchmarks
Siebel School of Computing & Data Science University of Illinois Urbana-Champaign
Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems
Mar 1, 2026Yuzhe Zhang, Feiran Liu, Yi Shan +8Multi-Agent LLM Systems Multi-Agent Coordination
Beijing University of Technology, Beijing, China · Zhejiang University, Hangzhou, China · Meituan LongCat Interaction Team +2

Jun 6, 2026J/K move · Enter open · S save

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker, Alexander Rutherford+6

University of Edinburgh · University of Oxford · University College London

Multi-Agent Coordination LLM Agent Evaluation AI Agent Benchmarks Long-Horizon LLM Agents Multi-Agent Communication

Open paper AskPDF arXiv

As language models are increasingly deployed as autonomous agents, they must coordinate with others over long horizons in open-ended interactive tasks. Yet existing evaluations rarely test these demands together, instead emphasising single-agent tasks, short interactions, or highly structured multi-agent settings. We introduce

alem

, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics. Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. We evaluate

13

modern LLMs zero-shot within homogeneous teams, with trained MARL agents as reference points. Current LLM agents remain far from solving alem, averaging only ~6% normalised return, but their failures are not uniform. On the hardest coordination setting, zero-shot Gemini-3.1-Pro-High approaches MARL agents trained for one billion steps, while GPT-5.4-High achieves strong base-task reward but much lower coordination reward. This contrast shows that individual task competence does not imply coordination competence. Ablations show that communication is the largest contributor to coordination, while memory and reasoning help when used to maintain multi-step plans. Overall, our results identify coordination as a distinct bottleneck for frontier LLM agents, separate from single-agent capabilities. Alem makes this bottleneck measurable and provides a controlled testbed for developing agents that communicate, allocate roles, and execute shared plans. Code is available at https://github.com/alem-world/alem-env.

Abstract

Explore similar work

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue

Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems