Scaling Discovery through Test-Time Communication

Jongho Park; Vasilis Kontonis; Shivam Garg; Akshay Krishnamurthy; Dimitris Papailiopoulos

cs.LGSep 17, 2026

Scaling Discovery through Test-Time Communication

Authors: Jongho Park, Vasilis Kontonis, Shivam Garg, Akshay Krishnamurthy, Dimitris Papailiopoulos

Organizations: UC Berkeley · Microsoft Research

Abstract

Science advances not in isolation but through collaboration, yet existing agentic systems capture little of this. Whether communicating agents help remains an open question with mixed prior results. We show that test-time communication can substantially outperform independent parallel attempts on challenging tasks, where sharing a breakthrough can push the whole group forward. We first study the effect of scaling multi-agent test-time communication, where agents have no predefined roles and communicate via a shared directory, on ARC-AGI-3, a benchmark requiring novel problem solving. We find that a team of $k$ communicating agents, team@ $k$ , matches the success rate of $4k$ independent agents, and this advantage grows with $k$ , suggesting gains compound with scale. The effect is not merely efficiency: a task that no single agent can solve, a team of agents can solve reliably. Furthermore, these gains transfer to research-oriented tasks, given sufficient compute. On polyomino packing, communicating agents outperform best@ $k$ and exceed the prior best-known score. On MNIST classifier compression, communication surpasses the best-known human solution. A team of four agents produced a 1,957-byte classifier submission achieving 99.4% test accuracy, smaller than both the best-known human solution and the best single-agent result. These gains are not unconditional. Independent agents may outperform communication when compute is limited or when a clear measure of progress is absent. However, under sufficient compute and clear feedback, multi-agent communication consistently yields stronger results.

Figures & tables

Figure 1: Communication outperforms independent agents in open-ended tasks. With communication, a team of

k

communicating agents not only outperforms best@

k

but scales success rate on ARC-AGI-3 faster. On algorithmic and ML optimization tasks, team@

k

beats single-agent runs and best-known human solutions, yielding new state-of-the-art results.

Benchmark	ARC-AGI-3	Polyomino packing	MNIST Compression
Model	Sonnet 4.6	Sonnet, Opus 4.6	GPT-5.6 Sol
Budget per agent	per-level action budget	3-72 hours	96 hours
Team size $k$	3, 5	3, 4	4
Metric	game solve rate	packing score	qualified bytes
Prior best-known result	–	0.894	2,461 B
Independent best@ $k$	1.4% ( $k$ =3), 2.2% ( $k$ =5)	0.893	3,160 B

Table 1: Summary table. We evaluate multi-agent communication and compare the best of

k

independent agents with a team of

k

communicating agents on three tasks. All agents are run using the GitHub Copilot CLI and communication harness we describe in Section 2 .

	$k=3$			$k=5$
Levels from target	team@3	best@3	best@13	team@5	best@5	best@33
5	52.8	45.4	63.9	64.1	52.8	69.5
4	28.2	25.8	46.5	33.6	32.9	55.4
3	16.6	9.7	20.8	22.9	12.8	31.3
2	7.2	5.0	9.3	16.2	6.1	15.8
1	4.8	2.6	6.3	10.0	3.7	10.2

Table 2: ARC-AGI-3 per-level solve rate. Proportion of trials reaching within

d

levels of a full solve, averaged over 25 games. Test-time communication pays off most where the task is hardest. At

k=5

, team@

k

leads best@

k

1.2\times

at shallow depths and by

3.6\times

at a full solve. Communication also scales with team size. Matching team@

k

’s final solve rate takes a pool of 13 independent agents at

k=3

and 33 agents at

k=5

Figure 2: Net communication effect on each ARC-AGI-3 game. (a) Average furthest level reached by team@5 and best@5, sorted by absolute difference. Communication helps on 18 of the 25 games and hurts on 7, and the best gain is

+4.30

levels on LP85 while no loss exceeds

0.85

. (b) Success rate across 64 single-agent trials and 20 team@5 trials. The aggregate gain comes from a handful of games, as most remain out of reach for Sonnet 4.6. Nevertheless, where communication helps, it helps substantially. Across the four games with any team@5 success, solve rate rises from

13.8\%

50.0\%

on average. Most strikingly, LP85 goes from no single-agent success to

65\%

Figure 3: Token efficiency of communicating agents. Solve rate against cumulative output tokens for team@

k

, best@

k

, and the number of independent agents needed to match the team’s final solve rate. Best@

k

never reaches the team’s final rate at any budget. For

k

=3 and

k

=5, respectively, matching that rate takes

3.8\times

and

4.9\times

more tokens than the team spends. The gain is not immediate. Best@

k

leads in a low-compute regime (

\leq

400K output tokens per agent), a coordination tax that the team overcomes at larger budgets.

Figure 4: Effective collaboration depends on compute allocation. For games SB26 and LP85, we vary the per-agent action budget while holding the total action budget at

5\times

the native budget (solid line). We also consider the case in which a team must divide one agent’s native budget (faded line). The marker shape indicates per-agent budget, while the color indicates agent configuration. When each agent is given the native budget, per-level team@5 leads at every level past the first, ahead of both best@5 and a single agent given

5\times

the budget. Cutting the same five agents to

0.2\times

each drops team@5 below a single agent.

Figure 5: Relative human action efficiency of communicating agents. RHAE (Eq. 1 ) for the team’s best agent and average agent across all games. RHAE scores trajectories based on whether and how quickly an agent solves each level relative to human performance. The figure shows that communication makes each agent better off. On average, each agent’s action efficiency increases with team size, while the best agent in the team is more efficient than the best independent agent.

Figure 6: Communicating agents surpass the single-agent plateau and set a state-of-the-art packing score. Best-so-far packing score for the highest-scoring run among 60 single-agent trials and 20 team@3 trials. Team@3 pulls decisively ahead at about 1.6 hours and reaches

0.945

, a new best score within the benchmark, above

0.883

for the single agent and the prior-best score of

0.894

. Numbered markers identify method discoveries and their contributing agents. Shaded bands track changes in the team’s leading method.

Figure 7: The Opus 4.6 team sustains its advantage over a longer horizon. Best-so-far packing score for the highest-scoring run among 12 single-agent trials and two team@4 trials, each allowed up to 72 hours. Time is shown on a logarithmic axis. The team takes a lasting lead at about two hours and reaches

0.922

, while the single agent plateaus at

0.893

, just below the dotted prior-best reference of

0.894

. Markers and shading follow Figure 6 .

Figure 8: The longer Sonnet 4.6 team run also clears the single-agent plateau. Best-so-far packing score for the highest-scoring run among 12 single-agent trials and two team@4 trials, each allowed up to 72 hours, with the selected team trajectory ending at roughly 42 hours. Time is shown on a logarithmic axis. The team takes a lasting lead at roughly four hours and reaches

0.910

, while the single agent finishes at

0.891

. The dotted line marks the prior best of

0.894

. Markers and shading follow Figure 6 .

Figure 9: Communication yields a denser polyomino packing. The single-agent solution and the best Sonnet team@3 solution pack the same 172 pieces (1,654 cells) at $74.9\%$ and $93.8\%$ density, respectively, a gain of 18.9 percentage points. The labels $0.753$ and $0.945$ are the runs’ benchmark-average scores.

Figure 10: Communication leads to state-of-the-art ML compression. Smallest submission with at least 99.4% test accuracy over 96 hours, against wall-clock time and against output tokens, using GPT-5.6 Sol agents. The advantage does not come from aggregating more attempts. All independent runs converge above the 2,461-byte best-known human solution, while the team reaches 1,957 bytes. The team pays the coordination tax here too, trailing for about the first hour and 100K output tokens.

Panel of Figure 11 — Figure 11: Combining complementary ideas produces the final MNIST classifier. (a) Two shared-log entries from a1 and a subsequent excerpt from a2 show successive adoption of a3 ’s and a2 ’s scoring-layer designs, followed by a2 taking up the combined model. (b) The same filter set, comprising depthwise $3\times 3$ and pointwise $1\times 1$ convolution filters, updates the 32 feature maps in three loops at each resolution (see Eq. 3 ). The scoring layer combines six learned weighted sums with twelve selected features. The final network contains 2,900 quantized parameters and 30 float16 scale factors.

Panel of Figure 12 — Figure 12: Verified progress sharing allows a team to advance faster. (a) Under independent parallel sampling, each agent advances only along its own trajectory, so success requires at least one agent to complete the entire trajectory independently. (b) By sharing verified progress, the task is divided into successive phases, and at each checkpoint, all agents continue from the improved state, where a different agent may supply the next breakthrough.

Method	Mean Accuracy	Max Accuracy
Single agent pass@1	52.53%	–
Independent pass@2	62.36%	64.04%
Communicating team@2	60.67%	61.80%

Table 3: Communication does not outperform pass@

k

on Terminal-Bench 2.0.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	Solo container	Team container	Task GPU
ARC-AGI-3	4 CPUs, 8 GiB	4 CPUs, 8 GiB	None
Polyomino packing	2 CPUs, 6 GiB	2 CPUs, 6 GiB	None
MNIST compression	3 CPUs, 8 GiB	12 CPUs, 32 GiB ( $k=4$ )	A100
Terminal-Bench 2.0	Task-defined	Task-defined, shared	Task-defined

Appendix

Table 4: CPU/GPU and container setup for each task.

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 305 papers

Scaling Inference-Time Compute with Teams of Coding Agents
Oct 12, 2026Keenan Samway, Daniel D'souza, Omar Darwiche Domingues +3Inference-Time Scaling Inference-Time Compute Allocation
Cohere Labs · Cohere
Benefits and Limitations of Communication in Multi-Agent Reasoning
Oct 14, 2025Michael Rizvi-Martel, Satwik Bhattamishra, Neil Rathi +2Multi-Agent Reasoning Communication Complexity
Mila & Universit´e de Montr´eal · University of Oxford · Stanford University +1
When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding
Aug 17, 2026Giuseppe Destefanis, Tomaso AsteMulti-Agent Coordination Multi-Agent Coding
Department of Computer Science, University College London, UK

Oct 12, 2026J/K move · Enter open · S save

Scaling Inference-Time Compute with Teams of Coding Agents

Keenan Samway, Daniel D'souza, Omar Darwiche Domingues, Marzieh Fadaee+2

Cohere Labs · Cohere

Inference-Time Scaling Inference-Time Compute Allocation Multi-Agent Coding Software Engineering Agents Cost-Aware Inference

Open paper AskPDF arXiv

Multi-agent systems promise a new axis for scaling inference: rather than having one agent think longer or sample more attempts, a task is split across many agents that work in parallel and coordinate. However, in practice, more agents can easily produce duplicate work, coordination failures, and higher costs. We therefore formulate team design as a compute allocation problem: starting from a solo agent, where should additional inference compute go to reach the reward-cost Pareto frontier? We build a minimal multi-agent harness from the ground up and trace this frontier by varying, one at a time, the workflow stage at which a lead delegates to its workers (planning, implementation, or verification), the models that form the team, and the number of workers. On ProgramBench, teams outperform solo agents but cost three to seven times as much. With the right compute allocation, however, a heterogeneous 17-agent team comes within 1.4 points of a strong homogeneous 5-agent team at 70% lower cost. This efficiency stems from three findings. First, planning is the most valuable use of workers: in a homogeneous team whose lead decides how many workers to spawn, a lead that delegates only planning to its workers, and then implements and verifies the code itself, matches the reward of a lead that delegates to workers throughout planning, implementation, and verification, at half the cost. Second, cheap models pay off as planners but not as implementers: a strong lead with cheap planners recovers 47.5% of the reward gain of an all-strong team for only 13.9% of its additional cost, whereas cheap implementers lower reward. Third, planners keep improving reward as the team grows, whereas implementers and verifiers plateau beyond four workers. Together, our results offer a guide to spending an agent budget where it counts, given the task, cost, and latency constraints a user has.

Abstract

Multi-Agent Collaboration

Explore similar work

Scaling Inference-Time Compute with Teams of Coding Agents

Benefits and Limitations of Communication in Multi-Agent Reasoning

When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding