GitSwarm: Decentralized Compounding Inference

Vedant Shah; Ankur Samanta; Paras Dahal; Mikhail Plekhanov; Carole-Jean Wu; Scott Yih; Remi Munos; Rob Fergus

cs.AIOct 4, 2026

GitSwarm: Decentralized Compounding Inference

Authors: Vedant Shah, Ankur Samanta, Paras Dahal, Mikhail Plekhanov, Carole-Jean Wu, Scott Yih, Remi Munos, Rob Fergus,

Organizations: Meta Superintelligence Labs · Mila - Québec AI Institute · Université de Montréal · Columbia University ·

Abstract

Long-horizon problem solving and scientific research require computation to accumulate across successive attempts. Partial solutions, experimental findings, and unsuccessful approaches can inform later work, yet most inference-time computation is organized around individual trajectories or candidates rather than a persistent body of reusable work. We call this paradigm compounding inference: organizing inference-time computation so that intermediate work persists and can be inspected, extended, combined, or challenged by subsequent computation. We instantiate compounding inference in GitSwarm, an asynchronous system where homogeneous agents independently decide how to advance a task while collaborating through structured persistent memory. Agents explore, experiment, verify, refine, and synthesize previous work in a shared, branch-able Git repository. Atomic commits preserve intermediate artifacts, while explicit semantic dependencies record how later contributions build on work across branches. We evaluate GitSwarm on long-horizon problem solving and sustained GPU-backed experimental research. On IMOProofBench-Advanced, GitSwarm solves all 30 problems in one run using GPT-5.5. On ProgramBench, it achieves a $79.4\%$ mean score, versus $65.1\%$ for the strongest reported baseline under the stated budget. On three neural architecture research tasks (Residual Matrix Transformer, Looped Transformer, NanoChat), GitSwarm improves upon the starting architectures through successive experimentation. Beyond final performance, we measure whether computation accumulates: on ProgramBench, $94.7\%$ of contributions are subsequently built upon, while the selected solution's ancestry covers $82-93\%$ of the contribution graph. These results show that inference-time computation can accumulate across otherwise independent episodes, forming an evolving body of work that subsequent inference can reuse.

Figures & tables

Figure 1: Overview of GitSwarm (Schematic). Homogeneous workers independently inspect, extend, and contribute to a shared, branchable repository. Contributions may include exploration, verification, repair, experimentation, or synthesis. Atomic commits preserve intermediate artifacts, while semantic dependencies link work across branches. The highlighted ancestry illustrates how successive contributions can accumulate into a final solution.

Figure 2: Live swarm state. (a) A dashboard combining fields from a ProgramBench worker (

B=200

) and a Residual Matrix Transformer worker; shading marks fields shown only for one task type. (b) A ProgramBench worker inspecting a peer’s activity log. Text is taken from worker traces and abridged.

Figure 3: Inference-time scaling. Performance against input and output tokens on ProgramBench (a, b) and IMOProofBench-Advanced with Gemini-3.1-Pro (c, d). Each GitSwarm point represents one run at the indicated worker budget.

Panel of Figure 4 — Figure 4: RMT research. (a) Best-so-far development loss over the 72-hour research budget; the reference loss is 3.158. (b) The 8-GPU swarm’s research graph, showing competing branches, integration into the final architecture, and recovery of an earlier failed idea.

Panel of Figure 5 — Figure 5: Looped Transformer research. (a) Best-so-far development loss at eight recurrences. Arrows mark ongoing runs; the two-node GPT-5.6 run is not compute-matched. (b) The winning adapter’s development path and its wider evidence dependencies, including reversal of an early rejection and reuse of negative results.

Panel of Figure 6 — Figure 6: NanoChat research over the first 24 active hours. (a) Validation BPB across methods and hardware configurations. (b) Selected dependencies in the Astra/GB300 swarm. Green marks incumbent ancestry, gray dashed edges mark code or evidence reuse, and purple marks reuse of negative results. Crossed red nodes are negative results; hollow amber nodes are inconclusive. The graph contracts omitted ancestry and is not a time axis.

Figure 7: Selected-solution ancestry. Share of contributions reached through Git-parent ancestry, additional informed_by= dependencies, or neither, across worker budgets. ProgramBench solutions incorporate much of the contribution graph, primarily through cross-branch dependencies. In contrast, selected IMOProofBench proofs have shallower ancestry: later work often evaluates and compares existing candidates rather than extending the selected proof.

Figure 8: Selected-solution ancestry in a ProgramBench run (FFmpeg B100). Commits over normalized task runtime. The selected winner’s transitive ancestry (green), reached through Git parents and informed_by= edges, covers 83.7% of the contribution graph; only a few commits, mostly late in the run, lie outside it (gray).

Figure 9: Collaboration workflows in two ProgramBench runs. Commits over normalized task runtime. (a) Separately developed branches (colored) converge late on the strongest integration (diamond), which the selected winner builds on. (b) Work pivots from CLI and configuration (orange) to renderers (blue), which become the selected lineage.

Figure 10: ProgramBench worker activity over inference. Share of worker episodes by activity category (Table 12 ) against worker-spawn progress, per worker budget; the black line tops all contribution commits. After a short bootstrap wave, workers keep expanding, repairing, and synthesizing candidates until termination, and nomination rises only near the end. The IMOProofBench-Advanced counterpart appears in Figure 20 .

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

System	Retained state	Alternative paths	Recorded dependencies	Next-work control
ToT; LATS ( Yao et al., 2023 ; Zhou et al., 2024 )	Search states	Reasoning tree	Search edges	Search policy
Reflexion; ExpeL ( Shinn et al., 2023 ; Zhao et al., 2024 )	Reflections and experiences	Successive attempts	Episode or insight memory	Individual agent
MetaGPT; AutoGen ( Hong et al., 2024 ; Wu et al., 2023 )	Messages and task artifacts	Configured interactions	Workflow and messages	Roles or configured dialogue
GPTSwarm ( Zhuge et al., 2024 )	Executable agent graph	Graph variants	Operation-level edges	Graph optimizer
AgentGit ( Li et al., 2025 )	Versioned workflow states	Rollback and branches	State lineage	Application workflow
CAID ( Geng and Neubig, 2026 )	Git worktrees and commits	Parallel branches and merges	Code integration lineage	Central delegator

Appendix

Table 1: How related systems organize intermediate work. Representative mechanisms for retaining state, exploring alternatives, recording dependencies, and choosing subsequent work. This is a comparison of system designs, not a performance ranking.

System	Shared memory	Durable branches	Self- directed	No fixed roles	Async workers	Cross-branch deps.	Separate sources
MetaGPT ( Hong et al., 2024 )	–	–	–	–	–	–	–
AutoGen ( Wu et al., 2023 )	$\sim$	$\sim$	$\sim$	$\sim$	$\sim$	–	–
AgentGit ( Li et al., 2025 )	$\sim$	$\checkmark$	$\sim$	$\sim$	$\sim$	–	–
CAID ( Geng and Neubig, 2026 )	$\checkmark$	$\checkmark$	–	–	$\checkmark$	$\sim$	–
SwarmResearch ( Virk et al., 2026 )	$\checkmark$	$\checkmark$	–	–	$\checkmark$	–	–
GitSwarm	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 2: Design properties of related systems. Shared memory persists across worker episodes; branching preserves competing contributions; self-direction means workers choose their next task rather than receive assignments. Cross-branch dependencies explicitly connect contributions from different branches. Separate sources means the physical starting state is recorded apart from additional semantic sources.

\checkmark

: part of the reported design;

\sim

: partial or configuration-dependent; –: not part of the reported protocol.

Language	Tasks
C (8)	abishekvashok__cmatrix.5c082c6 , arq5x__bedtools2.dd57059 , chirlu__sox.42b3557 , ffmpeg__ffmpeg.360a402 , jqlang__jq.b33a763 , lua__lua.c6b4848 , sqlite__sqlite.839433d , universal-ctags__ctags.243595e
C++ (3)	gromacs__gromacs.665ea4c , halitechallenge__halite.822cfb6 , osgeo__gdal.0847f12
Go (11)	ariga__atlas.6d81150 , cheat__cheat.b8098dc , dundee__gdu.ede21d2 , incu6us__goimports-reviser.81bd549 , mgechev__revive.201451e , multiprocessio__dsq.c3ae0ba , noborus__ov.b96c2ba , noborus__trdsql.d8c5ff6 , psampaz__go-mod-outdated.bb79367 , segmentio__chamber.5f93f5f , sheepla__pingu.926d475
Haskell (1)	jgm__pandoc.5caad90
Java (1)	stathissideris__ditaa.f2286c4
Rust (26)	agourlay__zip-password-finder.704700d , alexpovel__srgn.89f943b , bensadeh__tailspin.6278437 , brocode__fblog.3b54330 , burntsushi__ripgrep.3b7fd44 , burntsushi__xsv.f430466 , cordx56__rustowl.655bc5c , crowdagger__crowbook.ea214d7 , dalance__amber.69a0f52 , dandavison__delta.acd758f , isona__dirble.e2dea9f , jrnxf__thokr.09375ef , nachoparker__dutree.44e877d , nikolassv__bartib.6b9b5ce , o2sh__onefetch.e5958ce , oppiliappan__statix.e9df54c , parcel-bundler__lightningcss.aa2ed1e , rhysd__kiro-editor.4157485 , shashwatah__jot.a92aad8 , simeg__eureka.df3796c , sirwart__ripsecrets.34c9e03 , sitkevij__hex.61ae69b , trasta298__keifu.3331426 , wfxr__code-minimap.0ddeea5 , wgunderwood__tex-fmt.3f1aef6 , xampprocky__tokei.505d648

Appendix

Table 3: ProgramBench evaluation subset. The 50 evaluation tasks, grouped by implementation language. Identifiers have the form owner__repository.commit .

Sweep	Value	Shared persistent	Independent, $B=2$	Independent, $k=4$
$P$ ( $D=4$ )	4	$(2,10)$	$(2,14)$	$(7,4)$
	8	$(4,10)$	$(2,27)$	$(14,4)$
	12	$(6,10)$	$(2,40)$	$(20,4)$
	16	$(8,10)$	$(2,54)$	$(27,4)$
	32	$(16,10)$	$(2,107)$	$(54,4)$
$D$ ( $P=4$ )	4	$(2,10)$	$(2,14)$	$(7,4)$

Appendix

Table 4: Hidden Rulebook settings. Entries in the final three columns give

(B,k)

: the number of workers and queries per worker.

Figure 12: Persistence reduces rediscovery. Relevant-record coverage in Hidden Rulebook as (a) the number of targets

P

and (b) the chain depth

D

grow, for shared persistent memory and two independent baselines with larger total query budgets. Mean over three seeds; bands:

\pm 1

SD.

Figure 14: Branch-native memory reduces repeated traversal. Repeated-move fraction in Hidden Maze for branch-native and flat persistent memory as (a) maze depth

D

and (b) branching factor

N

grow. Lines: mean over three mazes; open markers: individual mazes.

Sweep	Varied values	Fixed values	Archive sizes	$k$
Depth	$L=3,4,5,6$	$A=4,\ F=6$	$54,94,134,174$	$14,24,34,44$
Ambiguity	$A=2,4,6,8$	$L=5,\ F=6$	$72,134,196,258$	$18,34,49,65$
Fan-out	$F=4,6,8,10$	$L=5,\ A=4$	$116,134,152,170$	$29,34,38,43$

Appendix

Table 5: Retracted Evidence settings. One factor is varied at a time. The per-worker artifact-read budget

k

is set to

\lceil|\mathcal{A}|/4\rceil

, where

|\mathcal{A}|

is the generated archive size, so the four workers can read the complete archive in principle.

Figure 16: Attribution reduces lineage reconstruction. Fraction of the archive read before the first exact submission in Retracted Evidence, with and without attribution, as (a) lineage depth

L

, (b) semantic ambiguity

A

, and (c) fan-out

F

grow. Mean over three seeds; bands:

\pm 1

SD; dashed line: about one worker’s read budget.

Figure 17: IMOProofBench-Advanced proof scaling. Problems solved (7/7) against mean input (a) and output (b) tokens per problem for GitSwarm with Gemini-3.1-Pro and base (tool-calling) GPT-5.5-high workers, and for Forced Codex and Forced GPT-5.5, shown only in (b) for lack of input telemetry. Gemini points: mean of three judge passes.

Mechanism	What it does
Depth-scaling fix	Removes the reference’s division of attention scores by the layer index.
Tied embeddings	Shares input and output token embeddings; freed parameters widen the feed-forward layers.
SwiGLU feed-forward	Replaces the GELU feed-forward layer with a gated one of equal size.
Cross-head router	A zero-initialized learned mixing of attention scores across heads before the softmax.
Wider attention	Decouples the number of attention heads from the memory rank and widens it (16 $\rightarrow$ 32 $\rightarrow$ 64).
Temporal memory mixer	Adds gated copies of the memory states of the 1, 2, 4 and 8 previous tokens.

Appendix

Table 6: Architectural mechanisms in the best GitSwarm RMT model (8-GPU run).

Family	Commits	+/ $-$	Outcome	Note
Attention depth-scaling fix	4	3/1	In final model	adopted in final model
Tied embeddings / token interface	7	3/4	In final model	adopted in final model
Gated MLP (SwiGLU) variants	7	4/3	In final model	adopted in final model
Cross-rank attention router	6	5/1	In final model	adopted in final model
Attention (and MLP) rank widening	9	7/2	In final model	adopted; widened to rank 64
Multi-lag causal matrix mixing	13	2/11	In final model	adopted; 9 later variants rejected

Appendix

Table 7: Idea families explored by the GitSwarm RMT run: number of commits, positive/negative commits as labelled by their authors, and outcome. Family names follow the run logs: the “cross-rank attention router” is the cross-head router of Table 6 , and “rank widening” corresponds to wider attention. The family assignment covers 119 of the 120 commits, and its positive/negative split (48/71) differs slightly from the 69 negative commits reported in Section 3.3 .

Mechanism	Origin
Depth-scaling fix	Written on the solution path (0.6 h).
Tied embeddings	Written on the solution path (1.8 h).
SwiGLU feed-forward	Side branch (0.8 h); ported into the path at 5.9 h.
Cross-head router	Side branch (5.2 h), validated at full size at 15.1 h; ported at 34.6 h.
Wider attention	Side branch (8.2 h), validated at full size at 23.6 h; ported at 53.1 h.
Temporal memory mixer	Side branch (50.8 h), generalizing a refuted single-token version (1.2 h); ported at 65.9 h.

Appendix

Table 8: Where the final RMT model’s mechanisms were first written, and how they reached the solution path.

Family	Commits	Neg.	Evals	Outcome
Feed-forward re-injection adapter	13	0	20	In final model
Adapter capacity and placement variants	37	27	0	Refuted
Gate on the recurrent state	32	15	10	Early leader; gain vanished at 124M/188M
Variants of the re-injected signal	43	23	1	Refuted
Attention to the input / first core block	58	18	11	Early versions refuted; later ones in progress
Parallel specialist feed-forward branches	66	17	13	Beat the adapter at 52M; stalled at larger sizes

Appendix

Table 9: Idea families explored by the GPT-5.5 GitSwarm Looped Transformer run in its first 26 hours: architectural commits, commits published as negative by their authors, protected evaluations (128, 512 or 4,873 examples), and outcome. Two of the 64 protected evaluations are not assigned to an architectural family.

Benchmark	Category	Role
ProgramBench	Captured evidence	Raw outputs, logs, diffs, snapshots, or transcripts from executions.
ProgramBench	Human-readable findings	Prose that records an observed behavior, diagnosis, or validation result.
ProgramBench	Exploratory probe	Executable experiment used to discover unknown behavior.
ProgramBench	Validation harness	Repeatable executable check for an understood behavior or implementation.
ProgramBench	Input fixture	Input or corpus created to exercise candidate behavior.
ProgramBench	Artifact documentation	Index or instructions for using an artifact collection.

Appendix

Table 10: Benchmark-specific voluntary artifact categories. The categories were derived separately from files newly created by workers in each benchmark.

ProgramBench		IMOProofBench-Advanced
Budget	Runs with artifacts	Budget	Runs with artifacts
–	–	B20	21/30
B100	30/50	B40	20/30
B200	30/50	B80	19/30
B300	35/50	B120	23/30
B500	37/50	B160	20/30

Appendix

Table 11: Voluntary artifact production by budget. Entries give the number of task runs with at least one voluntary artifact bundle, after bundling files that lack independent reuse value.

Figure 18: Voluntary artifact production on ProgramBench. Artifact bundles per task (columns) and charged-worker budget (rows within each category band), in log-scale bins; the right margin counts runs with at least one bundle. Production is sparse and bursty across tasks.

Figure 19: Artifact flow from creation to reuse. For each artifact type on (a) ProgramBench and (b) IMOProofBench-Advanced, the share of created artifacts that are accessed, actively reused, and linked to the selected winner (bold), pooled over budgets. Rows without an Actively reused node skip that stage.

Benchmark	Category	Operational description
ProgramBench	Bootstrap	Creates an initial runnable candidate.
	Capability expansion	Adds a previously missing behavior or interface.
	Candidate synthesis	Combines useful behavior from multiple candidates.
	Fidelity repair	Corrects an existing behavior against task evidence.
	Empirical validation	Builds or runs focused tests and probes whose main contribution is evidence.
	Candidate selection	Compares or nominates existing candidates.

Appendix

Table 12: Bottom-up worker-activity taxonomies. The two taxonomies are derived separately because implementation and proof construction leave different trace evidence.

Figure 20: IMOProofBench-Advanced worker activity over inference. Share of worker episodes by activity category (Table 12 ) against worker-spawn progress, per budget; the black line tops all contribution commits. Unlike ProgramBench workers, who keep constructing until the end (Figure 10 ), proof workers shift early to selection.

Figure 21: Coordination-tool use over inference. Mean coordination-tool calls per worker against worker-spawn progress, per budget (light = small), for (a) ProgramBench and (b) IMOProofBench-Advanced. Proof traces contain no gs_inspect or gs_explore calls.

Figure 22: Worker episode outcomes. Share of episodes ending in a contribution commit, nomination, voluntary abstention, or invalid/no action, per budget, for (a) ProgramBench and (b) IMOProofBench-Advanced. Only voluntary abstentions do not consume the budget

B

Figure 23: Voting dynamics. Mean compounding nominations per problem against worker-spawn progress, per budget, for (a) ProgramBench and (b) IMOProofBench-Advanced; shading marks the last 10% of spawns. ProgramBench votes arrive in a terminal pulse, whereas proof votes accumulate steadily.

Figure 24: Selection timing. Worker-spawn progress at which the eventual winner is first committed (black) and the first nomination is cast (green), per budget; medians and interquartile ranges over problems. The winner appears late on ProgramBench and early on IMOProofBench-Advanced.

Figure 25: Declared redundancy. Share of spawned workers that abstain because equivalent work is committed or in flight, per budget, for (a) ProgramBench and (b) IMOProofBench-Advanced. Dots: problems; line: pooled rate with 95% CI; dashed line: one worker in six.

Figure 26: Timing and cited evidence of redundancy abstentions. (a, c) Tool turns at which abstainers stop, by the evidence they cite; gray ticks mark committing workers. (b, d) Abstainers per problem by cited evidence. ProgramBench abstainers mostly cite in-flight work; proof abstainers cite committed proofs and stop early.

Figure 27: Abstaining workers still consume tokens. Share of input and output tokens (output including reasoning) used by voluntarily abstaining workers, per budget, for (a) ProgramBench and (b) IMOProofBench-Advanced. Whiskers: bootstrap 95% CIs.

Figure 28: Redundancy abstentions almost always cite real work. Label distribution per budget: (a) score-blind review of a stratified ProgramBench sample; (b) trace-evidence audit of all IMOProofBench-Advanced redundancy abstentions, which checks that cited work exists, not proof equivalence.

Figure 29: Selective inspection as the graph grows. (a, c) Distinct candidate commits each worker opens and (b, d) share of the spawn-time history it reaches, against worker-spawn progress, per budget (ten-worker moving averages). Workers open a bounded number of candidates, so their reach shrinks as the history grows.

Execution schedule
Configuration	Score (%)	$\Delta$ (pp)	Runtime (min)
Asynchronous (default)	$62.06\pm 2.44$	—	38.0
Synchronous waves	$63.11\pm 0.58$	$+1.05$	49.3
Candidate selection
Decentralized nomination (default)	$62.06\pm 2.44$	—	38.0
Central selector	$64.40\pm 1.49$	$+2.34$	34.9

Appendix

Table 13: Architectural tradeoffs on the ProgramBench development set. Score is the macro-average test score over the nine development tasks (mean

\pm

sample standard deviation across three random seeds);

\Delta

is the difference in mean score from the default configuration in percentage points; runtime is the mean generation wall-clock time per task. All configurations use GPT-5.5-medium,

B=50

worker episodes per task, and maximum concurrency five. The default configuration (asynchronous execution with decentralized nomination) is the control in both blocks.

Setup	System prompt	After user prompt	User prompt
ProgramBench, Codex (GPT-5.5-high) workers	Global + ProgramBench addendum	Codex transport override	ProgramBench
ProgramBench, GPT-5.5 workers	Global + ProgramBench addendum	—	ProgramBench
IMOProofBench-Advanced, Gemini-3.1-Pro workers	Global + IMOProofBench addendum + sequential shell-interaction extension	—	IMOProofBench
RMT, Codex (GPT-5.6-sol xHigh) workers	Global a + RMT addendum	Codex transport override b	Task description c
Looped Transformer, Codex (GPT-5.5 High or GPT-5.6-sol xHigh) workers	Global a + Loop addendum	Codex transport override b	Task description c

Appendix

Table 14: Prompt components used in each GitSwarm setup listed in this appendix.

Inference-Time Scaling

Inference-time scaling studies how extra computation during deployment can improve model performance through search, sampling, verification, or adaptive reasoning.

See all 115 papers

Agora: Git as Shared Memory for Collective AutoResearch
Sep 16, 2026Yifan Zhang, Yunheng Zou, Shaokun Zhang +5AI Agents for Scientific Discovery Multi-Agent Collaboration
NVIDIA
SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery
Jul 2, 2026Yuvraj Virk, Zack Edds, Chunqiu Steven Xia +1Automated Algorithm Discovery Large Language Model-Guided Optimization
University of Illinois Urbana-Champaign
GitOfThoughts: Version-Controlled Reasoning and Agent Memory You Can Replay, Diff, and Merge
Jun 12, 2026Pavan C Shekar, Abhishek H S, Aswanth KrishnanLLM Agent Memory Persistent Memory for Language Models
QpiAI, Bengaluru, India

Sep 16, 2026J/K move · Enter open · S save

Agora: Git as Shared Memory for Collective AutoResearch

Yifan Zhang, Yunheng Zou, Shaokun Zhang, Jian Hu+4

NVIDIA

AI Agents for Scientific Discovery Multi-Agent Collaboration Multi-Agent Systems Agent Memory Management Autonomous Research Agents Shared-Memory Multi-Agent Systems

Open paper AskPDF arXiv

Research agents working in separate sessions need to know what others have tried and which results they can build on. Agora stores their contributions as an append-only directed acyclic graph (DAG) in Git. Each commit records a result, insight, hypothesis, verification, or report and links it to prior work. Searchable views show leading results, neglected branches, and verification status; diversity-aware recommendations suggest experiments beyond the current leaders. We report a run of nearly 12 days in which 13 language-model workers, with no assigned tasks or central planner, used Agora to solve a weight-transfer problem. Given 141 pretrained donor models and a frozen 119.6M-parameter attention--SSM hybrid whose dimensions match no donor, the workers had to initialize the target without training data or gradient updates. They published 1,703 contributions and reduced the development evaluator score from 3.39 to 1.899 bits per byte, closing 62% of the gap to a trained GPT-2 124M. The best method compresses donor next-token statistics into the target's embedding and output head, then adds short-range context through sparse edits to attention, feed-forward, and state-space blocks. Its 145-commit ancestry spans 15 accounts. Participants also posted 165 verifications of 95 targets, each by an account other than the target's author, with no reported failures. The run documents how agents reused and verified shared work. Measuring the effect on discovery per unit of compute requires a matched comparison.

Abstract

Inference-Time Scaling

Explore similar work

Agora: Git as Shared Memory for Collective AutoResearch

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

GitOfThoughts: Version-Controlled Reasoning and Agent Memory You Can Replay, Diff, and Merge