OverForge: Reasoning Through Strategies and Tactics Helps Cooperative Lifelong Adaptation

Oana Madalina Fron; Ojas Shirekar; Chirag Raman

cs.AISep 30, 2026

OverForge: Reasoning Through Strategies and Tactics Helps Cooperative Lifelong Adaptation

Authors: Oana Madalina Fron, Ojas Shirekar, Chirag Raman

Organizations: Tapri Lab, Department of Pattern Recognition and Bioinformatics, Delft University of Technology, Delft, The Netherlands

Abstract

Cooperative language-model agents must coordinate over long horizons and adapt to changing environments and to partners with unfamiliar conventions, yet existing agents map observations to actions without separating persistent coordination strategies from their tactical execution. We introduce OverForge, a training-free hierarchical architecture that separates strategic reasoning over roles and divisions of labour from tactical reasoning over actions within each agent's private, partner-conditioned world model. A metacognitive Prefrontal Cortex Module couples the two levels by forming strategy-action branches, imagining their consequences with a forward model, and committing when confident. In OvercookedV2, OverForge delivers 7 soups in a connected kitchen versus 3 for each flat LLM baseline, retains agreed roles, and adopts roles proposed by unfamiliar partners. Ablations and a fixed-strategy probe show that persistent strategies guide tactical adaptation while each reasoning level contributes to coordination. Memory restarts show that cross-episode partner knowledge supports task performance and partner prediction, linking the hierarchy to continual adaptation.

Figures & tables

Figure 1: The OverForge agent. Observations and messages update the beliefs in the world model $w_{t}$ ; the PCM selects the action inside it; the other components are inherited from MindForge. Actions denote intentions, tasks denote desires.

Figure 4: Episode 5 of OverForge self-play in the connected room (blue: agent_1; red: agent_0): a wrong recipe served unpaid (steps 43–116), a loader/finisher split held for 100 steps (194–293), the finisher blocking the loader inside it (200–226). Five-episode view: Figure 9 , Section A.7 .

Figure 5: How often a seat does what its partner asked ( “you grab the plate” ): as its next sub-task (circles, all agents) or as a role in one of its own strategy proposals (diamonds, OverForge only). Hollow: chance level, requests shuffled.

Figure 6: Cumulative (a) blocked steps and (b) deliveries of the team in the connected room, original run and memory restarts (cut dotted). Per-seat partner predictions: Figure 10 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: The branch system (a) next to one logged decision (b). Rows follow one deliberation from the maintained world model $w_{t}$ through the $K=2$ strategies, the first-action branches $b=(g,\tilde{a})$ with their pairwise-judged $V_{\mathrm{imm}}$ , the rollout rounds that extend only the nucleus, and the commit gate. (b) is agent_1 at step 194 of episode 5 in the connected room ( Figure 4 , conversation C2): two strategies yield four branches; the comparison prefers interact (pick up the plate) under both; round 1 rolls every branch one step, rounds 2 and 3 extend only the nucleus, and after round 3 the posterior concentrates on $b_{2}$ with $\kappa=0.84\geq\theta$ , so the agent commits. Imagined states are the forward model’s own predictions and carry its errors: $b_{4}$ places the agent on the counter cell (2,0), and $b_{2}$ predicts no change at depths 2 and 3 while the state judge still raises $\nu$ ; Table 9 quantifies both. All values are read from the PCM trace.

Figure 8: Start states of cramped_room (left) and asymm_advantages (right) at $N{=}2$ ; the $N{=}3$ starts are in Figure 17 .

	lay	$D\uparrow$	FI $\downarrow$	MB $\downarrow$	IA $\uparrow$	$c/s$
Self-play (matched partner)
OverForge	cr	1.4 $\pm$ 0.5	0.65 $\pm$ 0.08	0.22 $\pm$ 0.07	0.48 $\pm$ 0.14	40.2
	as	0.2 $\pm$ 0.4	0.78 $\pm$ 0.08	0.00 $\pm$ 0.00	0.48 $\pm$ 0.09	40.5
MindForge	cr	0.6 $\pm$ 0.8	0.51 $\pm$ 0.18	0.29 $\pm$ 0.14	0.37 $\pm$ 0.09	12.9
	as	0.6 $\pm$ 0.8	0.32 $\pm$ 0.23	0.00 $\pm$ 0.00	0.38 $\pm$ 0.04	13.0
MindForge-C	cr	0.6 $\pm$ 0.5	0.24 $\pm$ 0.07	0.20 $\pm$ 0.07	–	14.1

Appendix

Table 5: Full-agent comparison, per-episode mean and sd over five episodes (complete version of Table 2 ). cr/as: connected/split room;

D

: deliveries per episode; FI: failed-interaction rate; MB: blocking rate; IA: partner-intent accuracy (two-seat mean; undefined for IPPO and MindForge-C);

c/s

: LLM calls per agent-step. Bold: best LM self-play value per room.

arm	lay	$D\uparrow$	FI $\downarrow$	$\Sigma^{-}\downarrow$	IA $\uparrow$	$\bar{\kappa}$	cmt	$c/s$
Full PCM	cr	1.4 $\pm$ 0.5	0.65 $\pm$ 0.08	27.6 $\pm$ 5.8	0.48 $\pm$ 0.14	0.340	0.12	40.2
	as	0.2 $\pm$ 0.4	0.78 $\pm$ 0.08	31.8 $\pm$ 9.5	0.48 $\pm$ 0.09	0.343	0.14	40.5
$-$ rollouts	cr	0.8 $\pm$ 0.8	0.69 $\pm$ 0.03	24.6 $\pm$ 12.4	0.47 $\pm$ 0.11	0.430	0.10	17.8
	as	0.0 $\pm$ 0.0	0.70 $\pm$ 0.09	18.8 $\pm$ 9.4	0.43 $\pm$ 0.11	0.415	0.12	17.0
$-$ hierarchy	cr	0.4 $\pm$ 0.5	0.71 $\pm$ 0.06	40.8 $\pm$ 8.0	0.43 $\pm$ 0.04	0.536	0.42	26.6
	as	0.4 $\pm$ 0.5	0.73 $\pm$ 0.09	33.0 $\pm$ 5.8	0.45 $\pm$ 0.06	0.605	0.52	25.2

Appendix

Table 6: Ablation (one PCM component removed) and pinned strategy (seat 1 holds

g_{0}=

“be the pot-loader: keep a pot filled with what the order needs, and leave plating and serving to your teammate”; seat 0 free), per-episode mean and sd (complete version of Table 1 ).

\Sigma^{-}

: sub-tasks abandoned per episode;

\bar{\kappa}

, cmt: confidence and commit rate;

c/s

: LLM calls per agent-step.

Figure 9: Five episodes of OverForge self-play in the connected room on one axis (blue: agent_1; red: agent_0): partner-intent accuracy and failed-interaction rate per seat and episode; dish progress with paid (filled) and unpaid (hollow) serves; loads, platings and serves per seat; steps on which a move was blocked by the partner. Only memory crosses an episode boundary.

Figure 10: Memory restart, cumulative correct partner-intent predictions per seat in the (a) connected and (b) split room, original run and the two restarts. Restart lines start at the cut (dotted); kept seat solid, erased seat dashed with hollow markers.

Figure 11: Memory restart, collaboration measures per 100-step window: partner-intent accuracy and partner-model update rate per seat; blocking, duplicated sub-tasks and realised social influence per team. Restart lines start at the cut (dotted); kept seat solid, erased seat dashed. Blocking is zero in the split room by construction (the halves are disconnected).

Figure 12: Memory restart, sub-task completion per 100-step window: sub-tasks completed and abandoned, executed role events and failed-interaction rate per seat; progress completeness and deliveries per team. Same encoding as Figure 11 .

Figure 13: Share of proposed strategies per family and episode for the unconstrained agent, the taxonomy prompts and the free seat of the pinned arm, in the connected room (top) and the split room (bottom). Families are grouped: supply (supplier, relay), cook, finisher, coordination (coordinate, split by item, yielder), other. The taxonomy arm concentrates on supply and finisher, the prompt’s two worked examples.

Figure 14: Sub-task families over episode 2 for the pinned and the free seat in both layouts (fetch plate, plate and serve grouped as plate & serve; move and other grouped). Markers give executed loads, platings and serves; dark bars mark sub-tasks aimed at a location the seat cannot reach. The pinned seat loads pots in the connected room and stages and fetches in the split room.

Figure 15: The pinned strategy $g_{0}$ in both rooms, in the style of Figure 4 : dish progress, executed loads, platings and serves per seat, blocked steps and the pinned seat’s commitments, with three windows. In the connected room (episode 2) the free seat’s concrete offer of the ready soup revises $g_{0}$ once and the pinned seat plates and serves (window 1). In the split room under an all- ingredient_0 order (episode 1) the pinned seat holds $g_{0}$ throughout: it keeps committing to fetch from the pile (window 2) and carries its soup towards the serving station for 55 steps (window 3), a persistence that only a message, not a wall, revises.

Figure 16: Metacognitive process per ablation arm in the connected room (top) and the split room (bottom): (a) distribution of confidence $\kappa$ with the commit threshold $\theta=0.6$ , (b) rollout depth used, (c) cumulative gap between the two best branch utilities. Removing rollouts raises the share of near-tied decisions; removing the strategic layer produces a mass of decisions at $\kappa=1$ from single-branch sets.

Figure 17: Start states at $N{=}3$ : a third agent (green) is added to the connected room and to the left half of the split room, so the split room plays two against one.

Figure 18: First conversation of connected-room self-play (step 3, same start state), MindForge against OverForge, with OverForge’s PCM decision before and after it: strategies, first actions with utility $U$ , chosen strategy (shaded), confidence $\kappa$ . MindForge settles only the next move; OverForge settles the roles and commits. Transcript in Section A.9 .

Explore similar work

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Jun 6, 2026Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker +7Multi-Agent Coordination Multi-Agent Large Language Model Systems
University of Edinburgh · University of Oxford · University College London
SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correction
Aug 3, 2026Shen You, Xiaoming Zhu, Weining Weng +17Coordination Gradient Staleness
1City University of Hong Kong · 2Tencent · 3Shenzhen University +1
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
May 1, 2026Yi Xie, Siao Liu, Falong Fan +3Multi-Agent Large Language Model Systems Teaming
Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1

Jun 6, 2026, cs.AIJ/K move · Enter open · S save

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker, Alexander Rutherford+6

University of Edinburgh · University of Oxford · University College London

Multi-Agent Coordination Multi-Agent Large Language Model Systems Large Language Model Agents Coordination Competence

Open paper AskPDF arXiv

As language models are increasingly deployed as autonomous agents, they must coordinate with others over long horizons in open-ended interactive tasks. Yet existing evaluations rarely test these demands together, instead emphasising single-agent tasks, short interactions, or highly structured multi-agent settings. We introduce

alem

, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics. Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. We evaluate

13

modern LLMs zero-shot within homogeneous teams, with trained MARL agents as reference points. Current LLM agents remain far from solving alem, averaging only ~6% normalised return, but their failures are not uniform. On the hardest coordination setting, zero-shot Gemini-3.1-Pro-High approaches MARL agents trained for one billion steps, while GPT-5.4-High achieves strong base-task reward but much lower coordination reward. This contrast shows that individual task competence does not imply coordination competence. Ablations show that communication is the largest contributor to coordination, while memory and reasoning help when used to maintain multi-step plans. Overall, our results identify coordination as a distinct bottleneck for frontier LLM agents, separate from single-agent capabilities. Alem makes this bottleneck measurable and provides a controlled testbed for developing agents that communicate, allocate roles, and execute shared plans. Code is available at https://github.com/alem-world/alem-env.

Abstract

Explore similar work

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correction

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination