Can AI Models Direct Each Other? Organizational Structure as a Probe into Training Limitations

Rui Liu

cs.SEMar 27, 2026

Can AI Models Direct Each Other? Organizational Structure as a Probe into Training Limitations

Authors: Rui Liu

Organizations: Independent Researcher

Abstract

Can an expensive AI model effectively direct a cheap one to solve software engineering tasks? We study this question by introducing ManagerWorker, a two-agent pipeline where an expensive "manager" model (text-only, no code execution) analyzes issues, dispatches exploration tasks, and reviews implementations, while a cheap "worker" model (with full repo access) executes code changes. We evaluate on 200 instances from SWE-bench Lite across five configurations that vary the manager-worker relationship, pipeline complexity, and model pairing. Our findings reveal both the promise and the limits of multi-agent direction: (1) a strong manager directing a weak worker (62%) matches a strong single agent (60%) at a fraction of the strong-model token usage, showing that expensive reasoning can substitute for expensive execution; (2) a weak manager directing a weak worker (42%) performs worse than the weak agent alone (44%), demonstrating that the directing relationship requires a genuine capability gap--structure without substance is pure overhead; (3) the manager's value lies in directing, not merely reviewing--a minimal review-only loop adds just 2pp over the baseline, while structured exploration and planning add 11pp, showing that active direction is what makes the capability gap productive; and (4) these behaviors trace to a single root cause: current models are trained as monolithic agents, and splitting them into director/worker roles fights their training distribution. The pipeline succeeds by designing around this mismatch--keeping each model close to its trained mode (text generation for the manager, tool use for the worker) and externalizing organizational structure to code. This diagnosis points to concrete training gaps: delegation, scoped execution, and mode switching are skills absent from current training data.

Figures & tables

Panel of Figure 1 — Figure 1: The ManagerWorker pipeline. The manager (top lane, text-only) analyzes, reviews, and plans. Workers (bottom lane, full repo access) explore and implement. Loop arrows on the left side show iterative rounds. Communication is structured natural language.

Configuration	Description	Manager	Worker	Pipeline
ManagerWorker	Full structured pipeline	Sonnet 4.6	GPT-5-mini	5-phase
Simple Loop	Minimal review loop	Sonnet 4.6	GPT-5-mini	execute $\to$ review
Strong Direct	Single strong agent	—	Sonnet 4.6	none
Weak Direct	Single weak agent	—	GPT-5-mini	none
Weak $\to$ Weak	Weak manager, weak worker	GPT-5-mini	GPT-5-mini	5-phase

Table 1: Experimental configurations. Direct configurations are single-agent baselines. ManagerWorker and Simple Loop test two levels of pipeline complexity with the same model pairing. Weak

\to

Weak tests whether structure helps without a capability gap.

Configuration	Manager	Worker	$n$	Resolved	Rate
ManagerWorker (full pipeline)	Sonnet 4.6	GPT-5-mini	200	124/200	62%
Strong Direct	—	Sonnet 4.6	200	120/200	60%
Simple Loop	Sonnet 4.6	GPT-5-mini	200	106/199	53%
Weak Direct	—	GPT-5-mini	200	101/200	51%
50-instance subset (for Weak $\to$ Weak comparison):
ManagerWorker (full pipeline)	Sonnet 4.6	GPT-5-mini	50	32/50	64%

Table 2: Main results. Top: 200-instance evaluation. ManagerWorker matches Strong Direct (+2pp) using primarily cheap worker tokens. Bottom: 50-instance subset including Weak

\to

Weak, which performs worse than Weak Direct (42% vs 44%)—structure without a capability gap is net negative.

Configuration	Resolve	Strong tokens	Weak tokens	Total tokens
ManagerWorker (full)	62%	$\sim$ 6.6K	$\sim$ 60K	$\sim$ 67K
Strong Direct	60%	$\sim$ 30K	0	$\sim$ 30K
Simple Loop	53%	$\sim$ 3K	$\sim$ 30K	$\sim$ 33K
Weak Direct	51%	0	$\sim$ 15K	$\sim$ 15K
Weak $\to$ Weak	42%	0	$\sim$ 75K	$\sim$ 75K

Table 3: Quality–cost tradeoff across configurations. Token estimates per instance based on typical prompt sizes and session lengths. Resolve rates for top four configs from

n=200

; Weak

\to

Weak from

n=50

. ManagerWorker achieves comparable quality to Strong Direct while shifting

\sim

90% of tokens to the cheap model. Weak

\to

Weak is the worst of both worlds: high total tokens and low quality.

Batch	Resolved	Rate	Notes
0–50	26/49	53%	Initial pipeline
50–100	30/51	59%	+ code context in reviews
100–150	33/52	63%	+ relaxed execution constraints
150–200	17/47	36%	Same pipeline as 100–150; harder instances

Table 4: Simple Loop (v2) performance by batch. Batches 50–150 benefited from both pipeline improvements and easier instances, so these rates are confounded. The clean comparison is batch 100–150 (63%) vs 150–200 (36%): same pipeline, harder instances. This 27pp drop exposes the ceiling of the review-only approach.

Combination	Resolved
ManagerWorker	32/50 (64%)
Strong Direct	29/50 (58%)
Union	33/50 (66%)
Intersection	28/50 (56%)
ManagerWorker only	4/50 (8%)
Strong Direct only	1/50 (2%)

Table 5: Complementarity analysis on 50-instance subset. ManagerWorker resolves nearly all instances that Strong Direct resolves (28/29), plus 4 additional. The approaches solve largely overlapping but not identical subsets.

Manager Configuration	Resolved (toy-5)	Key Change
v5–v6: Manager with repo access	0–1/5	Manager hallucinates “already fixed”
v7: Manager text-only	2/5	Forced abstract reasoning
v9: Text-only + iterative exploration	3/5	Manager can request more info

Table 6: Director design ablation on a 5-instance development set. Giving the manager repository access caused a regression—the model reverted to its training behavior (read files, form conclusions) rather than delegating investigation.

Strategy	Resolved	Empty	Eval Err	Description
Strict only	26/50 (52%)	8	6	Worker follows plan exactly; no autonomy
Guided only	27/50 (54%)	5	1	Worker has full autonomy on every round
Guided-then-strict	32/50 (64%)	0	2	Autonomy on round 1, strict on retries

Table 7: Communication strategy ablation on 50-instance subset. The guided-then-strict strategy produces the largest improvement (+10pp over strict-only), showing that cheap models need different levels of constraint at different stages.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Solved by ManagerWorker only	Solved by Strong Direct only
matplotlib-23299	astropy-14365
matplotlib-23563
matplotlib-23987
seaborn-3190
sympy-12171

Appendix

Table 8: Instances uniquely solved by each approach. 5 unique to ManagerWorker , only 1 unique to Strong Direct. The manager-worker pipeline subsumes nearly all of the single agent’s capability.

Multi-Agent Coding

Multi-agent coding studies how AI agents can collaborate to generate, test, debug, and maintain software.

See all 84 papers

Jun 30, 2026

SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks

Large language models (LLMs) embedded in multi-turn agentic harnesses are reshaping software engineering (SWE), but routing every task to a frontier model is wasteful when many issues admit cheap fixes. Existing LLM routers operate on the task description alone, which inherits an information-theoretic Bayes-error floor in agentic settings: a similar issue can hide either a localized typo or a multi-module refactor, and the prompt does not separate the two. We introduce SWE-Router, a value-based temporal approach that lets a cheap model run for a few exploratory turns and reads the resulting partial trajectory before deciding whether to continue cheaply or to escalate to an expensive model. We provide a Bayes-optimality theorem showing that conditioning on the partial trajectory never harms routing and is strictly better whenever exploration is informative. Across the LLM pairs of weak and strong models spanning the contemporary cost--capability frontier, we show that SWE-Router greatly improves the cost efficiency of SWE tasks, while maintaining the majority of the performances of the stronger model. We additionally release a multi-LLM trajectory dataset which allows reproduction of our trajectory-level routing.

Seongho Son, Sangwoong Yoon, Jiahua Tang +3

University College London, United Kingdom · Ulsan National Institute of Science and Technology, South Korea · PSL Research University, France +1

Software Engineering Agents LLM Agent Workflow Optimization LLM Routing LLM Agent Harnesses LLM Agent Routing

Ask PDF

Feb 25, 2026

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

Small language models (SLMs) offer compelling advantages in cost, latency, and adaptability, but have so far lagged behind larger models on long-horizon software engineering tasks such as SWE-bench, where they suffer from pervasive action looping and low resolution rates. We introduce SWE-Protégé, a post-training framework that reframes software repair as an expert-protégé collaboration problem. In SWE-Protégé, an SLM remains the sole decision-maker while learning to selectively seek guidance from a strong expert model, recognize stalled states, and follow through on expert feedback. Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration. We lightly post-train Qwen2.5-Coder-7B-Instruct to achieve 42.4% Pass@1 on SWE-bench Verified, a +25.4% improvement over the prior SLM state of the art, while using expert assistance sparsely (~4 calls per task and 11% of total tokens).

Patrick Tser Jern Kon, Archana Pradeep, Ang Chen +5

University of Michigan · Meta · Stanford University

Software Engineering Agents Small Language Models LLM Agents Automated Program Repair RL for Code Generation

Ask PDF

Oct 2, 2026

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

Large language models (LLMs) are increasingly used in software engineering, including agentic systems that coordinate multiple agents, but impose higher computational and environmental costs. In this paper, we present a comprehensive empirical study of agentic LLM systems across five software engineering tasks: code generation, technical debt identification, code vulnerability detection, log parsing, and log analysis. For each task, we compare LLM configurations that range from a non-agentic single-query baseline to multi-agent workflows, using six open-weight LLMs, two prompt strategies, and three hardware platforms. We assess each configuration in terms of accuracy, inference latency, and energy consumption. Our results reveal substantial trade-offs between agentic complexity and energy efficiency: multi-agent designs consume on average 6.36

\times

as much energy and run 6.07

\times

as long as the non-agentic baseline, with worst-case slowdowns of up to 160

\times

for individual task--hardware pairs. Accuracy gains from additional agents are limited and task-specific: multi-agent improves average vulnerability-detection accuracy, but lightweight non-agentic and single-agent configurations still dominate the Pareto front, accounting for 59 of 66 Pareto-optimal configurations. Model and prompt choice act as task-specific levers whose effective direction varies between tasks rather than as global defaults. We translate these findings into design guidelines for sustainable, task-aware LLM-based development tools.

Merve Astekin, Yan Naing Tun, Arda Goknil +5

SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye

Software Engineering Agents LLM Agent Evaluation Multi-Agent LLM Systems LLM Inference Efficiency Energy-Efficient ML

Ask PDF

Abstract

Multi-Agent Coding

Explore similar work

SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows