Organizations: International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou, 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Fakulti Teknologi Maklumat dan Komunikasi, Universiti Teknikal Malaysia Melaka, Melaka 76100, Malaysia · Computer Systems Engineering Department, Sukkur IBA University, Sindh, Pakistan · Yibin Park of University of Electronic Science and Technology of China, Yibin 644000, China
Large language models generate plausible code but cannot verify correctness. Existing multi-agent systems simulate execution or leave verification optional. We introduce execution-grounded verification as a first-class principle: every code change must survive sandboxed execution before propagation. We instantiate this principle in AGENTFORGE, a multi-agent framework where Planner, Coder, Tester, Debugger, and Critic agents coordinate through shared memory and a mandatory Docker sandbox. We formalize software engineering with LLMs as an iterative decision process over repository states, where execution feedback provides a stronger supervision signal than next-token likelihood. AGENTFORGE achieves 40.0% resolution on SWE-BENCH Lite, outperforming single-agent baselines by 26--28 points. Ablations confirm that execution feedback and role decomposition each independently drive performance. The framework is open-source at https://github.com/raja21068/AutoCodeAI.
Figures & tables
Framework
Execution Feedback
Role Decomposition
Memory/Retrieval
AgentForge
Mandatory sandbox
5 roles (plan, code, test, debug, critic)
Dual (episodic + repo)
SWE-agent
Shell/ACI (optional)
Single agent
None
OpenHands
Sandbox (optional)
Single agent
None
Trae Agent
Test-time scaling
3 roles (gen, prune, select)
None
MAGIS
Implied
4 roles (mgr, custodian, dev, qa)
None
AgentMesh
None
3 roles (plan, debug, review)
None
TABLE I : Positioning Multi-Agent Frameworks Along Three Axes
Fig. 1 : Overview of the AgentForge multi-agent coding framework, illustrating the sequential handover between specialized agents and the shared Vector Memory.
Fig. 2 : Retrieval-Augmented Generation (RAG) architecture: (a) Offline repository indexing phase into the vector store; (b) Online semantic retrieval at inference time.
Fig. 3 : Isolated Docker sandbox execution environment. The 512 MB memory limit and disabled networking ensure security and reproducibility.
Method
Resolve Rate
Patch Rate
Single-agent GPT-4o
14.0%
12.5%
ReAct (GPT-4o)
12.0%
11.0%
SWE-agent [ 31 ]
18.1%
17.0%
AgentForge (ours)
40.0%
37.5%
TABLE II : Resolution rates on SWE-bench Lite (300 tasks). AgentForge outperforms single-agent baselines under a fixed execution budget.
Fig. 4 : Performance of AgentForge across three evaluation axes on SWE-bench Lite. Left: Resolution rate as a function of debug retries ( N ). Shaded regions denote ±1 standard deviation across runs. Iterative execution and repair yield consistent gains, with diminishing returns after N=2 . Center: Resolution under k independent runs with majority voting (Pass@k). Performance scales with k without increasing per-run complexity. Right: Cost–performance tradeoff. AgentForge achieves higher resolution at lower cost compared to single-agent baselines, indicating improved sample efficiency.
Configuration
Resolved
Resolve Rate
Full pipeline (ours)
42
42.0%
w/o Critic agent
38
38.0%
w/o Debugger agent
31
31.0%
w/o Tester agent
28
28.0%
w/o Planner agent
19
19.0%
Single-agent baseline
14
14.0%
TABLE III : Ablation study on 100 SWE-bench Lite tasks. Each row removes one agent from the pipeline.
Fig. 5 : Resolve rates across ablation conditions. Removing any single agent degrades performance; the Tester–Debugger loop is the largest contributor.
Failure Category
Count
Percentage
Faulty localization (incorrect file/line)
12
40.0%
Ineffective patch generation (wrong or partial fix)
TABLE IV : Failure mode analysis on 30 failed tasks.
Fig. 6 : Resolve rates across ablation conditions. Removing any single agent degrades performance; the Tester–Debugger loop is the largest contributor.
Configuration
Input tokens
Output tokens
Cost (USD)
Full pipeline
13,600
5,100
$0.1445
Single-agent (GPT-4o)
5,000
1,800
$0.054
TABLE V : Token usage and estimated API cost per task (averaged).