SEMAG: Self-Evolutionary Multi-Agent Code Generation

Yulin Peng; Haowen Hou; Xinxin Zhu; Ying Tiffany He; F. Richard Yu

cs.SEMar 16, 2026

SEMAG: Self-Evolutionary Multi-Agent Code Generation

Authors: Yulin Peng, Haowen Hou, Xinxin Zhu, Ying Tiffany He, F. Richard Yu

Organizations: College of Computer Science and Software Engineering, Shenzhen University, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China · School of Information Technology, Carleton University, Canada

Abstract

Large Language Models (LLMs) have made significant progress in handling complex programming tasks. However, current methods rely on manual model selection and fixed workflows, which limit their ability to adapt to changing task complexities. To address this, we propose SEMAG, a Self-Evolutionary Multi-Agent code Generation framework that mimics human coding practices. It decomposes programming tasks into stages, including planning, coding, debugging, and discussion, while adapting workflows to task difficulty. Its self-evolutionary agents can access the latest models in real time and automatically upgrade the backbone model. SEMAG sets new state-of-the-art Pass@1 accuracy across benchmarks. Using identical backbone models, SEMAG outperforms prior methods by 3.3% on CodeContests. When augmented with self-evolutionary model selection that automatically identifies optimal backbones, SEMAG reaches 52.6%, showcasing both framework effectiveness and adaptability to evolving LLM capabilities.

Figures & tables

Figure 1: Overview workflow of Self-Evolution Agents. Agents integrate insights from recent research, news, and community discussions, dynamically identify and deploy the most suitable models.

Figure 2: Overview of SEMAG. (1) Self-Evolve: Agents dynamically select optimal backbone LLMs per task requirements. (2) Plan: Planning Agent creates solution plans validated by Plan Verifying Agent through I/O simulation. (3) Debug: Coding Agent generates code; upon failure, specialized agents (Embedding Trace, Code Explaining, Suggesting, Debugging) collaboratively refine using trace logs. (4) Debate: When debugging stalls, Debating Agents propose alternatives with Discriminating Agent selecting the optimal configuration.

Model	Method	HumanEval	MBPP	HumanEval-ET	MBPP-ET
GPT-3.5	Direct	72.0% $\pm$ 1.2%	55.2% $\pm$ 0.8%	62.8% $\pm$ 0.6%	45.6% $\pm$ 0.6%
	Self-Planning	77.4% $\pm$ 1.8%	69.2% $\pm$ 0.4%	69.5% $\pm$ 0.6%	52.4% $\pm$ 1.0%
	MapCoder	77.4% $\pm$ 0.6%	72.0% $\pm$ 0.6%	66.5% $\pm$ 1.2%	56.6% $\pm$ 0.8%
	LDB	81.1% $\pm$ 0.6%	72.4% $\pm$ 0.2%	72.6% $\pm$ 1.8%	55.6% $\pm$ 0.4%
	LPW	89.0% $\pm$ 0.8%	76.0% $\pm$ 0.2%	77.4% $\pm$ 0.8%	57.6% $\pm$ 0.2%
	SEMAG (Ours)	91.5% $\pm$ 1.8%	76.2% $\pm$ 0.8%	79.9% $\pm$ 0.6%	64.4% $\pm$ 0.4%

Table 1: Pass@1 accuracy comparison of different methods using GPT-3.5 on code generation benchmarks. The values enclosed in parentheses represent the improvement over the Direct Prompting approach. The standard deviation (

\pm

) is calculated based on the results of three independent runs and applies to the data analysis of subsequent experiments.

Model	Method	HumanEval	MBPP	HumanEval-ET	MBPP-ET
GPT-4o	Direct	91.5% $\pm$ 1.8%	62.8% $\pm$ 0.4%	79.3% $\pm$ 1.2%	51.0% $\pm$ 0.2%
	LDB	92.1% $\pm$ 1.2%	82.4% $\pm$ 0.8%	81.7% $\pm$ 1.8%	65.4% $\pm$ 1.0%
	LPW	98.2% $\pm$ 0.6%	84.8% $\pm$ 0.6%	84.8% $\pm$ 1.2%	65.8% $\pm$ 0.8%
	SEMAG (Ours)	98.8% $\pm$ 0.6%	87.6% $\pm$ 0.4%	86.6% $\pm$ 0.6%	71.8% $\pm$ 0.2%
	SEMAG (Ours)	(+8.0%)	(+38.9%)	(+9.2%)	(+40.8%)

Table 2: Pass@1 accuracy comparison of different methods using GPT-4o (2024-05-13) across multiple benchmarks. The values enclosed in parentheses represent the improvement over the Direct Prompting approach.

Level	Benchmark
Level	HumanEval	MBPP	HumanEval-ET	MBPP-ET	APPS	LiveCode	CodeContests
Level 1	148	314	130	255	66	65	37
Level 2	8	18	6	10	9	16	6
Level 3	4	48	2	46	7	4	5
Level 4	4	120	26	189	57	55	102

Table 3: Distribution of prompt difficulty levels across multiple benchmarks using GPT-4o (2024-05-13).

Figure 3: Pass@1 accuracy on CodeContests using GPT-4o(2024-05-13), GPT-4.1(2025-04-14), DeepSeek-v3(2025-03-24) and Claude-3.7-Sonnet(2025-02-19).

Figure 4: Comparison of Pass@1 accuracy and average token count per question for LPW and SEMAG across benchmarks, using GPT-4o as the LLM backbone. Here,

K=10^{3}

Plan	Refine	Discussion and	Pass@1
Verification	Suggestion	Decision	accuracy
$\times$	$\times$	$\times$	71.9% (-21.4%)
$\checkmark$	$\times$	$\times$	77.4% (-15.4%)
$\times$	$\checkmark$	$\times$	80.5% (-12.0%)
$\times$	$\times$	$\checkmark$	81.7% (-10.7%)
$\times$	$\checkmark$	$\checkmark$	83.5% (-8.7%)

Table 4: Pass@1 accuracy of different component combinations in SEMAG, showing relative decreases from the full implementation (91.5% baseline). Results obtained using GPT-3.5 on the HumanEval benchmark.

With Tool Using	Without Tool Using
91.5%	87.8%

Table 5: Pass@1 accuracy of SEMAG with and without tool usage in the planning stage. Results are obtained using GPT-3.5 on the HumanEval benchmark.

$N_{\text{links}}$	$\Pr(\%)$ $\uparrow$	Tokens ( $K$ ) $\downarrow$	Latency (min) $\downarrow$
10	40.0	30.4	3.5
15	60.0	39.1	4.6
20	80.0	45.9	6.0
25	80.0	65.2	7.8
30	80.0	78.3	9.2

Table 6: Impact of crawl depth on the probability (%) of discovering Claude-3.7-Sonnet in Top-3 and associated resource costs (averaged over five runs, 30-day window).

Figure 5: Pass@1 accuracy (right y-axis) and its variance (left y-axis, scaled by $\times 10^{-4}$ ) on the HumanEval benchmark using GPT-3.5 as the backbone, measured over three independent runs for each temperature setting (0.1 to 1.0).

Figure 6: Pass@1 accuracy on the HumanEval benchmark with GPT-3.5 as the backbone, evaluated under different combinations of

M_{try}

and

M_{debug}

values. Each cell represents the mean Pass@1 accuracy for a specific parameter pair.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Pass@1 accuracy on the APPS benchmark across different difficulty levels, Introductory , Interview , and Competition , of Direct, LDB, LPW and SEMAG, when using GPT-4o as the LLM backbone.

Figure 8: Pass@1 accuracy on the LiveCode benchmark across different difficulty levels, Easy , Medium , and Hard , of Direct, LDB, LPW and SEMAG, when using GPT-4o as the LLM backbone.

Figure 9: The prompt of Planning Agent.

Figure 10: The prompt of Plan verifying Agent.

Figure 11: The prompt of Coding Agent.

Figure 12: The prompt of Adding Trace Agent.

Figure 13: The prompt of Code Explaining Agent.

Figure 14: The prompt of Suggesting Agent.

Figure 15: The prompt of Debugging Agent.

Figure 16: The prompt of Discussing Agent.

Figure 17: The prompt of Discriminating Agent.

Figure 18: The prompt of Code Refining Agent.

Figure 19: The prompt of LLM Selecting Agent.

Figure 20: The prompt of Link Selecting Agent.

Figure 21: The prompt of Content Summarizing Agent.

Figure 22: The prompt of LLM Deciding Agent.

Figure 23: The prompt of Model Matching Agent.

Figure 24: An example of Planning Agent.

Figure 25: An example of Plan Verifying Agent.

Figure 26: An example of Coding Agent.

Figure 27: An example of Embedding Trace Statement Agent.

Figure 28: An example of Code Explaining Agent.

Figure 29: An example of Suggesting Agent.

Figure 30: An example of Debugging Agent.

Figure 31: An example of Debating Agent.

Figure 32: An example of Debating Agent, following Figure 31 .

Figure 33: An example of Discriminating Agent.

Figure 34: An example of Coding Agent (Refine stage).

Agentic Code Generation

Agentic code generation uses AI agents to plan, write, test, and revise executable programs with tools and feedback.

See all 125 papers

Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution
May 14, 2026Han Li, Jinyu Tian, Rili Feng +10RL for Code Generation Continual Learning for LLM Agents
Nanjing University · Independent Researcher · Tsinghua University
CoRe-Code: Collaborative Reinforcement Learning for Code Generation
May 24, 2026Zhihao Dou, Qinjian Zhao, Zhongwei Wan +2RL for Code Generation Multi-Agent Coordination
Department of Computer Science Kean University · Department of Computer Science Case Western Reserve University
Self-Execution Simulation Improves Coding Models
Mar 11, 2026Gallil Maimon, Ori Yoran, Felix Kreuk +4Execution-Guided Code Generation LLM Self-Correction
FAIR team, Meta · Hebrew University of Jerusalem · Inria

May 14, 2026J/K move · Enter open · S save

Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

Han Li, Jinyu Tian, Rili Feng, Yuqiao Du+9

Nanjing University · Independent Researcher · Tsinghua University

RL for Code Generation Continual Learning for LLM Agents Competitive Programming Multi-Agent Coding Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language models (LLMs) still struggle with the rigorous reasoning demands of hard competitive programming. While recent multi-agent frameworks attempt to bridge this reliability gap, they remain fundamentally stateless: they rely on static retrieval and discard the valuable problem-solving and debugging experience gained from previous tasks. To address this, we present Solvita, an agentic evolution framework that enables continuous learning without requiring weight updates to the underlying LLM. Solvita reorganizes problem-solving into a closed-loop system of strategy selection, program synthesis, certified supervision, and targeted hacking, executed by four specialized agents: Planner, Solver, Oracle, and Hacker. Crucially, each agent is paired with a trainable, graph-structured knowledge network. As the system operates, outcome signals, such as pass/fail verdicts, test certification quality, and adversarial vulnerabilities discovered by the Hacker, are recast as reinforcement learning updates to these network weights. This allows the agents to dynamically route future queries based on past successes and failures, effectively accumulating transferable reasoning experience over time. Evaluated across CodeContests, APPS, AetherCode, and live Codeforces rounds, Solvita establishes a new state-of-the-art among code-generation agents, outperforming existing multi-agent pipelines and nearly doubling the accuracy of single-pass baselines.

Abstract

Agentic Code Generation

Explore similar work

Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

CoRe-Code: Collaborative Reinforcement Learning for Code Generation

Self-Execution Simulation Improves Coding Models