SAGE: Multi-Agent Self-Evolution for LLM Reasoning
Authors: Yulin Peng, Xinxin Zhu, Chenxing Wei, Nianbo Zeng, Leilei Wang, Ying Tiffany He, F. Richard Yu
Organizations: College of Computer Science and Software Engineering, Shenzhen University, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China · School of Information Technology, Carleton University, Canada
Reinforcement learning with verifiable rewards improves reasoning in large language models (LLMs), but many methods still rely on large human-labeled datasets. While self-play reduces this dependency, it often lacks explicit planning and strong quality control, limiting stability in long-horizon multi-step reasoning. We present SAGE (Self-evolving Agents for Generalized reasoning Evolution), a closed-loop framework where four agents: Challenger, Planner, Solver, and Critic, co-evolve from a shared LLM backbone using only a small seed set. The Challenger continuously generates increasingly difficult tasks; the Planner converts each task into a structured multi-step plan; and the Solver follows the plan to produce an answer, whose correctness is determined by external verifiers. The Critic scores and filters both generated questions and plans to prevent curriculum drift and maintain training signal quality, enabling stable self-training. Across mathematics and code-generation benchmarks, SAGE delivers consistent gains across model scales, improving the Qwen-2.5-7B model by 8.9% on LiveCodeBench and 10.7% on OlympiadBench.
Figures & tables
Figure 1: Overview of the SAGE framework. Four specialized agents—Challenger, Planner, Solver, and Critic—interact through quality filtering and format validation to enable closed-loop self-evolution.
Figure 2: The SAGE training pipeline. (1) The Challenger generates questions from reference examples, filtered by the Critic for quality; (2) verified questions expand the dataset; (3) sampled questions are processed by the Planner and Solver to produce solutions; (4) all agents are jointly updated using Task-Relative REINFORCE++ with per-role advantage normalization.
Method
HEval+
MBPP+
LCB v1-5
GSM8K
Math
AI24
AI25
AMC
Olympiad
C Avg.
M Avg.
O Avg.
Qwen - 2.5 - 3B-Instruct
Base Model
68.3
60.6
12.0
84.6
60.4
3.3
6.7
40.0
28.0
46.9
37.2
40.4
AZR
68.9
61.4
15.0
81.2
62.4
3.3
3.3
35.0
28.9
48.4
35.7
39.9
MAE
68.3
61.1
15.9
82.2
65.8
3.3
3.3
32.5
32.5
48.4
36.6
40.5
SAGE
68.9
62.4
16.9
85.5
66.2
6.7
6.7
35.0
29.8
49.4
38.3
42.0
Qwen-2.5-7B-Instruct
Table 1: Main results on reasoning benchmarks. Comparison of post-training methods across three model scales. We report pass@1 accuracy (%) on code generation (HumanEval+, MBPP+, LiveCodeBench) and mathematical reasoning (GSM8K, MATH, AIME 2024, AIME 2025, AMC, and OlympiadBench). C Avg., M Avg., and O Avg. denote the mean scores over code, math, and all benchmarks. SAGE achieves the best overall performance across all three model backbones. Bold indicates best per LLM backbone.
Backbone
Method
ID Avg.
OOD Avg.
Qwen-2.5-3B
Base Model
68.4
18.0
AZR
68.5
17.1
MAE
69.4
17.5
SAGE
70.8
19.0
Qwen-2.5-7B
Base Model
76.3
24.6
AZR
77.4
27.4
Table 2: ID and OOD generalization comparison. SAGE consistently improves OOD performance (+4.2% on 7B) without sacrificing in-distribution accuracy.
Method
HEval+
MBPP+
LCB v1-5
GSM8K
Math
AI24
AI25
AMC
Olympiad
C Avg.
M Avg.
O Avg.
SAGE (full implementation)
68.9
62.4
16.9
85.5
66.2
6.7
6.7
35.0
29.8
49.4
38.3
42.0
SAGE (w/o challenger training)
66.5
61.3
9.0
86.7
65.5
0.0
3.3
35.0
28.0
45.6
36.4
39.5
SAGE (w/o solver training)
67.7
64.3
9.0
81.2
60.4
3.3
0.0
30.0
28.0
47.0
33.8
38.2
SAGE (w/o critic training)
66.5
53.7
14.1
86.0
65.9
3.3
6.7
40.0
27.4
44.8
38.2
40.4
Table 3: Ablation study of SAGE components on Qwen-2.5-3B. We evaluate the impact of removing individual agent training while keeping other components active.
Figure 3: Training dynamics on Qwen-2.5-3B. The Challenger steadily expands the question pool (bars) throughout training, while validation accuracy (line) reaches peak performance around step 100–120 before gradual decline, suggesting potential over-specialization on the self-generated curriculum.
Figure 4: Qualitative case study. The Challenger generates a math word problem, the Planner decomposes it into structured steps, the Solver executes the plan to produce the final answer, and the Critic provides quality scores for both the question and the plan.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training Configuration
Batch Size
128
Learning Rate
3×10−6
Training Steps
200
Generation Settings
Maximum Prompt Length
8192
Appendix
Table 4: Training Hyperparameters of our experiments.
Benchmark
Count
MATH ( Hendrycks et al., 2021a )
156
GSM8K ( Cobbe et al., 2021 )
148
HumanEval ( Chen et al., 2021 )
87
MBPP ( Austin et al., 2021 )
109
Total
500
Appendix
Table 5: Distribution of Training Samples Across Benchmarks
Figure 5: The prompt of the Challenger Agent.
Figure 6: The prompt of the planner Agent.
Figure 7: The prompt of the Solver Agent.
Figure 8: The prompt of the Critic Agent(question).
Figure 9: The prompt of the Critic Agent(plan).
Figure 10: The prompt of the Critic Agent(answer).