Recursive self-improvement (RSI) is one of the most promising paths toward an AI singularity for superintelligence. Achieving this vision requires AI systems to undertake the full training pipeline for iterative improvement. Reinforcement learning (RL) plays an essential role in this cycle to improve model capabilities and generalization, yet automating RL remains difficult due to unstable training dynamics and the need for sustained supervision. We introduce AutoRL, a multi-agent system for automating RL training, as a cornerstone for future RSI. Different from conducting research by end-to-end complete experiments, AutoRL monitors the evolving training state and integrates process-level supervision and research into an ongoing RL training, following the practical workflow of an RL researcher. Specifically, AutoRL consists of three RL-specific agentic roles. Perceptors track training dynamics, rollouts, evaluation, and efficiency, triggering deliberation when they detect anomalies, degradation, or stagnation. Three complementary Proposers diagnose problems and refine candidate interventions through shared evidence and checks. The Orchestrator reviews their recommendation and applies approved changes through online hot updates or checkpoint-based resumption. Experiments on Qwen3 and LongCat-Flash-Lite demonstrate the superiority of AutoRL over static RL recipes and a general agent. The research records show that existing agents can handle unattended RL supervision and dynamic adjustment. The difficulty for future RSI lies in developing fundamental methods that are valuable for resource-consuming validation.
Figures & tables
Figure 1: AutoRL: autonomous RL supervision and research. Right: AutoRL continually adopts interventions, such as fixes and adjustments, to sustain gains over a static recipe.
Figure 2: Framework of AutoRL. Four Perceptors monitor training and diagnose problems. Three specialized Proposers discuss candidates, review evidence, and perform CPU-only checks to guide revision and selection. The Orchestrator reviews the recommendation, authorizes a HOT update or COLD checkpoint-based resumption, and confirms runtime activation. Shared records retain evidence, interventions, and checkpoint lineage.
Model
Method
AIME 24
AIME 25
HMMT 25
Beyond AIME
IMO-AnswerBench
Qwen3-1.7B
Baseline
15.00
13.33
5.00
4.50
10.25
GRPO (Static Recipe)
26.67
21.67
18.33
11.00
8.63
General Agent
31.67
18.33
11.67
12.00
15.00
AutoRL
30.00( +3.33 )
30.00( +8.33 )
18.33(0.00)
12.50( +1.50 )
14.63( +6.00 )
Qwen3-4B
Baseline
25.00
23.33
11.67
10.5
18.44
GRPO (Static Recipe)
46.67
45.00
20.00
26.00
27.00
Table 1: Mathematics benchmark results on Qwen3 models (%). The numbers in parentheses show the performance comparison between AutoRL and GRPO (Static Recipe).
Model
Method
General knowledge
Coding
MMLU-Pro
GPQA-Diamond
MBPP
LiveCodeBench v6 Hard
Qwen3-1.7B
Baseline
38.85
28.79
33.20
1.57
GRPO (Static Recipe)
34.05
22.60
10.95
1.50
General Agent
25.55
32.20
32.00
1.07
AutoRL
44.88( +10.83 )
34.34( +11.74 )
40.05( +29.10 )
2.00( +0.50 )
Qwen3-4B
Baseline
55.88
43.43
62.55
4.93
Table 2: General-knowledge and coding results on Qwen3 models (%). The numbers in parentheses show the performance comparison between AutoRL and GRPO (Static Recipe).
Figure 3: Repairing the training recipe for LongCat-Flash-Lite. AutoRL improves reasoning accuracy as the GRPO (Static Recipe) run deteriorates. Light lines and markers show raw evaluations, and dark curves show a smooth average. Dashed lines mark six representative HOT updates and no restart occurs within the experiment. The star marks the highest raw evaluation accuracy. The upper axis gives elapsed wall-clock time.
Figure 4: Overcoming a performance plateau with Qwen3-4B. Shaded regions indicate initial improvement, the plateau, and renewed improvement. Light lines and markers show raw evaluations, and dark curves show a smooth average. Dashed lines mark applied interventions. Hollow diamonds mark COLD resumptions from checkpoints 100 and 135, with the modifications taking effect at steps 101 and 136. The star marks the highest raw evaluation accuracy. The upper axis gives elapsed wall-clock time.
Figure 5: Recovering from failures during LongCat-Flash-Lite coding-agent training. The curve illustrates evaluations on SWE-related tasks, with a star marking the highest pass@1. Red dashed lines indicate failures. Paired crosses and dots connect the 11 failures to their recovery checkpoints, while the labels give the failure step and restored checkpoint. Two representative repairs are highlighted. The upper axis gives elapsed wall-clock time.
Figure 6: Comparison between AutoRL and general agent on Qwen3-4B. (a) Proposal composition and applied versus reverted interventions. (b) New and repeated training steps. (c) Raw accuracy (light), best-so-far accuracy (dark), and peaks (stars).
Figure 7: AutoRL with different LLM backbones. (a) Online intervention shares over complete runs. (b) Distinct training steps reached. (c) Best accuracy (diamonds) and the mean of the last five evaluations (circles).
Research category
Proposed
Adopted
Representative cases
Parameter tuning
12
9
Entropy coefficients; clipping thresholds and numerical guards.
Grading and data refinement
110
10
Answer-choice binding; reference correction and training-pool revision.
Reward-mechanism extension
5
1
A −0.02 reward adjustment for incorrect looping responses.
Diagnostic instrumentation
1
0
Token-level logging of policy ratios and advantages.
Learning-rule redesign
3
0
Covariance-based gradient selection; positive-advantage and entropy-guided gradient weighting.
Total
131
20
Table 3: Research scope in the GPT-driven Qwen3-4B run. Proposed counts all proposal candidates the agents generated; adopted counts distinct plans deployed after discussion and candidate merging, including subsequently withdrawn changes.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Qwen3-1.7B / 4B
LongCat reasoning
LongCat coding-agent
Training size
17,390
9,340
700+
Prompts per rollout batch
128
128
96
Trajectories per group
8
8
16
Optimizer batch size
256
256
384
Optimization epochs per batch
1
1
1
Initial learning rate
10−6
10−6
5×10−6
Appendix
Table 4: Initial training configurations. A rollout batch contains the listed number of prompts or tasks, each sampled with the indicated group size. Optimizer batch sizes count responses or trajectories. The token budgets for coding-agent trajectories include accumulated interaction content.
Decision, authorized modification, activation record, and updated run lineage
Train Perceptor
Loss, gradients, entropy, KL, ratios, clipping, and finite-value checks
Numerical or optimization diagnosis tied to a run and step window
Rollout Perceptor
TRAIN responses, rewards, truncation, group statistics, and grader inspection
Learning-signal diagnosis with representative training evidence
Eval Perceptor
Fixed-protocol aggregate scores and sanitized failure analysis
Performance trend, plateau, or degradation signal
Efficiency Perceptor
Completed steps, elapsed time, throughput, job status, and runtime logs
Progress or failure report distinguishing waiting from stalled execution
Tuner
Frozen evidence, current control values, source inspection, and CPU checks
Local adjustment with current and proposed values
Appendix
Table 5: Role instructions, evidence, and outputs. Tool access follows each role’s responsibility.
Fixture
Initial hook
Revised hook
Expected
Neither penalty
1.0
1.0
1.0
Repetition only
1.0
0.3
0.3
Truncation only
1.0
1.0
1.0
Both penalties
1.0
0.3
0.3
Appendix
Table 6: CPU replay of the reward-hook fallback. The input score is 1.0 and the mock repetition penalty is 0.7. The intended change removes truncation penalties while preserving repetition penalties.
Measurement
General Agent
AutoRL
First-visit steps
151
273
Revisited steps
122
0
Revisited share
44.7%
0.0%
Checkpoint rewinds
4
0
Applied interventions
6
20
Subsequently reverted interventions
3
4
Appendix
Table 7: Qwen3-4B under a common budget of 273 completed training steps. Applied and reverted interventions use the same budget cutoff. A reversion withdraws a deployed change and can occur without rewinding a checkpoint.
Category
General Agent
AutoRL
Optimization
7
12
Gradient design
4
3
Reward shaping
0
5
Answer parsing
0
54
Reward verification
0
15
Reference repair
1
36
Appendix
Table 8: Initial proposal categories over the recorded research histories. Categories describe the target mechanism; counts include proposed changes that were not executed.
Step
Applied change
3
Repair multiple-choice option mapping and multipart-answer extraction.
6
Split compound references and compare multivalue answers as multisets.
10
Improve LaTeX and array parsing and normalize degree notation.
22
Introduce KL regularization with coefficient 0.001.
27
Increase the KL coefficient to 0.01.
38
Normalize whitespace and handle units for numeric-only references.
Appendix
Table 9: Selected LongCat reasoning interventions. Steps indicate runtime activation. All grader changes apply to TRAIN; the evaluation grader remains fixed.
Step
Mode
Applied change
101
COLD
Repair TRAIN answer-option binding after checkpoint 100.
108
HOT
Set the entropy coefficient to 0.001.
111
HOT
Apply an auxiliary reward of −0.02 to already incorrect responses satisfying a strict looping criterion.
121
HOT
Increase the entropy coefficient to 0.002.
129
HOT
Increase the entropy coefficient to 0.003.
136
COLD
Repair two TRAIN references after checkpoint 135.
Appendix
Table 10: Selected Qwen3-4B interventions around the plateau. COLD changes resume from the immediately preceding checkpoint shown in the text; HOT changes retain the running state.
ID
Failure time
Symptom
Step
Minutes
Recovery path
F1
09-24 21:05
NaN
9→8
38.9
Checkpoint restart
F2
09-26 12:12
NaN
18→17
39.1
Checkpoint restart
F3
09-27 00:14
NaN
19→18
59.1
Agent: new job
F4
09-27 06:10
NaN
18→18
36.6
Checkpoint restart
F5
09-28 07:11
NaN
22→22
295.1
Agent: ratio protection
F6
09-28 18:18
NaN
23→22
106.5
Agent: LR/precision
Appendix
Table 11: Failure and recovery log for LongCat coding-agent training. Times are UTC+8 in 2026. The arrow gives completed step → restored checkpoint. Latency ends at the resumption shown in the figure.
Recursive self-improvement (RSI) of a model on non-verifiable tasks, such as open-ended research, faces a supervision bottleneck when its outputs exceed what even human experts can reliably assess, leaving the model itself (optimizee) as the best available optimizer and evaluator. However, a single model instance struggles to critique and improve its own complex reasoning under this homogeneous loop. To address this, we propose Multi-Agent Self-Supervision (MASS), an RSI method that alternates between evolutionary workflow optimization and supervised fine-tuning on self-generated trajectories. Guided by early findings that multi-agent topologies excel at complex reasoning, MASS prompts a single base model to iteratively propose, execute, and self-evaluate multi-agent workflows. Through an evolutionary search constrained by structural guardrails, the model optimizes these computational-graph-like orchestrations, discovering the most effective distinct roles and information routing for a given task. Over two MASS cycles with Qwen3.6-27B, the model achieves 1.2-1.6x higher performance per output tokens on four open-ended public benchmarks. Because the improved model subsequently acts as a better optimizer and evaluator, this alternating framework enables a continuous, recursive bootstrapping of the model's capabilities. Moreover, multi-agent traces are also more training-efficient: a student trained on them outperforms a single-agent student trained on 1.4x more training tokens. These findings suggest that jointly learning orchestration and bounded subagent execution from multi-agent trajectories can provide an effective signal for RSI.
Recursive self-improvement (RSI) seeks to enable AI systems to participate in improving their own capabilities. A concrete pathway is autonomous model development, where agents iteratively explore post-training strategies to improve a base model. This setting faces two challenges: agents may exploit open-ended experimental actions through hacking, and repeated experimentation may lead to strategy lock-in, where an early direction is refined rather than reconsidered. We introduce RSI-Master, which addresses the two challenges at two levels: regularize step-wise actions, avoiding hacking behaviors, and promote well-structured exploration of research directions, avoiding strategy lock-in. RSI-Master consists of an Experiment OS, which enables regularized experimental actions and maintains persistent, traceable experimental records, and Reviewer-Guided Research Orchestration, which organizes Workers and Reviewers in a dynamically growing research DAG. Workers explore diverse research directions and Reviewers compare evidence across related experiments for subsequent explorations. On PostTrainBench with Qwen3-4B-Base, it averages 54.49 versus 46.53 for the strongest agent baseline, with a 0.0% hacking rate. Scaling to 35B model, RSI-Master surpasses the human-developed Instruct model on LiveCodeBench-v6 (41.21 vs. 37.36) and SciCode, and reaches a nonzero score on HorizonMath, a benchmark of unsolved research problems on which most frontier models score near zero.
Yaxin Du, Xiyuan Yang, Zhifan Zhou +10
Shanghai Jiao Tong University · Carnegie Mellon University · University of Waterloo
Recursive self-improvement (RSI) envisions agent systems that autonomously improve their own models and harnesses, with each updated system continuing to improve itself. Existing approaches typically restrict optimization objectives, editable components, or evaluation procedures, limiting open-ended exploration and continuity across generations. We introduce RSI-Env, a persistent environment for coordinated model-harness improvement, supporting autonomous training, harness modification, and retention of research state. An independent Host Controller manages services, evaluation, and versioning. Under Self, each accepted model-harness version takes over the next improvement round. We define three settings that vary goal specificity and access to benchmark feedback: Open-General, Capability-Directed, and Benchmark-Guided. Across these three settings, we compare Self with two fixed external Teachers on seven tasks, starting from Qwen3.8-27B with a minimal harness. Without benchmark feedback, Open-General Self raises GPQA Diamond accuracy from 65.2% to 73.2% and AIME 2026 accuracy from 53.3% to 56.7%. All three Capability-Directed Self runs achieve target-aligned gains. Knowledge Self demonstrates continued improvement across system updates, with SimpleQA accuracy rising from 82.0% to 85.0% and then 88.0%. Benchmark guidance and Teacher benefits vary by task. These results demonstrate RSI-Env's support for open-ended exploration and sustained recursive improvement within the studied runs, while highlighting challenges in preserving gains across capabilities.
Wanying Qu, Zhiyuan Fan, Jiyao Liu +4
Fudan University · The Hong Kong University of Science and Technology · Georgia Institute of Technology +3