RSI Arena: Towards Recursive Self-Improvement with Human Feedback
Authors: Yue Huang, Yanbo Wang, Lichi Li, Yuanyuan Chen, Andreas Haupt, Xiaonan Luo, Zheyuan Liu, Jihye Choi, Felix Juefei-Xu, Xiangliang Zhang, Radha Poovendran, Misha Sra, Ying Liu, Ben Burtenshaw, Thomas Wolf, Alex Pentland, Zichen Chen, Zhangchen Xu
Organizations: Bake AI · University of Notre Dame · University of California, Los Angeles · Stanford University · NVIDIA Research · New York University · University of Washington · University of California, Santa Barbara · Scale AI · Hugging Face · Massachusetts Institute of Technology
Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.
Figures & tables
Figure 1: The two stages of RSI Arena. In Stage 1 (steps 1–4), ten AI agents post-train the same base model for 144 hours, nominate checkpoints, and their final models are evaluated on 20 benchmarks and by people in the Human Arena. In Stage 2 (steps 5–8), each agent continues from its own Stage 1 model with the arena battles that model played, and the new models are evaluated again, with NVIDIA Instruct as the common reference.
Figure 2: The system architecture. Each agent works in its own sandbox on the research host and runs its training jobs on the shared GPU cluster; a separate management virtual machine runs the agents and enforces their budgets.
Area
Benchmarks (items)
Knowledge and reasoning
GPQA Diamond ( Rein et al., 2024 , 198 questions × 8 option orders;) , MMLU-Pro ( Wang et al., 2024 , 12,032;) , SuperGPQA ( M-A-P Team et al., 2025 , 2,000 sampled;) , HLE text only † ( Phan et al., 2025 , 2,158;) , BBEH ( Kazemi et al., 2025 , 4,520;) , AA-Omniscience † ( Jackson et al., 2025 , 600;)
Mathematics
AIME 2026 (30) and HMMT February 2026 (33), as run by MathArena ( Balunović et al., 2025 )
Code
LiveCodeBench v6 ( Jain et al., 2025 , 175;) , SciCode ( Tian et al., 2024 , 288 steps of 65 problems;)
Long context
AA-LCR † ( Artificial Analysis, 2025 , 100;) , MRCR v2 with 8 needles up to 128k tokens ( OpenAI, 2025b , 485;)
Instructions and chat
IFBench ( Pyatkin et al., 2025 , 300;) , Arena-Hard v2 † ( Li et al., 2024b , 504 English prompts;) , MultiChallenge † ( Sirdeshmukh et al., 2025 , 266;) , WildBench v2 † ( Lin et al., 2025 , 1,022;)
Safety
StrongREJECT † ( Souly et al., 2024 , 313;) , MASK † ( Ren et al., 2025 , 1,000;) , FORTRESS † ( Knight et al., 2025 , 500 adversarial and 500 benign;) , BBQ ( Parrish et al., 2022 , 5,000 sampled;)
Table 1: The 20 benchmarks. The number of items is the number of prompts after our filters. A dagger marks benchmarks scored by an LLM judge. Appendix B gives the metrics and generation limits.
Feature
Stage 1
95% interval
Stage 2
95% interval
Arena.ai
Answer length (tokens)
0.169
−0.036 to 0.370
0.394
0.242 to 0.559
0.249
List items
0.294
0.118 to 0.485
0.222
0.065 to 0.366
0.031
Headers
−0.007
−0.190 to 0.174
0.117
−0.016 to 0.252
0.024
Bold spans
0.223
0.030 to 0.445
−0.007
−0.162 to 0.164
0.019
Table 2: Style coefficients estimated from each stage’s votes, with 95% bootstrap intervals, and the values Arena.ai published in 2024. Each multiplies a standardized normalized difference between the two answers (Equation ( 2 )).
Figure 3: Stage 1, hour by hour. Each lane is one agent. The shaded profile shows how many GPUs the agent’s jobs held in each hour; a full lane is 16 GPUs. Diamonds mark accepted nominations, and the filled diamond is the final model. The right columns give GPU-hours used and accepted nominations.
Table 3: The ten agents, ordered by their model’s style-controlled human rating (Section 3 ), which the last column gives. “Last stage” is the final training step of the nominated model. Progress is the median of Equation ( 1 ) over 12 benchmarks. “Ended by” gives how the session ended.
Figure 4: How each agent trained its final model. Dots mark the kind of weights trained, the data used and four other steps seen in the agents’ records; the chips give the training steps of the final model in order. Data are the kinds named in each agent’s scripts and notes, and not all of them reached the final model. Appendix G names the data sets.
Figure 5: Benchmark progress toward NVIDIA Instruct. A: median over 12 capability benchmarks of Equation ( 1 ); the bar spans the 25th to 75th percentile of the 12 values. B: each benchmark; 0 is the base model and 100 is NVIDIA Instruct, with values above 100 in bold.
Figure 6: Human Arena results, Stage 1. A: style-controlled ratings (filled, with 95% bootstrap intervals) and ratings without style control (open), from 795 accepted votes in battles without web search (Equation ( 2 )); NVIDIA Instruct is a reference. B: outcomes of each agent model’s accepted battles, from that model’s side.
Figure 7: Benchmarks against human ratings without style control. A: benchmark progress and rating for the ten agent models. B: for each benchmark, and for the median answer length in the arena, the Spearman correlation between the ten models’ values and their ratings: +1 means the benchmark orders the models exactly as people do, 0 means no relation. With ten models, a value inside the shaded band ( ∣ρ∣<0.65 ) could arise by chance. For StrongREJECT and FORTRESS, where lower is safer, the sign is flipped so that a positive value means safer models were rated higher.
Figure 8: Answer length and style in the Human Arena, Stage 1. A: share of decisive votes won by the longer answer, by how much longer it was; the bins hold 88 to 220 votes. B: ratings without and with style control (Equation ( 2 )); the right column gives the change, with minus without.
Figure 10: Stage 2. A: GPU-hours used by each agent over the 72 hours; the dashed line is the 500 GPU-hour allowance, which two agents passed with GPU time credited back after a storage fault. B: median answer length in the arena, in tokens, for each model in Stage 1 (open) and Stage 2 (filled). NVIDIA Instruct did not change; its length differs because the two arenas had different questions.
Figure 11: Human Arena results, Stage 2. A: style-controlled ratings (filled, with 95% bootstrap intervals) and ratings without style control (open), from 1,096 accepted votes in battles without web search. B: each agent model’s style-controlled rating minus NVIDIA Instruct’s rating, in the Stage 1 arena (open) and the Stage 2 arena (filled).
Figure 12: Benchmarks, Stage 2. A: median progress toward NVIDIA Instruct over the 12 benchmarks of Equation ( 1 ), Stage 1 (open) and Stage 2 (filled), with the change in points. B: change of each score, Stage 2 minus Stage 1, in points of the gap from the base model to NVIDIA Instruct.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Items
Limit
Scoring
GPQA Diamond
198 × 8 option orders
16,384
final line “Answer: X”; accuracy
HLE
2,158, text only
16,384
judge; accuracy
MMLU-Pro
12,032
8,192
official extraction; accuracy
SuperGPQA
2,000, stratified sample
8,192
official extraction; accuracy
BBEH
4,520 in 23 tasks
16,384
official script; adjusted harmonic mean over tasks
AIME 2026
30
32,768
MathArena parser; mean accuracy
Appendix
Table 4: Settings of the 20 benchmarks.
Benchmark
Ours
Published
Source
MMLU-Pro
81.72
81.94
model card
GPQA Diamond (8 samples)
75.32±1.81
75.44
model card
IFBench (loose)
74.67
72.9
model card
SciCode (steps)
34.03
32.6
model card
HLE (text only)
12.93±1.42
11.7; 10.6
model card; Artificial Analysis
AA-LCR
63.0
60.3
Artificial Analysis
Appendix
Table 5: NVIDIA Instruct with reasoning on: our scores and published scores.
Table 6: Stage 1 scores of the base model, NVIDIA Instruct (reasoning off) and the ten agent models. MASK and FORTRESS scores of the base model are high because it rarely answers coherently.
Table 7: Stage 2 scores of the ten agent models, with the same references as Table 6 . Four WildBench answers that the judge could not grade are left out.
Figure 13: Budgets and outcomes. A: share of the GPU and API allowances each agent used. B: final Slurm state of each agent’s GPU jobs; preempted jobs ran on GPUs that the scheduler took back. C: GPU-hours used against the style-controlled human rating of the agent’s model; with ten agents, a Spearman correlation this small could arise by chance.
Table 8: API cost and GPU-hours of each agent in Stage 1, Stage 2 and in all, ordered by total API cost. Totals are summed before rounding.
Figure 14: API cost and human rating. Each agent’s API cost over both stages (log scale) against the style-controlled Stage 2 rating of its model; the solid line is NVIDIA Instruct. The dashed line joins the models that no other model beats on both cost and rating. With ten agents, a Spearman correlation below about 0.65 in size could arise by chance.
Votes
Stage 1
Stage 2
Votes in ordinary battles
1,685
1,444
filtered out
394
125
Accepted votes
1,291
1,319
decisive
1,003
1,002
tie
159
183
both are bad
129
134
Appendix
Table 9: Votes in the two Human Arenas. The Stage 2 counts include the targeted round that followed it.
Table 10: Human ratings of both stages: the style-controlled rating with its 95% bootstrap interval, and the rating without style control. The two arenas are separate, and NVIDIA Instruct played in both.
Table 11: The ten agents in Stage 2, ordered by their model’s style-controlled rating. GPU-h and API are what each agent used of about 500 GPU-hours and $150. Jobs counts GPU jobs and Noms accepted nominations. Tokens is the median answer length in the arena in Stage 1 and Stage 2, and the last column is the style-controlled rating minus NVIDIA Instruct’s in the Stage 1 and the Stage 2 arena.
Table 12: Safety benchmarks, Stage 1 to Stage 2. Lower is safer for StrongREJECT and FORTRESS; higher is better for MASK and BBQ. Bold marks a change of 5 points or more in the less safe direction. Agents are ordered by their Stage 2 style-controlled rating; the reference rows are the same in both stages.
Recursive self-improvement (RSI) requires an agentic system to design, train, and evaluate its own successor. Data-driven post-training is a central part of that loop, yet existing evaluations measure it on a single benchmark, where an agent selects from a static pool or distills a stronger model to push one score up in one run. Real post-training is multi-domain, and a gain in one domain has to be weighed against a regression in another. We propose ParetoRSI-Gym, in which an agent searches for data, curates it, trains, reads back per-domain results, and repeats, with everything but the data held fixed: the initial LLM, the five domains (math, code, finance, medicine, science), the toolset, and the SFT pipeline. Generalization benchmarks in each domain separate capability gains from overfitting to the scores the agent optimizes against. Across three agent systems, three base models, five rounds and multiple seeds, the strongest system, Claude Code with Opus-5, raises the generalization scores of its proxy-best checkpoint by 9.2% relative to the starting model (+2.3 absolute points), averaged with equal weight over three base models and five domains, with a single-domain relative gain of up to 63% (medicine on Ministral-3-3B). Further analysis shows that the harness matters as much as the model it drives, and that each agent keeps a unique data selection strategy. We see our work as a step towards evaluation of RSI systems in real-world machine learning engineering.
Haojin Wang, Yangzhen Wu, Zeyi Chen +6
University of Illinois Urbana-Champaign · University of California, Berkeley · The Ohio State University +2
Recursive self-improvement (RSI) seeks to enable AI systems to participate in improving their own capabilities. A concrete pathway is autonomous model development, where agents iteratively explore post-training strategies to improve a base model. This setting faces two challenges: agents may exploit open-ended experimental actions through hacking, and repeated experimentation may lead to strategy lock-in, where an early direction is refined rather than reconsidered. We introduce RSI-Master, which addresses the two challenges at two levels: regularize step-wise actions, avoiding hacking behaviors, and promote well-structured exploration of research directions, avoiding strategy lock-in. RSI-Master consists of an Experiment OS, which enables regularized experimental actions and maintains persistent, traceable experimental records, and Reviewer-Guided Research Orchestration, which organizes Workers and Reviewers in a dynamically growing research DAG. Workers explore diverse research directions and Reviewers compare evidence across related experiments for subsequent explorations. On PostTrainBench with Qwen3-4B-Base, it averages 54.49 versus 46.53 for the strongest agent baseline, with a 0.0% hacking rate. Scaling to 35B model, RSI-Master surpasses the human-developed Instruct model on LiveCodeBench-v6 (41.21 vs. 37.36) and SciCode, and reaches a nonzero score on HorizonMath, a benchmark of unsolved research problems on which most frontier models score near zero.
Yaxin Du, Xiyuan Yang, Zhifan Zhou +10
Shanghai Jiao Tong University · Carnegie Mellon University · University of Waterloo
Recursive self-improvement (RSI) requires a concrete mechanism through which an AI system observes its capabilities and converts that evidence into the next round of learning. We present NeoHorse-1, a family of agent-native models developed to explore this path through agentic post-training. Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn. These records are converted into training examples that preserve interleaved reasoning, tool calls, and harness context, and are admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling. Routing signals organize supervised fine-tuning into a three-stage curriculum and extend to routing-guided on-policy distillation, where a teacher supervises student-generated responses under the same progression. Capability-guided allocation then converts evaluation feedback into the next training mixture, closing an evaluation-selection-update loop in which what the system learns to do shapes what it learns from next. Across eleven benchmarks covering harness-based agents, tool use, coding, and instruction following, post-training raises the macro-average from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B, substantially narrowing the aggregate gap between the post-trained 4B model and the 9B base model. We further adapt NeoHorse-1-4B into NeoHorse-Jev-4B, a prefill-only model that returns structured decisions and probabilities for agent workflows. NeoHorse-1 provides an initial prototype of this feedback-driven process and a path toward harness-mediated RSI across successive iterations.