cs.CLOct 24, 2026
SaveRSI Arena: Towards Recursive Self-Improvement with Human Feedback
Organizations: Bake AI · University of Notre Dame · University of California, Los Angeles · Stanford University ·
Abstract
Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.
Figures & tables
Figure 1: The two stages of RSI Arena. In Stage 1 (steps 1–4), ten AI agents post-train the same base model for 144 hours, nominate checkpoints, and their final models are evaluated on 20 benchmarks and by people in the Human Arena. In Stage 2 (steps 5–8), each agent continues from its own Stage 1 model with the arena battles that model played, and the new models are evaluated again, with NVIDIA Instruct as the common reference.
Figure 2: The system architecture. Each agent works in its own sandbox on the research host and runs its training jobs on the shared GPU cluster; a separate management virtual machine runs the agents and enforces their budgets.
| Area | Benchmarks (items) |
|---|---|
| Knowledge and reasoning | GPQA Diamond ( Rein et al., 2024 , 198 questions 8 option orders;) , MMLU-Pro ( Wang et al., 2024 , 12,032;) , SuperGPQA ( M-A-P Team et al., 2025 , 2,000 sampled;) , HLE text only † ( Phan et al., 2025 , 2,158;) , BBEH ( Kazemi et al., 2025 , 4,520;) , AA-Omniscience † ( Jackson et al., 2025 , 600;) |
| Mathematics | AIME 2026 (30) and HMMT February 2026 (33), as run by MathArena ( Balunović et al., 2025 ) |
| Code | LiveCodeBench v6 ( Jain et al., 2025 , 175;) , SciCode ( Tian et al., 2024 , 288 steps of 65 problems;) |
| Long context | AA-LCR † ( Artificial Analysis, 2025 , 100;) , MRCR v2 with 8 needles up to 128k tokens ( OpenAI, 2025b , 485;) |
| Instructions and chat | IFBench ( Pyatkin et al., 2025 , 300;) , Arena-Hard v2 † ( Li et al., 2024b , 504 English prompts;) , MultiChallenge † ( Sirdeshmukh et al., 2025 , 266;) , WildBench v2 † ( Lin et al., 2025 , 1,022;) |
| Safety | StrongREJECT † ( Souly et al., 2024 , 313;) , MASK † ( Ren et al., 2025 , 1,000;) , FORTRESS † ( Knight et al., 2025 , 500 adversarial and 500 benign;) , BBQ ( Parrish et al., 2022 , 5,000 sampled;) |
Table 1: The 20 benchmarks. The number of items is the number of prompts after our filters. A dagger marks benchmarks scored by an LLM judge. Appendix B gives the metrics and generation limits.
| Feature | Stage 1 | 95% interval | Stage 2 | 95% interval | Arena.ai |
|---|---|---|---|---|---|
| Answer length (tokens) | 0.169 | −0.036 to 0.370 | 0.394 | 0.242 to 0.559 | 0.249 |
| List items | 0.294 | 0.118 to 0.485 | 0.222 | 0.065 to 0.366 | 0.031 |
| Headers | −0.007 | −0.190 to 0.174 | 0.117 | −0.016 to 0.252 | 0.024 |
| Bold spans | 0.223 | 0.030 to 0.445 | −0.007 | −0.162 to 0.164 | 0.019 |
Table 2: Style coefficients estimated from each stage’s votes, with 95% bootstrap intervals, and the values Arena.ai published in 2024. Each multiplies a standardized normalized difference between the two answers (Equation ( 2 )).
Figure 3: Stage 1, hour by hour. Each lane is one agent. The shaded profile shows how many GPUs the agent’s jobs held in each hour; a full lane is 16 GPUs. Diamonds mark accepted nominations, and the filled diamond is the final model. The right columns give GPU-hours used and accepted nominations.
Table 3: The ten agents, ordered by their model’s style-controlled human rating (Section 3 ), which the last column gives. “Last stage” is the final training step of the nominated model. Progress is the median of Equation ( 1 ) over 12 benchmarks. “Ended by” gives how the session ended.
Figure 4: How each agent trained its final model. Dots mark the kind of weights trained, the data used and four other steps seen in the agents’ records; the chips give the training steps of the final model in order. Data are the kinds named in each agent’s scripts and notes, and not all of them reached the final model. Appendix G names the data sets.
Figure 5: Benchmark progress toward NVIDIA Instruct. A: median over 12 capability benchmarks of Equation ( 1 ); the bar spans the 25th to 75th percentile of the 12 values. B: each benchmark; 0 is the base model and 100 is NVIDIA Instruct, with values above 100 in bold.
Figure 6: Human Arena results, Stage 1. A: style-controlled ratings (filled, with 95% bootstrap intervals) and ratings without style control (open), from 795 accepted votes in battles without web search (Equation ( 2 )); NVIDIA Instruct is a reference. B: outcomes of each agent model’s accepted battles, from that model’s side.
Figure 7: Benchmarks against human ratings without style control. A: benchmark progress and rating for the ten agent models. B: for each benchmark, and for the median answer length in the arena, the Spearman correlation between the ten models’ values and their ratings: means the benchmark orders the models exactly as people do, 0 means no relation. With ten models, a value inside the shaded band ( ) could arise by chance. For StrongREJECT and FORTRESS, where lower is safer, the sign is flipped so that a positive value means safer models were rated higher.
Figure 8: Answer length and style in the Human Arena, Stage 1. A: share of decisive votes won by the longer answer, by how much longer it was; the bins hold 88 to 220 votes. B: ratings without and with style control (Equation ( 2 )); the right column gives the change, with minus without.
Figure 10: Stage 2. A: GPU-hours used by each agent over the 72 hours; the dashed line is the 500 GPU-hour allowance, which two agents passed with GPU time credited back after a storage fault. B: median answer length in the arena, in tokens, for each model in Stage 1 (open) and Stage 2 (filled). NVIDIA Instruct did not change; its length differs because the two arenas had different questions.
Figure 11: Human Arena results, Stage 2. A: style-controlled ratings (filled, with 95% bootstrap intervals) and ratings without style control (open), from 1,096 accepted votes in battles without web search. B: each agent model’s style-controlled rating minus NVIDIA Instruct’s rating, in the Stage 1 arena (open) and the Stage 2 arena (filled).
Figure 12: Benchmarks, Stage 2. A: median progress toward NVIDIA Instruct over the 12 benchmarks of Equation ( 1 ), Stage 1 (open) and Stage 2 (filled), with the change in points. B: change of each score, Stage 2 minus Stage 1, in points of the gap from the base model to NVIDIA Instruct.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Items | Limit | Scoring |
|---|---|---|---|
| GPQA Diamond | 198 8 option orders | 16,384 | final line “Answer: X”; accuracy |
| HLE | 2,158, text only | 16,384 | judge; accuracy |
| MMLU-Pro | 12,032 | 8,192 | official extraction; accuracy |
| SuperGPQA | 2,000, stratified sample | 8,192 | official extraction; accuracy |
| BBEH | 4,520 in 23 tasks | 16,384 | official script; adjusted harmonic mean over tasks |
| AIME 2026 | 30 | 32,768 | MathArena parser; mean accuracy |
Table 4: Settings of the 20 benchmarks.
| Benchmark | Ours | Published | Source |
|---|---|---|---|
| MMLU-Pro | 81.72 | 81.94 | model card |
| GPQA Diamond (8 samples) | 75.44 | model card | |
| IFBench (loose) | 74.67 | 72.9 | model card |
| SciCode (steps) | 34.03 | 32.6 | model card |
| HLE (text only) | 11.7; 10.6 | model card; Artificial Analysis | |
| AA-LCR | 63.0 | 60.3 | Artificial Analysis |
Table 5: NVIDIA Instruct with reasoning on: our scores and published scores.
Table 6: Stage 1 scores of the base model, NVIDIA Instruct (reasoning off) and the ten agent models. MASK and FORTRESS scores of the base model are high because it rarely answers coherently.
Table 7: Stage 2 scores of the ten agent models, with the same references as Table 6 . Four WildBench answers that the judge could not grade are left out.
Figure 13: Budgets and outcomes. A: share of the GPU and API allowances each agent used. B: final Slurm state of each agent’s GPU jobs; preempted jobs ran on GPUs that the scheduler took back. C: GPU-hours used against the style-controlled human rating of the agent’s model; with ten agents, a Spearman correlation this small could arise by chance.
Table 8: API cost and GPU-hours of each agent in Stage 1, Stage 2 and in all, ordered by total API cost. Totals are summed before rounding.
Figure 14: API cost and human rating. Each agent’s API cost over both stages (log scale) against the style-controlled Stage 2 rating of its model; the solid line is NVIDIA Instruct. The dashed line joins the models that no other model beats on both cost and rating. With ten agents, a Spearman correlation below about 0.65 in size could arise by chance.
| Votes | Stage 1 | Stage 2 |
|---|---|---|
| Votes in ordinary battles | 1,685 | 1,444 |
| filtered out | 394 | 125 |
| Accepted votes | 1,291 | 1,319 |
| decisive | 1,003 | 1,002 |
| tie | 159 | 183 |
| both are bad | 129 | 134 |
Table 9: Votes in the two Human Arenas. The Stage 2 counts include the targeted round that followed it.
Table 10: Human ratings of both stages: the style-controlled rating with its 95% bootstrap interval, and the rating without style control. The two arenas are separate, and NVIDIA Instruct played in both.
Table 11: The ten agents in Stage 2, ordered by their model’s style-controlled rating. GPU-h and API are what each agent used of about 500 GPU-hours and $150. Jobs counts GPU jobs and Noms accepted nominations. Tokens is the median answer length in the arena in Stage 1 and Stage 2, and the last column is the style-controlled rating minus NVIDIA Instruct’s in the Stage 1 and the Stage 2 arena.
Table 12: Safety benchmarks, Stage 1 to Stage 2. Lower is safer for StrongREJECT and FORTRESS; higher is better for MASK and BBQ. Bold marks a change of 5 points or more in the less safe direction. Agents are ordered by their Stage 2 style-controlled rating; the reference rows are the same in both stages.
More in this topic
LLM Agent Self-Improvement
Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.
See all 276 papers