Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at https://github.com/GAIR-NLP/801-30B .
Figures & tables
Figure 1: Left: recursive self-improvement (RSI) takes a different form in pre-training than in post-training. Right: 801-30B-A3B (red) against open baselines on base-model benchmarks (top) and after supervised fine-tuning (bottom).
Post-training headroom, not readable from any single score
Data
A fixed dataset or environment
Built from raw sources: 6 pipelines, > 30 stages, 1,053 subsets, 10.5T unique tokens
Infrastructure
A training script
> 130 code versions, 106 hyperparameter fields
Table 1: RSI for Post-training vs. RSI for frontier pre-training operate in different regimes.
Figure 2: The standard RSI loop ( propose→run→measure→update ) does not close in frontier pre-training: a run takes months and cannot be repeated, and today’s score does not reflect the headroom a checkpoint leaves for post-training. We propose three types of loops, instantiated as six practical loops. Debugging loops comprise the infra debugging loop and the data debugging loop, which trace internal-metric and output anomalies to their causes. The headroom loop is instantiated as the recipe loop, using headroom measurements to guide the whole pre-training process. Experience-to-metric loops comprise the classifier development loop, the site discovery loop, and the crawl-coverage loop, turning experience-based judgment into measurable feedback. All six loops share a context infrastructure that integrates evidence across teams at scale.
Loop type
Propose
Run
Measure
Update
Debugging
A suspected cause: code version, configuration, or data subset
Targeted diagnosis on the infra or data stack; cheap and repeatable
Does the anomaly disappear?
Apply the fix; record ruled-out causes
Headroom
A recipe change, or whether to continue training
Pilots and main-run checkpoints; costly and not repeatable
Tiny-SFT, learning aptitude, max-over-protocols
Revise the recipe; judge training progress
Experience-to-metric
A new prompt, model, or crawler version
Component-level evaluation; cheap
Task-specific evaluation set; SEO page-count estimate
Deploy, or iterate again
Table 2: How each loop type instantiates the four RSI steps. Debugging and experience-to-metric loops close quickly on cheap components; the headroom loop closes only a few times per project, because its run is the pre-training run itself.
Figure 3: Overview of internal training metrics and external evaluation protocols for pre-training monitoring.
Figure 4: Two pilots and the main pre-training in a three-month period. Top : main configuration changes; matching colors link each iteration to its evaluation curves below. Bottom : initial pilot (warm brown) and revised pilot (gray-blue) stopped at 1T and 3T tokens, respectively; main pre-training (dark teal) is shown through Stage1-1 at 7T tokens. The five subset scores are means over their constituent benchmarks, and Overall is the mean over all shared benchmarks.
Figure 5: The debugging loops are instantiated as (a) the infra debugging loop and (b) the data debugging loop.
Figure 6: Two contrasting load-balancing anomalies in pilot runs. Gray curves show layer-wise MaxVio across 48 layers; the dashed curve shows aggregate MaxVio. (a) In the initial pilot, both aggregate and layer-wise MaxVio decline under strong balancing, suggesting potential over-balancing. The inset enlarges the low-MaxVio region. (b) In the revised pilot, layers 15, 18, and 48 exhibit elevated MaxVio despite a moderate aggregate, revealing layer-specific imbalance obscured by aggregate monitoring.
Figure 7: Layer-wise load imbalance rises as downstream evaluation gains diminish during training. The gray shaded region marks the same interval (approximately 2.2–6T tokens) in both panels. (a) Layer-15 MaxVio increases rapidly during this interval, while global MaxVio remains relatively stable. (b) Over the same interval, downstream evaluation gains gradually diminish. Higher MaxVio indicates greater load imbalance. This contrast illustrates how layer-wise internal metrics provide a more granular signal for detecting potential training issues than downstream evaluation alone.
Figure 8: Diagnosing bad patterns in model outputs. Top : three representative output patterns, a credential leak with a verbatim-repeated config block, a jump into a fixed “Does this sentence make sense?” QA template, and a self-spawning chain of near-identical word problems. Bottom : the eight-step loop that traces each pattern back to a data-pipeline defect. Agents handle detection, tracing, and inspection at scale, humans set triage priority and approve the fix.
Area
Finding
Fix
Load balancing
Expert load was either over-balanced across all layers or severely imbalanced in a few individual layers (Figure 6 ).
Adopt global-batch load balancing ( Qiu et al., 2025 ) , which relaxes the per-micro-batch constraint and allows experts to specialize.
Optimizer
Weight-norm and gradient-norm trends diverged across modules.
Reassign which modules use Muon and which stay on AdamW (Table 5 ).
Data quality
Pilot models generated advertisements, which were traced to advertisements and low-quality web pages remaining in the training corpus.
Filter them further with quality classifiers.
Weight decay
Router logits showed abnormal trends during training.
Exclude normalization and bias parameters from weight decay.
Data loader
When resuming a training task with the same global batch size but a different DP size, the sampler did not preserve the per-step data order of the previous task.
Restore the per-step sample order of the previous task when resuming with a different DP size.
Table 3: The five cross-stack diagnoses resolved with the context infrastructure across the two pilots and main pre-training.
Figure 9: Tiny-SFT data composition and validation. Left: Token composition of the 6.33B-token mix. Math, Science ( Xu et al., 2026a ) , and Code ( Ahmad et al., 2025 ) each contribute approximately 2.00B tokens (31.6%), while instruction following ( Pyatkin et al., 2025 ) contributes 0.331B tokens (5.2%). Right: Qwen3-30B-A3B versus its base model (Qwen3-30B-A3B-Base) after one epoch of Tiny-SFT. Scores are percentages across 13 benchmarks; the fine-tuned model scores higher on 10 of them. This validation supports the effectiveness of Tiny-SFT as a proxy for monitoring model performance after limited post-training.
Figure 10: Apparent stagnation in base evaluation can coexist with improving post-SFT performance. (a) Direct base-model Overall_Avg over pre-training; red boxes mark the checkpoint intervals examined in (b) and (c). Each point in (b) and (c) reports SFT AVG13 after applying the same SFT procedure independently to a checkpoint from (a). (b) Stage1-1, steps 123,000–210,000 (4.1T-7T tokens): post-SFT scores improve by 1.49 points despite the near-flat base trend. (c) Stage1-3, steps 243,000–345,000 (8.1T-11.5T tokens): post-SFT scores improve by 2.39 points while base scores fluctuate around a plateau.
Figure 11: Base and post-SFT checkpoint rankings are not interchangeable. Left : base Overall_Avg and SFT AVG13 for the same checkpoints. Right : all paired checkpoints, sorted by base rank within each group, with their corresponding SFT ranks. Rank 1 is best, ties receive average ranks, and ρ denotes Spearman correlation computed within each group from the reported scores. Base and SFT scores use different benchmark suites.
Figure 12: A toy illustration of learning aptitude. Thin lines with hollow markers are raw scores at SFT checkpoints, and each straight line is their least-squares fit of Eq. ( 1 ): the intercept b is the score at the first checkpoint N0 , and the slope k is the gain per tenfold increase in post-training data. The model with high learning aptitude starts lowest, yet reaches the target score s∗ with about 15× less data than the baseline (Eq. ( 2 )). The high-start, flat model only reaches it at the end of the plotted budget.
Figure 13: Assessing base-model capabilities with max-over-protocols evaluation.
Figure 14: Iterative development of a data classifier. Left : a human defines the classification task (1); a strong model samples data and annotates an evaluation set (2); an agent analyzes cases and tunes the labeling prompt (3); the strong model uses that prompt to label training data (4); a compact classifier is distilled from these labels (5) and evaluated (6). The component can be deployed as a tuned prompt (Option A) or a distilled model (Option B). Right : accuracy and F1 over three iterations (v1–v3) for classifiers of ad intent, computing subject, and clinical health subject.
Figure 15: Structure-aware web data acquisition. A human requirement seeds an agent-driven pipeline with two loops. Loop 1 (site discovery) expands and verifies related sites through the web’s link structure. Loop 2 (crawl coverage) crawls each confirmed site to a coverage target informed by its SEO page estimate. Links, search indexes, and robots directives are treated as structured signals rather than discarded.
Figure 16: Overview of agentic-data pipeline. The key component is the Validation gates, which admits a task only when it clears policy, execution, and semantic checks and the same evaluation fails on the initial code but passes on the reference, and sanitization into a Docker/OCI image that keeps the external eval but strips the reference patch and original history. Rejected tasks return with evidence and fix hints (dashed path).
Figure 17: The multi-harness rollout layer. A single launch(harness, prompt, infer_cfg) call in the unified adapter drives all nine integrated harnesses, which among them speak three API protocols (Anthropic Messages ( Anthropic, 2026 ) , OpenAI Responses ( OpenAI, 2026b ) , and OpenAI Chat Completions ( OpenAI, 2026a ) ). The Bifrost gateway ( Maxim AI, 2025 ) normalizes the three protocols onto one LLM serving backend and injects inference settings (context length, maximum output, temperature) centrally.
Category
Specific behavior
Pull the reference fix from version control
Fetch its own merged PR diff or the fixing commit patch
Recover the fix from unreachable Git objects ( git fsck , cat-file )
Fetch upstream code over the network
Download files from raw.githubusercontent , the GitHub API, or PyPI JSON
Clone, fetch, or add a remote pointing at the upstream repository
Install an already-fixed version
pip download or install --target with a pinned version range
Pull packages through conda, apt, or an external mirror (aliyun, tuna, ustc)
Table 4: Agentic anti-hack patterns, grouped by category. Each is a concrete way a model can pass a synthesized task without solving it. Agents scan trajectories for these behaviors and flag candidates for review or re-synthesis.
Figure 18: Funnel chart of agentic data pipeline. 260K raw PRs are filtered into 220K executable tasks. Rolling each task out under 9 harnesses gives about 2M trajectories. After verification, 1.1M trajectories remain, totaling more than 100B training tokens (about 91K tokens per trajectory on average). Tasks and trajectories are drawn on separate scales.
Figure 19: Pretrain data pipelines except the agentic data pipeline. Filled boxes mark model-based components such as classifiers, OCR, VLMs, and LLM rewriters, which we build and tune with the workflow in Figure 14 ( Qin et al., 2026a ) .
Figure 20: Composition of the pre-training corpus, measured in unique tokens that entered training (10.5T in total).
Figure 21: Training recipe of 801-30B-A3B across the five training stages. (a) Key recipe changes, including the progression from 4K to 32K and 256K sequence lengths and the RoPE-base updates during Stage1. (b) Configured data sampling proportions. Stage1-2 and Stage1-3 retain the same category-level mixture despite updates to their data sources. Other denotes capability-targeted data.
Figure 22: (a) Grouping packs by attention cost reduces imbalance; values are illustrative. (b) Imbalance in our 256K corpus across sorting windows, using 128 DP ranks and one pack per rank. The ratio is maximum to mean rank cost, averaged across microbatches; 1 denotes perfect balance.
Figure 23: Training loss with and without attention-balanced data sampling (32-step moving average).
Setting
Experts
Attention
Router
Embedding/output/norm/bias
LBL
AdamW–Global
AdamW
AdamW
AdamW
AdamW
Global
Muon-E–Global
Muon
AdamW
AdamW
AdamW
Global
Muon-E-A–Global
Muon
Muon
AdamW
AdamW
Global
Muon-E-A–Local
Muon
Muon
AdamW
AdamW
Local
Table 5: Optimizer allocation in the second pre-training pilot. E denotes expert matrices and E-A denotes expert and attention matrices. LBL denotes the load-balancing scope. All Muon variants use RMS matching to AdamW ( Liu et al., 2025 ) , momentum 0.95, and five Newton–Schulz iterations; AdamW uses (β1,β2)=(0.9,0.95) . Weight decay is 0.1. Router parameters use AdamW in every setting.
Figure 24: Optimizer allocation and load-balancing comparisons for the four settings in Table 5 . Left : training LM loss, excluding auxiliary losses, averaged in consecutive 100-step bins after the 2,000-step warmup. Right : direct base-model Overall Avg scores at 100B-token intervals. The two expert-and-attention settings share a color, with a dashed line denoting local load balancing. Both global Muon allocations improve on all-AdamW in this interval, while the local/global score differences are small.
Category
Benchmark
Metric
Shots
Decoding
T
Top- p
Candidates
Max. tokens
Commonsense
ARC-Easy
Exact match
0
Greedy
1
16,384
ARC-Challenge
Exact match
0
Greedy
1
16,384
Reasoning
MMLU-Pro
Exact match
0
Greedy
1
16,384
MMLU-Redux
Exact match
0
Greedy
1
4,096
BBH
Exact match
3
Greedy
1
1,024
AGIEval-EN
Exact match
5
Greedy
1
2,048
Table 6: Default base-evaluation settings (except Gemma-4-26B-A4B). Rows follow the five benchmark categories introduced in the text. T denotes temperature, candidates are counted per problem, and generation limits are in tokens. Greedy decoding uses T=0 . MultiPL-E scores are weighted by the number of problems in each language.
Table 7: Base-model evaluation. All entries are base checkpoints. Scores are percentages; AVG is the unweighted mean of the 13 benchmark scores. Bold indicates the highest score among the models shown. ∗ Gemma-4-26B-A4B reports the maximum score across multiple prompts and evaluation protocols for each benchmark, as described in Section 8.1.1 .
Table 8: General capabilities of Base+SFT models(%). All base models are fine-tuned on the same 6B-token SFT data without reinforcement learning. Scores are percentages; higher is better. AVG5 and AVG13 are the unweighted means of the upper and lower blocks, respectively. Bold indicates the highest score among the models shown, including ties.
Model
SWE-bench Verified
SWE-bench Pro
CyberGym
KernelBench
AVG
Qwen3-30B-A3B-Thinking-2507
20.30
7.01
1.19
5.60
8.53
Qwen3.5-35B-A3B
69.10
39.23
13.47
17.20
34.75
Qwen3.6-35B-A3B
71.65
45.28
24.22
37.20
44.59
K2-Horizon-MoVA-36B-A4B
64.15
44.19
39.35
33.60
45.32
Gemma-4-26B-A4B-it
57.15
34.57
8.49
44.00
36.05
Nemotron-3.5-Lightning-30B-A3B
55.35
32.49
8.29
20.40
29.13
Table 9: Agentic benchmark results (%). Bold indicates the highest score among the models shown; AVG is the unweighted mean of the four benchmark scores. All baselines are provider-released post-trained models; 801-30B-A3B is evaluated after the lightweight SFT stage only.
Figure 25: Learning aptitude of 801-30B-A3B and Qwen3-30B-A3B on SWE-bench Pro (left) and BFCL Full (right). Hollow markers are scores at successive SFT checkpoints; the pale line is the least-squares fit of Eq. ( 1 ), and k in the legend is its slope, the gain in points per tenfold increase in SFT steps.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 26: Layer-wise MaxVio trajectories for all 48 MoE layers.
Benchmark
Description
General commonsense (7)
ARC-Easy ( Clark et al., 2018 )
School-level science questions, easy split.
ARC-Challenge ( Clark et al., 2018 )
More challenging science questions requiring reasoning.
HellaSwag ( Zellers et al., 2019 )
Selecting plausible continuations of everyday events.
OpenBookQA ( Mihaylov et al., 2018 )
Combining basic science facts with commonsense.
PIQA ( Bisk et al., 2019 )
Physical commonsense and practical problem solving.
Appendix
Table 10: The 22-benchmark pre-training monitoring suite, with brief introductions to the evaluated tasks.
Protocol
In-context examples
Final-answer format
Scoring
Non-code tasks
5-shot
5 solved examples: final answers only for ARC, GPQA-Main, MMLU-Redux, and AGIEval-EN; worked solutions for MMLU-Pro and SuperGPQA
Recursive Self-Improvement (RSI) aims to enable AI systems to autonomously build and improve AI systems themselves, including models, harnesses, and underlying infrastructure. Within this broader agenda, a central capability is enabling agents to train and improve base models. However, existing studies on automated model training primarily focused on workflow- or harness-level optimizations, rather than internalizing the ability to conduct post-training into the model itself. Developing this capability requires executable model training environments that can produce high-quality trajectories demonstrating how to select data, train models, diagnose failures, and iteratively improve results. In this paper, we introduce RSI-Dojo, a collection of executable model training environments designed to teach LLM agents to conduct post-training. To construct RSI-Dojo, we develop RSI-Craft, an automatic pipeline that converts human research artifacts into executable model training environments. RSI-Craft automatically collects research papers from top venues, instantiates datasets and metrics into a unified interface, and reproduces evaluation dependencies. It then filters tasks by data size, execution reliability, and discriminability, yielding RSI-Dojo with 281 seed model training tasks. Building on RSI-Dojo, we employ a trajectory collection method with an improved harness to collect high-quality model-training trajectories from frontier models, then retain examples through rubric-based and outcome-based filtering. We fine-tune Qwen3.5-9B and Qwen3.6-35B-A3B on these trajectories. The resulting RSI-Dojo-9B and RSI-Dojo-35B-A3B improve over their base models by 8.74 and 16.87 PostTrainBench points, respectively. RSI-Dojo-35B-A3B reaches 29.95 on PostTrainBench and 53.29 on PostTrainBench-Lite, matching the performance of frontier models such as GLM-5.2 and GPT 5.5.
Shuzheng Gao, Zhaojian Yu, Penghao Yin +4
Department of Computer Science and Engineering, The Chinese University of Hong Kong · ByteDance · Tsinghua University +1
Frontier models, general-purpose and domain-specific alike, now include a midtraining stage between pretraining and post-training, but little has been published about how this stage is built. We present an open midtraining recipe that primes base models for agentic post-training, and we show its effectiveness on two Nemotron models of different sizes. Pretraining typically runs at short context and sees almost no agent interactions, while SFT data is mostly synthetic, and very long SFT stages can hurt the model's later RL trainability. Our midtraining stage sits between the two. It trains at 256K context for hundreds of billions of tokens on a diverse mixture of natural software-engineering data like packed repositories, pull requests, commit histories, and build logs together with a small share of agentic traces. On Nemotron 3.5 Lightning, 500B midtraining tokens improve SWE-bench Verified from 60.7 to 64.5, SWE-bench Multilingual from 55.8 to 62.8, and SWE-bench Pro from 32.0 to 35.5 after identical full SFT. On the larger Nemotron 3.5 Super, 280B midtraining tokens improve DeepSWE, a harder benchmark of long-horizon tasks, from 11.7 to 15.7 and SWE-bench Verified from 72.9 to 74.3 after identical full SFT. Midtraining also improves robustness across agent harnesses on both models. We report the ablations behind the recipe and an evaluation protocol built on short fixed SFT runs, since base-model proxy metrics did not correlate reliably with post-training results.
Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.
Yue Huang, Yanbo Wang, Lichi Li +15
Bake AI · University of Notre Dame · University of California, Los Angeles +8