Training LLM Agents to Automate LLM Post-Training with RSI-Dojo
Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong · ByteDance · Tsinghua University · Shanghai Jiao Tong University
Abstract
Recursive Self-Improvement (RSI) aims to enable AI systems to autonomously build and improve AI systems themselves, including models, harnesses, and underlying infrastructure. Within this broader agenda, a central capability is enabling agents to train and improve base models. However, existing studies on automated model training primarily focused on workflow- or harness-level optimizations, rather than internalizing the ability to conduct post-training into the model itself. Developing this capability requires executable model training environments that can produce high-quality trajectories demonstrating how to select data, train models, diagnose failures, and iteratively improve results. In this paper, we introduce RSI-Dojo, a collection of executable model training environments designed to teach LLM agents to conduct post-training. To construct RSI-Dojo, we develop RSI-Craft, an automatic pipeline that converts human research artifacts into executable model training environments. RSI-Craft automatically collects research papers from top venues, instantiates datasets and metrics into a unified interface, and reproduces evaluation dependencies. It then filters tasks by data size, execution reliability, and discriminability, yielding RSI-Dojo with 281 seed model training tasks. Building on RSI-Dojo, we employ a trajectory collection method with an improved harness to collect high-quality model-training trajectories from frontier models, then retain examples through rubric-based and outcome-based filtering. We fine-tune Qwen3.5-9B and Qwen3.6-35B-A3B on these trajectories. The resulting RSI-Dojo-9B and RSI-Dojo-35B-A3B improve over their base models by 8.74 and 16.87 PostTrainBench points, respectively. RSI-Dojo-35B-A3B reaches 29.95 on PostTrainBench and 53.29 on PostTrainBench-Lite, matching the performance of frontier models such as GLM-5.2 and GPT 5.5.
Figures & tables
| Model | PostTrainBench | PostTrainBench-Lite | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Avg. | Qwen-1.7B | Qwen-4B | SmolLM3-3B | Gemma3-4B | Avg. | Qwen-1.7B | Qwen-4B | SmolLM3-3B | |
| Our Evaluation Results | |||||||||
| Qwen3.5-9B | 7.72 | 6.65 | 15.09 | 4.52 | 4.60 | 0.56 | 0.00 | 1.67 | 0.00 |
| RSI-Dojo-9B | 16.46 | 15.05 | 21.48 | 14.65 | 14.66 | 15.71 | 11.47 | 15.67 | 19.99 |
| Qwen3.6-35B-A3B | 13.08 | 10.15 | 17.93 | 11.60 | 12.63 | 5.80 | 12.87 | 4.54 | 0.00 |
| RSI-Dojo-35B-A3B | 29.95 | 28.60 | 34.85 | 27.65 | 28.69 | 53.29 | 52.49 | 58.48 | 48.90 |
| Model | At Least One | Improved | Eff./Total (Rate) |
|---|---|---|---|
| Successful Run | |||
| Qwen3.5-9B | 10/28 | 1/28 | 2/13 (15.38%) |
| RSI-Dojo-9B | 24/28 | 15/28 | 29/247 (11.74%) |
| Qwen3.6-35B-A3B | 15/28 | 11/28 | 17/68 (25.00%) |
| RSI-Dojo-35B-A3B | 28/28 | 26/28 | 61/175 (34.86%) |
| GPT-5.4 | 28/28 | 23/28 | 112/586 (19.11%) |
| Method | Overall | AIME 2025 | BFCL | GSM8K | HumanEval | Improved | At Least One Successful Run |
|---|---|---|---|---|---|---|---|
| Behavior only | 30.66 | 0.00 | 46.23 | 47.50 | 28.90 | 7/12 | 12/12 |
| Outcome only | 26.13 | 4.12 | 32.62 | 39.88 | 27.89 | 8/12 | 10/12 |
| Both filters | 53.29 | 0.00 | 95.05 | 69.49 | 48.62 | 9/12 | 12/12 |
LLM Post-Training
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.
See all 258 papers