Organizations: Zhejiang University · Westlake University · Alibaba Token Hub, Alibaba Group · South China University of Technology · The Chinese University of Hong Kong
Recursive self-improvement (RSI) requires agents to diagnose system bottlenecks and coordinate improvements across training and inference. We refer to the agents' capability to jointly optimize training data, the trainer, and the inference harness as cross-component co-design. We introduce VibeTraining-Bench, a benchmark for evaluating this capability across tasks spanning terminal operation, software engineering, multi-turn tool use, and mathematical reasoning. Within each task, research agents start from an identical initial agent system under a fixed resource budget. Guided by validation feedback, they iteratively refine data, trainers, and inference harnesses, then select and submit a final agent system for evaluation on the test set. To prevent reward hacking, we enforce three safeguards: test data isolation, separate research and grading containers, and network restrictions for research agents. Our evaluation results show that frontier research agents substantially improve target agent systems. For example, on SWE-bench Verified, GPT-6-Astra raises the target agent system's performance from 32.9% to 53.7%, comparable to the human-developed instruct model's 55.4%. However, there is still room to improve their ability to co-design and iteratively refine data, trainer, and harness to achieve cumulative performance gains. We hope that VibeTraining-Bench will serve as a unified platform for advancing RSI, deepening our understanding of cross-component optimization for agent systems. Code and data are available at https://github.com/D2I-ai/VibeTraining-Bench.
Figures & tables
Benchmark
Research Loop
Editable Training
System Co-Design
Controlled Infrastructure
Trajectory Feedback
Iterative Revision
Data
Trainer
Harness
Joint
Dev/Test Isolation
Train/Eval Isolation
MLS-Bench
△
✓
✗
✓
✗
✗
△
✗
PostTrainBench
✓
✗
✓
✓
✗
✗
✗
✗
Curation-Bench
✓
✓
✓
✗
✗
✗
✗
△
RSIBench-Data
✓
✓
✓
✗
✗
✗
✗
✓
Agent 2 RL-Bench
✗
✓
✓
✓
✗
✗
✗
✓
Table 1: Protocol comparison of benchmarks for recursive self-improvement. Legend: ✓ = enforced; △ = partial or insufficiently documented; ✗ = unsupported; – = not applicable. Trajectory feedback requires task-level traces or diagnostics, and iterative revision requires repeated propose–evaluate–revise cycles. Joint requires coordinated changes to training data, the trainer, and the inference harness within one task. Dev/test isolation keeps the final test inaccessible during development; train/eval isolation separates training, serving, and evaluation services.
Figure 1: Overview of VibeTraining-Bench . The benchmark assesses a research agent’s capacity for systemic co-design across three interdependent components: training data, the trainer, and the inference harness. (Left) Research agents initialize from domain-specific data sources and task-specific resource budgets shared across research-agent configurations. (Center) Within the autonomous loop, research agents iteratively orchestrate component modifications using validation signals until budget exhaustion. (Right) Strict architectural decoupling isolates the optimization workspace from the grading server to prevent reward hacking. Only the visible validation set provides feedback; the hidden test set remains inaccessible until independent test evaluation of the selected system.
Capability
Benchmark
Initial Harness
Editable Space
Validation / Hidden Test
Budget
Terminal operation
Terminal -Bench 2.1
Terminus 2
{D,T,H}
OpenThoughts-TBLite / Terminal-Bench 2.1
24 h
Software engineering
SWE-bench Verified
mini-SWE- agent
{D,T,H}
Validation subset / remaining test tasks
24 h
Multi-turn tool use
τ3 -Bench
Benchmark- provided
{D,T}
Validation subset / remaining test tasks
16 h
Multi-turn tool use
Toolathlon
Benchmark- provided
{D,T}
ToolGym / Toolathlon
16 h
Mathematical reasoning
AIME
None
{D,T}
AIME 2025 / AIME 2026
16 h
Table 2: Task composition of VibeTraining-Bench. The editable space indicates whether the research agent may modify training data ( D ), the trainer ( T ), and the inference harness ( H ).
Research agent
Avg.
SWE-bench Verified
Terminal- Bench 2.1
τ3 - Bench
Toolathlon
AIME 2026
Hours
Initial system
22.7
32.9
8.9
20.3
4.5
46.7
–
Human-tuned baseline
47.3
55.4
20.2
64.1
21.6
75.0
–
GPT-6-Astra +Codex
37.1
53.7
13.5
49.7
7.9
60.8
85.9
GPT-5.6-Sol +Codex
31.0
42.9
8.9
42.1
10.2
50.8
64.0
Kimi-K3 +CC
38.3
51.7
22.4
52.1
10.2
55.0
73.0
GLM-5.2 +CC
31.4
40.6
10.1
36.2
6.8
63.3
56.9
Table 3: Test performance (%) of the selected agent systems. Initial system uses Qwen3.5-9B-Base before agent-led optimization. Human-tuned baseline uses Qwen3.5-9B-Instruct, a human-developed instruct model. Subscripts indicate the harness used by each research agent; CC denotes Claude Code. Avg. denotes the mean score across the five tasks. Bold marks the highest score among the six research agents in each performance column, excluding the two baselines. Hours reports the sum of elapsed research time across the five task runs.
Figure 2: Mean generated tokens and macro-averaged final performance across five benchmarks. Dashed line marks the initial system score.
Figure 3: Each point represents a completed validation evaluation, ordered chronologically within each run. Repeated evaluations of the same candidate are shown separately.
Editable
SWE
Terminal
Initial system
32.9
8.9
H only
38.0
8.9
D/T only
–
13.5
D/T/H
51.7
22.4
Table 4: Kimi-K3 test scores (%) on SWE-bench Verified and Terminal-Bench 2.1. D/T/H denotes data, trainer, and harness; Kimi-K3 did not edit the harness in the SWE D/T/H run.
Figure 4: Number of designs involving training data, the trainer, and the inference harness. A design may involve multiple components. Numbers below the research-agent labels indicate the total number of distinct designs in each run.
Figure 5: Case study of GPT-6-Astra’s research trajectory on SWE-bench Verified. Selected decisions illustrate how validation feedback guides changes to training data, the trainer, and the inference harness. Ellipses indicate omitted decisions.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Kimi-K3 on Terminal-Bench. The selected system combines the expanded data mixture with the revised harness.
Figure 7: Qwen-3.8-Max on Terminal-Bench. The data curriculum changes while the training recipe and harness remain fixed.
Figure 8: GPT-5.6-Sol on Terminal-Bench. Decisions 6 and 7 hold data, learning rate, and harness fixed while correcting initialization.
Figure 9: GLM-5.2 on Terminal-Bench. Data expansion fails to improve on the first selected model–harness pair.
Figure 10: DeepSeek-V4-Pro on Terminal-Bench. Two-model merging and three-checkpoint averaging are Trainer interventions.
Figure 11: GPT-6-Astra on Terminal-Bench. The selected Data + Trainer intervention lies between two fixed-weight Harness comparisons.
Figure 12: Qwen-3.8-Max on SWE-bench Verified. Source, length, and coverage change together. Validation denominators reflect the actual graded tasks.
Figure 13: DeepSeek-V4-Pro on SWE-bench Verified. Checkpoint-saving cadence changes training-system overhead rather than the learning algorithm.
Figure 14: Kimi-K3 on SWE-bench Verified. The per-issue quota changes; the accompanying matched-task comparison distinguishes outcome changes from grading coverage.
Figure 15: GPT-5.6-Sol on SWE-bench Verified. Subset and full-scope validation retain their actual denominators.
Figure 16: GLM-5.2 on SWE-bench Verified. The fixed-weight Harness comparison preserves the aggregate score while changing the success set.
Table 5: Initial data resources provisioned for each benchmark.
Benchmark
Evaluation settings
Terminal-Bench 2.1
Maximum 200 agent turns; 65,536-token context; 4,096 maximum output tokens; temperature 1.0; thinking disabled; one run per task.
SWE-bench Verified
256K-token context; temperature 1.0; thinking enabled; one run per task.
AIME
16,384 maximum output tokens; temperature 0.6; thinking enabled; average@4.
Toolathlon
Maximum 100 agent turns; 256K-token context; 8,192 maximum output tokens; temperature 1.0; thinking disabled; pass@3.
τ3 -Bench
Maximum 200 agent turns; 256K-token context; temperature 0; thinking disabled; one run per task.
Appendix
Table 6: Evaluation settings shared by validation and test runs.
Figure 17: Elapsed research time by activity for six research-agent runs on Toolathlon, with one bar per run. waiting and submit+grade together account for 66 % of the GPT-6-Astra run and 88 % of the Kimi-K3 run. For GLM-5.2, a one-hour gap in the episode records leaves 9.0 % of elapsed time classified as uncovered .
Figure 18: Elapsed research time by activity for six research-agent runs on AIME. waiting and submit+grade together account for 76–91 % of elapsed time across the six runs. Evaluation episodes include submissions of individual model checkpoints or weight-averaged models on either the full question set or a prefix selected by the limit parameter. These episodes are classified as submit+grade . The uncovered share never exceeds 0.6 %.
Figure 19: Elapsed research time by activity for six research-agent runs on τ3 -Bench. waiting and submit+grade together account for 69–92 % of elapsed time across the six runs. Some candidates are constructed by interpolating the weights of existing models without additional training. Their evaluation episodes are classified as submit+grade .
Figure 20: Elapsed research time by activity for six research-agent runs on Terminal-Bench 2.1. The DeepSeek-V4-Pro episode records contain qualitative markers such as “session open”, “first full split”, “mid”, and “last”, but no durations. Its row is retained with an explanation, without a time breakdown. Across the five plotted runs, waiting and submit+grade together account for 35–95 % of elapsed time. GPT-6-Astra has the lowest combined share, with 36 % of its elapsed time classified as uncovered .
Figure 21: Elapsed research time by activity for six research-agent runs on SWE-bench Verified. The GPT-6-Astra episode records specify episode order but contain only one timestamp, so its row is shown without a time breakdown. Each episode in the other five runs has both a start and an end timestamp; no episode duration is interpolated. Across these five runs, waiting and submit+grade together account for 63–88 % of elapsed time.
Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents' research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.
Recursive self-improvement requires carrying accepted changes into later improvement cycles, while studying agent-proposed changes also requires substantial research infrastructure. Existing settings often leave agents to rebuild routine infrastructure or restrict exploration to individual components. We introduce RSIGym, an agent-native research environment based on Everything as a Service (EaaS). RSIGym exposes training, inference, rollout, evaluation, and sandbox execution through reusable services, with shared budget and permission controls supporting Data, Harness, and Joint improvement tracks. This design enables agents to investigate individual interventions and jointly optimize data, training settings, and execution harnesses within the same environment. We define RSI-Index as the mean fraction of the remaining performance gap closed across five benchmarks covering software engineering, terminal interaction, mathematics, scientific reasoning, and skill-based tasks. Comparing six frontier research models in independent Joint runs, Opus 5 achieves the highest RSI-Index of 0.4809 under a $500 platform-service budget per benchmark run. Its selected systems improve all five benchmarks, raising SWE-bench Verified from 17.67% to 50.33% and AIME from 31.67% to 97.78%. Additional experiments examine DSH-harness refinement, budget variation, and restricted network access, while recorded trajectories reveal how agents diagnose failures and select candidates. We open-source the full RSIGym codebase and results to support reproducibility and further research.
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Yaolun Zhang, Tianyi Xu, Yujie Zhao +3
Oregon State University · AG2 AI · University of California, San Diego +1