VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement

Ruotian Peng; Yuanyi Ren; Shuaiqi Liu; Xin Guo; Fang Zhu; Yangyi Huang; Yue Wu; Zheng Wang

cs.AIOct 19, 2026

VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement

Authors: Ruotian Peng, Yuanyi Ren, Shuaiqi Liu, Xin Guo, Fang Zhu, Yangyi Huang, Yue Wu, Zheng Wang,

Organizations: Zhejiang University · Westlake University · Alibaba Token Hub, Alibaba Group · South China University of Technology ·

Abstract

Recursive self-improvement (RSI) requires agents to diagnose system bottlenecks and coordinate improvements across training and inference. We refer to the agents' capability to jointly optimize training data, the trainer, and the inference harness as cross-component co-design. We introduce VibeTraining-Bench, a benchmark for evaluating this capability across tasks spanning terminal operation, software engineering, multi-turn tool use, and mathematical reasoning. Within each task, research agents start from an identical initial agent system under a fixed resource budget. Guided by validation feedback, they iteratively refine data, trainers, and inference harnesses, then select and submit a final agent system for evaluation on the test set. To prevent reward hacking, we enforce three safeguards: test data isolation, separate research and grading containers, and network restrictions for research agents. Our evaluation results show that frontier research agents substantially improve target agent systems. For example, on SWE-bench Verified, GPT-6-Astra raises the target agent system's performance from 32.9% to 53.7%, comparable to the human-developed instruct model's 55.4%. However, there is still room to improve their ability to co-design and iteratively refine data, trainer, and harness to achieve cumulative performance gains. We hope that VibeTraining-Bench will serve as a unified platform for advancing RSI, deepening our understanding of cross-component optimization for agent systems. Code and data are available at https://github.com/D2I-ai/VibeTraining-Bench.

Figures & tables

Benchmark	Research Loop		Editable Training		System Co-Design		Controlled Infrastructure
Benchmark	Trajectory Feedback	Iterative Revision	Data	Trainer	Harness	Joint	Dev/Test Isolation	Train/Eval Isolation
MLS-Bench	$\triangle$	✓	✗	✓	✗	✗	$\triangle$	✗
PostTrainBench	✓	✗	✓	✓	✗	✗	✗	✗
Curation-Bench	✓	✓	✓	✗	✗	✗	✗	$\triangle$
RSIBench-Data	✓	✓	✓	✗	✗	✗	✗	✓
Agent 2 RL-Bench	✗	✓	✓	✓	✗	✗	✗	✓

Table 1: Protocol comparison of benchmarks for recursive self-improvement. Legend: ✓ = enforced;

\triangle

= partial or insufficiently documented; ✗ = unsupported; – = not applicable. Trajectory feedback requires task-level traces or diagnostics, and iterative revision requires repeated propose–evaluate–revise cycles. Joint requires coordinated changes to training data, the trainer, and the inference harness within one task. Dev/test isolation keeps the final test inaccessible during development; train/eval isolation separates training, serving, and evaluation services.

Figure 1: Overview of VibeTraining-Bench . The benchmark assesses a research agent’s capacity for systemic co-design across three interdependent components: training data, the trainer, and the inference harness. (Left) Research agents initialize from domain-specific data sources and task-specific resource budgets shared across research-agent configurations. (Center) Within the autonomous loop, research agents iteratively orchestrate component modifications using validation signals until budget exhaustion. (Right) Strict architectural decoupling isolates the optimization workspace from the grading server to prevent reward hacking. Only the visible validation set provides feedback; the hidden test set remains inaccessible until independent test evaluation of the selected system.

Capability	Benchmark	Initial Harness	Editable Space	Validation / Hidden Test	Budget
Terminal operation	Terminal -Bench 2.1	Terminus 2	$\{D,T,H\}$	OpenThoughts-TBLite / Terminal-Bench 2.1	24 h
Software engineering	SWE-bench Verified	mini-SWE- agent	$\{D,T,H\}$	Validation subset / remaining test tasks	24 h
Multi-turn tool use	$\tau^{3}$ -Bench	Benchmark- provided	$\{D,T\}$	Validation subset / remaining test tasks	16 h
Multi-turn tool use	Toolathlon	Benchmark- provided	$\{D,T\}$	ToolGym / Toolathlon	16 h
Mathematical reasoning	AIME	None	$\{D,T\}$	AIME 2025 / AIME 2026	16 h

Table 2: Task composition of VibeTraining-Bench. The editable space indicates whether the research agent may modify training data (

D

), the trainer (

T

), and the inference harness (

H

Research agent	Avg.	SWE-bench Verified	Terminal- Bench 2.1	$\tau^{3}$ - Bench	Toolathlon	AIME 2026	Hours
Initial system	22.7	32.9	8.9	20.3	4.5	46.7	–
Human-tuned baseline	47.3	55.4	20.2	64.1	21.6	75.0	–
GPT-6-Astra +Codex	37.1	53.7	13.5	49.7	7.9	60.8	85.9
GPT-5.6-Sol +Codex	31.0	42.9	8.9	42.1	10.2	50.8	64.0
Kimi-K3 +CC	38.3	51.7	22.4	52.1	10.2	55.0	73.0
GLM-5.2 +CC	31.4	40.6	10.1	36.2	6.8	63.3	56.9

Table 3: Test performance (%) of the selected agent systems. Initial system uses Qwen3.5-9B-Base before agent-led optimization. Human-tuned baseline uses Qwen3.5-9B-Instruct, a human-developed instruct model. Subscripts indicate the harness used by each research agent; CC denotes Claude Code. Avg. denotes the mean score across the five tasks. Bold marks the highest score among the six research agents in each performance column, excluding the two baselines. Hours reports the sum of elapsed research time across the five task runs.

Figure 2: Mean generated tokens and macro-averaged final performance across five benchmarks. Dashed line marks the initial system score.

Figure 3: Each point represents a completed validation evaluation, ordered chronologically within each run. Repeated evaluations of the same candidate are shown separately.

Editable	SWE	Terminal
Initial system	32.9	8.9
H only	38.0	8.9
D/T only	–	13.5
D/T/H	51.7	22.4

Table 4: Kimi-K3 test scores (%) on SWE-bench Verified and Terminal-Bench 2.1. D/T/H denotes data, trainer, and harness; Kimi-K3 did not edit the harness in the SWE D/T/H run.

Figure 4: Number of designs involving training data, the trainer, and the inference harness. A design may involve multiple components. Numbers below the research-agent labels indicate the total number of distinct designs in each run.

Panel of Figure 5 — Figure 5: Case study of GPT-6-Astra’s research trajectory on SWE-bench Verified. Selected decisions illustrate how validation feedback guides changes to training data, the trainer, and the inference harness. Ellipses indicate omitted decisions.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Kimi-K3 on Terminal-Bench. The selected system combines the expanded data mixture with the revised harness.

Panel of Figure 7 — Figure 7: Qwen-3.8-Max on Terminal-Bench. The data curriculum changes while the training recipe and harness remain fixed.

Panel of Figure 8 — Figure 8: GPT-5.6-Sol on Terminal-Bench. Decisions 6 and 7 hold data, learning rate, and harness fixed while correcting initialization.

Panel of Figure 9 — Figure 9: GLM-5.2 on Terminal-Bench. Data expansion fails to improve on the first selected model–harness pair.

Panel of Figure 10 — Figure 10: DeepSeek-V4-Pro on Terminal-Bench. Two-model merging and three-checkpoint averaging are Trainer interventions.

Panel of Figure 11 — Figure 11: GPT-6-Astra on Terminal-Bench. The selected Data + Trainer intervention lies between two fixed-weight Harness comparisons.

Panel of Figure 12 — Figure 12: Qwen-3.8-Max on SWE-bench Verified. Source, length, and coverage change together. Validation denominators reflect the actual graded tasks.

Panel of Figure 13 — Figure 13: DeepSeek-V4-Pro on SWE-bench Verified. Checkpoint-saving cadence changes training-system overhead rather than the learning algorithm.

Panel of Figure 14 — Figure 14: Kimi-K3 on SWE-bench Verified. The per-issue quota changes; the accompanying matched-task comparison distinguishes outcome changes from grading coverage.

Panel of Figure 15 — Figure 15: GPT-5.6-Sol on SWE-bench Verified. Subset and full-scope validation retain their actual denominators.

Panel of Figure 16 — Figure 16: GLM-5.2 on SWE-bench Verified. The fixed-weight Harness comparison preserves the aggregate score while changing the success set.

Benchmark	Initial Data Resources
Terminal-Bench 2.1	TMAX, Recursive Task Synthesis Trajectories, TerminalTraj 1 1 1 TMAX: https://huggingface.co/datasets/allenai/TMax-SFT-16.5K ; Recursive Task Synthesis Trajectories: https://huggingface.co/datasets/Zhongzhi1228/Recursive-Task-Synthesis-Trajectories ; TerminalTraj: https://huggingface.co/datasets/m-a-p/TerminalTraj .
SWE-bench Verified	Orchard, SWE-Dev-Train, Nebius SWE, Nvidia Open SWE 2 2 2 Orchard: https://huggingface.co/datasets/microsoft/Orchard ; SWE-Dev-train: https://huggingface.co/datasets/zai-org/SWE-Dev-train ; Nvidia_open_swe: https://huggingface.co/datasets/nvidia/Open-SWE-Traces ; Nebius_swe: https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories .
$\tau^{3}$ -Bench	AReal-tau2-data, Nemotron-SFT-Agentic-v2, Simia-Tau-SFT 3 3 3 AReal-tau2: https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data ; Nemotron-SFT-Agentic-v2: https://huggingface.co/datasets/nvidia/Nemotron-SFT-Agentic-v2 ; Simia-Tau-SFT: https://huggingface.co/datasets/Simia-Agent/Simia-Tau-SFT-90k-Hermes .
Toolathlon	DIVE-SFT 4 4 4 DIVE-SFT: https://huggingface.co/datasets/dive-team/DIVE-SFT-20K .
AIME	Nemotron-SFT-MATH, OpenR1-Math, OpenThoughts3 5 5 5 Nemotron-SFT-MATH: https://huggingface.co/datasets/nvidia/Nemotron-SFT-Math-v4 ; OpenR1-Math: https://huggingface.co/datasets/open-r1/OpenR1-Math-220k ; OpenThoughts3: https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M .

Appendix

Table 5: Initial data resources provisioned for each benchmark.

Benchmark	Evaluation settings
Terminal-Bench 2.1	Maximum 200 agent turns; 65,536-token context; 4,096 maximum output tokens; temperature 1.0; thinking disabled; one run per task.
SWE-bench Verified	256K-token context; temperature 1.0; thinking enabled; one run per task.
AIME	16,384 maximum output tokens; temperature 0.6; thinking enabled; average@4.
Toolathlon	Maximum 100 agent turns; 256K-token context; 8,192 maximum output tokens; temperature 1.0; thinking disabled; pass@3.
$\tau^{3}$ -Bench	Maximum 200 agent turns; 256K-token context; temperature 0; thinking disabled; one run per task.

Appendix

Table 6: Evaluation settings shared by validation and test runs.

Figure 17: Elapsed research time by activity for six research-agent runs on Toolathlon, with one bar per run. waiting and submit+grade together account for 66 % of the GPT-6-Astra run and 88 % of the Kimi-K3 run. For GLM-5.2, a one-hour gap in the episode records leaves 9.0 % of elapsed time classified as uncovered .

Figure 18: Elapsed research time by activity for six research-agent runs on AIME. waiting and submit+grade together account for 76–91 % of elapsed time across the six runs. Evaluation episodes include submissions of individual model checkpoints or weight-averaged models on either the full question set or a prefix selected by the limit parameter. These episodes are classified as submit+grade . The uncovered share never exceeds 0.6 %.

Figure 19: Elapsed research time by activity for six research-agent runs on

\tau^{3}

-Bench. waiting and submit+grade together account for 69–92 % of elapsed time across the six runs. Some candidates are constructed by interpolating the weights of existing models without additional training. Their evaluation episodes are classified as submit+grade .

Figure 20: Elapsed research time by activity for six research-agent runs on Terminal-Bench 2.1. The DeepSeek-V4-Pro episode records contain qualitative markers such as “session open”, “first full split”, “mid”, and “last”, but no durations. Its row is retained with an explanation, without a time breakdown. Across the five plotted runs, waiting and submit+grade together account for 35–95 % of elapsed time. GPT-6-Astra has the lowest combined share, with 36 % of its elapsed time classified as uncovered .

Figure 21: Elapsed research time by activity for six research-agent runs on SWE-bench Verified. The GPT-6-Astra episode records specify episode order but contain only one timestamp, so its row is shown without a time breakdown. Each episode in the other five runs has both a start and an end timestamp; no episode duration is interpolated. Across these five runs, waiting and submit+grade together account for 63–88 % of elapsed time.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
Jul 28, 2026Fanqing Meng, Lingxiao Du, Qiguang Chen +4LLM Agent Evaluation LLM Agent Self-Improvement
1Evolvent AI · 2National University of Singapore
RSIGym: A Flexible Environment for Recursive Self-Improvement
Oct 7, 2026Fanqing Meng, Lingxiao Du, Haocheng Lu +6LLM Agent Self-Improvement Recursive Self-Improvement
Evolvent AI · National University of Singapore
EvalResearchBench: Can AI Agents Design Their Own Evaluations?
Oct 3, 2026Yaolun Zhang, Tianyi Xu, Yujie Zhao +3Automated Evaluation Benchmark Construction
Oregon State University · AG2 AI · University of California, San Diego +1

Jul 28, 2026J/K move · Enter open · S save

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao+3

1Evolvent AI · 2National University of Singapore

LLM Agent Evaluation LLM Agent Self-Improvement AI Agent Benchmarks LLM Post-Training Recursive Self-Improvement

Open paper AskPDF arXiv

Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents' research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.

Abstract

LLM Agent Training

Explore similar work

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

RSIGym: A Flexible Environment for Recursive Self-Improvement

EvalResearchBench: Can AI Agents Design Their Own Evaluations?