AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

Ruifeng Yuan; Yizhi Li; Yaxin Du; Fengyu Cai; Yiqi Liu; Hou Pong Chan; Chenghua Lin; Yun Chen

cs.CLSep 30, 2026

AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

Authors: Ruifeng Yuan, Yizhi Li, Yaxin Du, Fengyu Cai, Yiqi Liu, Hou Pong Chan, Chenghua Lin, Yun Chen,

Organizations: The Hong Kong Polytechnic University · IQuest Research · Shanghai Jiaotong University · TU Darmstadt ·

Abstract

Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to specific research capabilities. In this work, we isolate and systematically evaluate Data Intelligence: an agent's ability to understand, manipulate, and improve the data that shapes model capabilities. We introduce AutoDataBench, a controlled testbed built on a conceptual framework of data intelligence spanning data diagnosis, data organization, and data construction, instantiated through three highly curated optimization tasks while holding non-data factors fixed. Across tool use, retrieval, and knowledge injection, we evaluate frontier LLMs' ability to improve training data through iterative experimentation under task-specific resource budgets. Beyond optimization performance, we ask: do LLMs understand what their data interventions do? We compare predictions made before training with observed outcomes to seek evidence of data-effect reasoning beyond trial and error, and explore whether iterative feedback helps LLMs better understand how changes to training data affect model performance. Finally, we show that reusing AutoDataBench trajectories for mid-training improves downstream coding performance, highlighting its value in both evaluating data intelligence and generating high-quality training data. Code and resources are available at https://github.com/AutoDataBench/AutoDataBench.

Figures & tables

Figure 1: Overview of AutoDataBench. (a) Three dimensions of data intelligence: data diagnosis and repair, data organization, and data construction. (b) Agents iteratively propose data interventions, construct training datasets, and use feedback from a fixed training and evaluation pipeline. Final evaluation measures target-task improvement and generalization to held-out evaluation distributions.

		Tool use			Retrieval			Knowledge
Method	Overall	ID mean	ID best	OOD	ID mean	ID best	OOD	ID mean	ID best	OOD
Baseline	47.98	70.45	–	55.18	34.38	–	31.13	39.10	–	47.84
Expert	56.89	81.65	–	60.23	40.62	–	34.76	48.40	–	48.55
DeepSeek-V4-Flash	55.20	$80.97\!\pm\!0.59$	81.40	57.26	$36.22\!\pm\!1.26$	37.59	29.53	$48.40\!\pm\!7.71$	57.30	49.38
Qwen-3.7-Max	56.80	$80.82\!\pm\!0.24$	81.00	57.90	$39.22\!\pm\!2.51$	41.25	30.18	$50.37\!\pm\!1.31$	51.40	47.33
Claude-4.7	58.31	$81.00\!\pm\!0.15$	81.15	58.95	$38.99\!\pm\!2.31$	40.86	29.61	$54.93\!\pm\!0.57$	55.40	50.37

Table 1: Performance across tasks (scores

\times 100

). Overall averages the three task-wise ID means (Novel for knowledge injection). For each LLM and task, Mean/Best denote the mean/maximum across three standard runs;

\pm

indicates sample standard deviation. OOD reports mean hidden-evaluation scores at ID-selected checkpoints: BFCL for tool use, held-out retrieval datasets for retrieval, and Retention for knowledge injection. Bold marks the best LLM result.

Figure 2: Best-so-far ID scores in the best-performing standard run of each LLM on each task (scores

\times 100

; Knowledge uses Novel). Runs are selected by their maximum ID score, with ties resolved by the earliest run. Each trajectory ends at its own last successful evaluation; failed evaluations are excluded. These selected runs illustrate optimization behavior, not average performance or equal-compute comparisons.

Figure 3: Predicted and observed scores across training evaluations for GPT-5.6-Sol and Kimi-3. Horizontal lines show the mean best score across standard runs. The first evaluation has no forecast.

	MBPP Base	MBPP Plus	CRUXEval Input	LiveCodeBench v5	SWE-bench Multilingual
Control	85.45	72.75	72.00	44.89	27.67
+ Auto-Research	86.24	73.28	74.12	45.80	33.33
$\Delta$	+0.79	+0.53	+2.12	+0.91	+5.66

Table 2: Benchmark as Data Engine: Transfer of auto-research trajectories to general coding tasks. Both models use a 5B-token mid-training budget followed by identical SFT.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Operator	Operation	Attempt probability	Application count
Argument type error	Convert a numeric or Boolean argument value to a string.	12%	1,516
Missing argument	Delete one supplied argument, prioritizing required parameters and otherwise selecting an optional parameter.	12%	4,139
Extra argument	Add a parameter borrowed from another tool in the same example that is absent from the called tool’s schema.	10%	2,617
Wrong function label	Replace a called function with the most similar alternative among the available tools, using cosine similarity of embeddings of tool names and descriptions.	12%	3,132
Argument value corruption	Replace an argument value with a different value of the same type drawn from the pool of values for the same parameter name.	12%	4,140
Schema inconsistency	Mark an omitted parameter as required, or change the declared type of a supplied parameter.	8%	1,910

Appendix

Table 3: Corruption operators for the tool-use training pool. Attempt probabilities apply to callable examples; application counts record operations that pass their applicability checks. Counts overlap because an example may receive multiple corruptions.

LLM	Run	Evals	First ID	Best ID	At	Last ID	OOD
DeepSeek-V4-Flash	1	10	70.00	81.20	9	81.15	53.83
	2	10	75.90	80.30	8	80.20	59.16
	3	10	71.45	81.40	8	81.25	58.80
Qwen-3.7-Max	1	10	71.35	81.00	7	79.80	56.85
	2	10	77.20	80.55	10	80.55	58.56
	3	10	71.35	80.90	10	80.90	58.28

Appendix

Table 4: Tool-use results for individual standard runs (scores

\times 100

). ID is macro AST accuracy; OOD is BFCL accuracy.

LLM	Run	Evals	First ID	Best ID	At	Last ID	OOD
DeepSeek-V4-Flash	1	6	32.99	35.10	6	35.10	28.37
	2	11	33.98	37.59	11	37.59	30.24
	3	13	30.76	35.98	13	35.98	29.99
Qwen-3.7-Max	1	10	34.85	41.25	9	40.86	30.05
	2	12	35.14	36.41	11	36.09	29.13
	3	10	34.86	39.99	9	38.73	31.35

Appendix

Table 5: Retrieval results for individual standard runs (scores

\times 100

). ID is mean nDCG@10; OOD is mean nDCG@10 on the five held-out datasets.

LLM	Run	Evals	First ID	Best ID	At	Last ID	OOD
DeepSeek-V4-Flash	1	3	57.30	57.30	1	55.20	49.55
	2	11	41.80	43.90	7	42.90	48.20
	3	16	43.80	44.00	2	41.40	50.39
Qwen-3.7-Max	1	13	46.30	48.90	12	48.80	47.34
	2	10	49.00	51.40	9	49.10	47.11
	3	10	48.10	50.80	10	50.80	47.53

Appendix

Table 6: Knowledge Injection results for individual standard runs (scores

\times 100

). ID is Novel accuracy; OOD is hidden Retention accuracy.

Task	Scored runs	Improved runs	Mean gain
Tool use	21	21	8.11
Retrieval	21	21	6.46
Knowledge injection	21	19	4.91

Appendix

Table 7: Refinement across all scored standard runs. Improvement compares each run’s best ID score with its first successful evaluation. Mean gains include runs with no improvement and are measured in score points.

Agent Benchmarks

Agent benchmarks measure how well AI systems plan, use tools and memory, and complete tasks in interactive environments.

See all 122 papers

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
Jul 28, 2026Fanqing Meng, Lingxiao Du, Qiguang Chen +4LLM Agent Evaluation LLM Agent Self-Improvement
1Evolvent AI · 2National University of Singapore
AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?
Sep 28, 2026Haotian Luo, Haoyu Wang, Zeyu Qin +5AI Agent Benchmarks Synthetic Benchmark Generation
HKUST · SLAI · NTU
AutoMind: Adaptive Knowledgeable Agent for Automated Data Science
Jun 12, 2025Yujie Luo, Yixin Ou, Jingsheng Zheng +9Data Analysis Agents LLM Agents
Zhejiang University · Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph · Ant Group

Jul 28, 2026J/K move · Enter open · S save

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao+3

1Evolvent AI · 2National University of Singapore

LLM Agent Evaluation LLM Agent Self-Improvement AI Agent Benchmarks LLM Post-Training Recursive Self-Improvement

Open paper AskPDF arXiv

Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents' research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.

Abstract

Agent Benchmarks

Explore similar work

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?

AutoMind: Adaptive Knowledgeable Agent for Automated Data Science