Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study

Ruomeng Ding; Qianli Shen; Zhaoyang Han; Meixin Chen; Daoyuan Chen; Yaliang Li

cs.CLOct 9, 2026

Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study

Authors: Ruomeng Ding, Qianli Shen, Zhaoyang Han, Meixin Chen, Daoyuan Chen, Yaliang Li

Organizations: University of North Carolina at Chapel Hill · Alibaba Group · Peking University

Abstract

LLM data engineering requires iterative experimentation to determine how data selection, rewards, and curricula affect model learning, making it a natural setting for AI-for-AI (AI4AI) research. Can agents use experimental feedback to progressively improve strategies, and do the resulting strategies improve target-model performance on held-out data? We study how agents use experimental feedback to develop data-engineering strategies under fixed training and evaluation protocols. Our Agentic Data Engineering (ADE) framework lets agents pursue, revise, or combine research directions using cumulative findings. Strategies are selected by in-loop scores and evaluated on held-out benchmarks. Across four tasks-math and code data selection, math reward design, and math curriculum learning-selected strategies achieve absolute gains over generic baselines of up to 13.34% on AIME25 Pass@32 and 5.85% averaged across four held-out benchmarks under fixed training conditions. In-loop scores improve over successive experiments, while held-out trajectories show that later in-loop gains do not always improve generalization. Comparisons of research organization and feedback access favor combining evidence analysis with cumulative findings. Research trajectories show how earlier findings inform strategy revisions and combinations across research directions.

Figures & tables

Figure 2: ADE under Harness control: Agents develop strategies using In-loop feedback and Memory; the Engine executes controlled experiments. Oversight and held-out evaluation remain external.

Task	Domain	Training paradigm	Agent-controlled strategy	Generic baseline
Data Selection	Math	Long-CoT SFT	Subset selection	Random subset
Data Selection	Code	Long-CoT SFT	Subset selection	Random subset
Reward Design	Math	RFT	Reward + group credit	Outcome-only reward
Curriculum Learning	Math	RFT	Training-data schedule	Random ordering

Table 2: Controlled settings for the four data-engineering tasks.

Method	In-loop	Held-out Benchmark
Data Selection (Math)	AIME24	MATH-500	MATH-S	AIME25	HMMT	Macro Avg.
Base model	36.67 / 11.15	85.80 / 76.00	86.00 / 78.00	30.00 / 7.71	16.67 / 2.29	54.62 / 41.00
Generic Baseline	43.33 / 12.60	87.60 / 78.00	88.00 / 77.87	33.33 / 11.15	30.00 / 7.40	59.73 / 43.61
ADE ( $C=1$ )	56.67 / 17.81	90.20 / 80.00	88.80 / 79.73	46.67 / 14.90	36.67 / 6.67	65.58 / 45.32
ADE ( $C=3$ )	56.67 / 16.98	87.20 / 77.33	87.60 / 78.73	43.33 / 16.25	36.67 / 7.08	63.70 / 44.85
Data Selection (Code)	LCB $\leq$ 03/30	LCB $>$ 03/30	HumanEval+	Codeforces	CodeElo	Macro Avg.

Table 3: Main results for in-loop-selected strategies (%). Cells report Pass@

K

/ Average@

K

Figure 3: Research progress and cost for $C=1$ (blue) and $C=3$ (orange): rows show best-so-far in-loop scores, held-out scores of in-loop-selected models, cumulative token cost, and GPU-hours with allocation occupancy. Lines average three Runs per task and $C$ ; shading shows min–max ranges.

Figure 4: Estimated research-stage times at

C=1

. Each bar shows a task’s stage.

Figure 5: Data Selection–Math research organization and feedback comparisons. Each panel shows Pass@

K

(left axis) and Average@

K

(right axis); held-out scores average four benchmarks.

Task	Agent-developed strategy
Data Selection (Math)	Select valid trajectories with direct, low-redundancy reasoning. Filter by training cutoff and Judge assessments, then deduplicate and select under fixed size, length-quota, and token-budget constraints. $\rightarrow$ ( Appx. J.1 )
Data Selection (Code)	Stratified Repetition Control. Replace highly repetitive records with low-repetition candidates matched by prompt subtype, cutoff status, length band, and nearest token count. $\rightarrow$ ( Appx. J.2 )
Reward Design (Math)	Group-conditioned process credit. Use sibling correctness and corroboration to weight reasoning quality; require corroboration in all-wrong groups. $\rightarrow$ ( Appx. J.3 )
Curriculum Learning (Math)	Difficulty cycling with paired reinforcement. Prioritize unseen or less-used problems; after full-pool coverage, pair foundation and bridge batches using same-subject problems. Freeze the schedule before training. $\rightarrow$ ( Appx. J.4 )

Table 4: Selected strategies across four tasks; linked appendices provide detailed strategy scripts.

Figure 6: Data Selection (Code) trajectories: R/C denote revisited/combined parents; K1–K5, knowledge links; #, proposal order; star, selected Plan. Scores are in-loop Pass@3 (%).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Work / source	Intervention and execution	Research and evidence reuse	Selection and final evaluation
AutoResearch ( Karpathy, 2026 ) program.md	Agent edits training code, including architecture and optimization. Data loading, evaluator, and a five-minute training limit are fixed.	Agent proposes and runs changes. Git history and a results ledger record retained, discarded, and failed experiments.	Validation bits per byte guide retention. The documented loop does not specify a separate untouched final audit.
POISE ( Xia et al., 2026b ) §2–3	Agent changes RL optimization components under a common training interface, aligned budgets, and fixed experiment hyperparameters.	Search selects archive parents; lineage, reference algorithms, and reflective analyses inform new proposals. Failed or context-dependent designs remain in the archive.	A weighted reasoning-suite score guides evolution. The cited protocol does not specify an additional untouched evaluation after selection.
DataMaster ( Du et al., 2026 ) §3; App. B.2	Data discovery, composition, and processing change under a fixed learning algorithm. Benchmark settings fix the training code or the training script and base model.	Tree scheduling allocates experiments. Global Memory and a shared Data Pool carry findings and reusable artifacts across branches.	Downstream node scores guide search; results use best submissions. An additional untouched audit after selection is not specified in the cited sections.
EvoTrainer ( Chen et al., 2026a ) §3–4; App. E.1	RL recipes and diagnostic tools evolve. Humans bootstrap the stack and approve training launches and version promotion or pruning.	Agent diagnoses versions and proposes interventions. Persistent memory, reusable skills, and negative evidence support later branches.	Versions are compared under shared validation conditions. A separate untouched final split is not specified in the cited evaluation protocol.

Appendix

Table 5: Research boundaries in representative autonomous research systems. Source locations identify the basis for each row.

Work / source	Intervention and execution	Research and evidence reuse	Selection and final evaluation
RSIBench-Data ( Meng et al., 2026 ) §3, 5–6	Agents construct training experience and choose whitelisted configurations. The base model and training, serving, and evaluation infrastructure are fixed.	Scores, trajectories, verifier outputs, and attempt history inform revision, stopping, and checkpoint selection; recorded trajectories support process analysis.	Agent selects a checkpoint before official scoring in fresh environments. Selection and official evaluation use the same task subset; §6 explicitly distinguishes this from held-out tasks.
AI4AI-Bench ( Chi et al., 2026 ) §2.3; App. A	Agents edit repository code and run proxy experiments. Submitted code is frozen, then executed from the prescribed starting state under a fixed final evaluator.	The exploration phase permits adaptive proxy use; source code alone is submitted. The benchmark accommodates different researcher systems rather than prescribing one memory mechanism.	Training tasks score up to three recent checkpoints and report the best. Proxy/final separation varies by task: OPD uses different benchmarks; BTRM’s final set includes its exposed proxy subset.
DataAgent ( Luo et al., 2026 ) §3–4; App. C–D	Agents synthesize and curate data with a fixed teacher, student, and training setup within each configuration.	Iterative agents inspect public scores and failure examples to revise datasets. Public-score history determines the selected submission.	After research, the submitted dataset is used to retrain the student and evaluate on private held-out data. This evaluates the selected dataset through a fresh training run.
ADE (ours) § 3 – 4	Agents change selection, curriculum, or reward strategies. The Harness enforces a fixed protocol and task-specific training conditions.	Agents revise or combine strategies using attributed findings, including observations from unsuccessful experiments. Records link proposals to the evidence available at decision time.	In-loop scores select the strategy and checkpoint. The same frozen checkpoint is evaluated on benchmarks or splits withheld from in-loop evaluation; results do not return to research or selection.

Appendix

Table 6: Research boundaries in data-engineering studies and benchmarks, including this work. Freezing a submission, retraining from that submission, and evaluating on data withheld from in-loop evaluation are distinct protocol choices.

Role	Available evidence	Required output and authority
Coordinator	Frozen RM, remaining budget, Plan catalog, ranking and admitted source evidence	Research Plan, relation to prior work, and precommitted expectations.
Builder	Accepted Plan, frozen Plan-local evidence, source strategies, fixed task conditions and development feedback	Executable selection, reward, or curriculum strategy and implementation rationale.
Analyzer	Realized intervention, permitted training and in-loop behavior; review results only at synthesis	Review design, then source-linked analysis and findings; no access to the Plan or its expectations.
Plan Summarizer	Parent PM, immutable Plan, current Trial outcome, realization, comparison records and accepted analysis	Complete successor PM and hypothesis adjudication.
Run Summarizer	Parent RM, one queue-head PM update, and same-version Run catalog and ranking	Complete successor RM with cross-Plan findings and unresolved questions.
Review Worker	Exact records and Analyzer-designed rubrics	Record-level judgments; no research choice or memory publication.

Appendix

Table 7: Role-specific evidence and decisions. Only the Harness publishes canonical state. No research role receives held-out evaluation records as part of its declared inputs.

Record	Bound contents and use
Identity and catalog	Run/Coordinator/Plan/Trial scope, artifact identity, kind, availability, source experiment and record-level attribution.
Realized intervention	Executed strategy and final realization report; selected trajectories, reward/group evidence, or curriculum batches and consumed problem identities.
Training and behavior	Normalized training telemetry, task-permitted rollout records and complete fixed-checkpoint in-loop responses with assessments.
Review overlay	Frozen packet and coverage, added only for synthesis; missing and failed rows remain explicit.
Excluded context	Accepted Plan and rationale, PM/RM, planning comparators, objective-comparison record, online validation results, held-out records, and undeclared operational files.

Appendix

Table 8: Analyzer package contents and exclusions.

Stage	Abstract transition	Scientific boundary
Hypothesis commitment	$\mathcal{V}_{p}\mapsto(\eta_{p},\rho_{p})$	The Plan commits its question, mechanism, expectations, rejection conditions, and controls. Each Trial inherits that commitment. The catalog in $\mathcal{V}_{p}$ exposes active directions to help avoid duplicate proposals.
Intervention admission	$(\psi_{i},\Gamma_{\mathcal{Z}})\mapsto z_{i}\in\mathcal{Z}$	Interface and execution checks establish that the artifact can run under the fixed configuration; the permission boundary restricts it to the declared selection, reward, or curriculum surface.
Measurement	$(z_{i},\mathcal{B})\mapsto(\theta_{i},\xi_{i},e_{i},s_{i})$	Training and evaluation execute outside the proposing agent. Recovery may replay the same frozen logical experiment, but cannot change $\psi_{i}$ , $z_{i}$ , or $\mathcal{B}$ .
Blinded evidence interpretation	$(z_{i},\xi_{i},e_{i})\mapsto(q_{i},r_{i},a_{i})$	Without receiving $\psi_{i}$ , the Analyzer commits its review design before the Review Worker tool is called, then binds the returned review to primary evidence in the blinded interpretation.
Hypothesis adjudication	$(\psi_{i},a_{i},e_{i},s_{i})\mapsto\widetilde{\delta}_{i}$	Plan-level synthesis compares the independently formed interpretation with the precommitted expectation and rejection condition. It may qualify the claim but cannot rewrite it.
Plan update	$(\mathcal{K}^{P}_{p},\tau_{i},\delta_{i})\mapsto\mathcal{K}^{P+}_{p}$	Findings with valid source references cross the local knowledge boundary with their assessments preserved; the Trial is admitted exactly once.

Appendix

Table 9: Stage semantics of a ADE research iteration.

Parameter	Data Selection–Math	Data Selection–Code	Math RFT tasks
Target model	Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	Qwen2.5-0.5B
Training source	OpenThoughts mixed pool	OpenThoughts mixed pool	Fixed MATH training set
Source size	3,840 trajectories	3,840 trajectories	3,072 problems
Intervention	Select 384 trajectories	Select 384 trajectories	Reward/group credit or data schedule
Learning method	Full-parameter SFT	Full-parameter SFT	GRPO

Appendix

Table 10: Target-learning configuration.

Parameter	Value
Training type	Full-parameter supervised fine-tuning
Selected examples	384
Maximum epochs	13
Effective global batch	32 examples
Optimizer steps per full epoch	12
Optimizer	AdamW

Appendix

Table 11: Shared SFT configuration for the math and code data-selection settings.

Parameter	Value
Algorithm	GRPO
Training problems	3,072
Actor updates / data epochs	100 / at most 20
Prompt groups per update	256
Sampled responses per prompt	8
Generated trajectories per update	2,048

Appendix

Table 12: Shared math RFT training configuration.

Capacity	Coordinators	Plans per Coordinator	Trials per Plan	Maximum candidates
$C=1$	1	5	1	5
$C=3$	3	5	1	15

Appendix

Table 13: Research capacity per Run, shared by all four tasks. Counts exclude the base-target evaluation and generic reference.

Setting	Evaluation stage	Benchmark	Problems	$K$
Data Selection–Math	Online checkpoint evaluation	AIME 2024	30	8
Data Selection–Math	In-loop Trial evaluation	AIME 2024	30	32
Data Selection–Math	Held-out evaluation	AIME 2025, HMMT	30 each	32
Data Selection–Math	Held-out evaluation	MATH-500, MATH-S	500 each	3
Data Selection–Code	Online checkpoint evaluation	LiveCodeBench v5, on/before 2024-03-30	439	1
Data Selection–Code	In-loop Trial evaluation	LiveCodeBench v5, on/before 2024-03-30	439	3

Appendix

Table 14: Benchmarks by evaluation stage.

K

is the number of generated responses per problem.

Dataset	Rows	Start date	End date
LiveCodeBench v5, in-loop split	439	2023-05-07	2024-03-30
LiveCodeBench v5, held-out split	441	2024-04-06	2025-01-04

Appendix

Table 15: LiveCodeBench v5 temporal split used for code data selection.

Method	In-loop	Held-out
Data Selection–Math	AIME24	MATH-500	MATH-S	AIME25	HMMT	Macro Avg.
Base model	36.67 / 11.15	85.80 / 76.00	86.00 / 78.00	30.00 / 7.71	16.67 / 2.29	54.62 / 41.00
Generic Baseline	43.33 / 12.60	87.60 / 78.00	88.00 / 77.87	33.33 / 11.15	30.00 / 7.40	59.73 / 43.61
ADE ( $C=1$ )	51.11 / 16.46	87.87 / 77.98	88.60 / 78.71	40.00 / 13.85	31.11 / 6.18	61.89 / 44.18
STD	5.09 / 1.17	2.08 / 1.82	0.53 / 0.92	6.67 / 2.76	5.09 / 0.47	3.22 / 1.25
ADE ( $C=3$ )	54.44 / 16.01	87.40 / 77.00	87.53 / 77.24	41.11 / 12.57	31.11 / 5.87	61.79 / 43.17

Appendix

Table 16: Across-Run means and sample STD (%). Cells report Pass@

K

/ Average@

K

for all tasks. Each ADE mean row is followed by its sample STD over three Runs; bold marks per-metric best scores, excluding STD rows.

Method	Run	In-loop	Held-out
		AIME24	MATH-500	MATH-S	AIME25	HMMT	Macro Avg.
Base model	1	36.67 / 11.15	85.80 / 76.00	86.00 / 78.00	30.00 / 7.71	16.67 / 2.29	54.62 / 41.00
Generic Baseline	1	43.33 / 12.60	87.60 / 78.00	88.00 / 77.87	33.33 / 11.15	30.00 / 7.40	59.73 / 43.61
ADE ( $C=1$ )	1	50.00 / 15.83	87.20 / 77.47	88.00 / 77.93	40.00 / 15.94	26.67 / 6.15	60.47 / 44.37
ADE ( $C=1$ )	2 ∗	56.67 / 17.81	90.20 / 80.00	88.80 / 79.73	46.67 / 14.90	36.67 / 6.67	65.58 / 45.32
ADE ( $C=1$ )	3	46.67 / 15.73	86.20 / 76.47	89.00 / 78.47	33.33 / 10.73	30.00 / 5.73	59.63 / 42.85

Appendix

Table 17: Data Selection–Math run-level results (%).

\ast

marks the Best Run within each ADE setting .

Method	Run	In-loop	Held-out
		LCB $\leq$ 03/30	LCB $>$ 03/30	HumanEval+	Codeforces	CodeElo	Macro Avg.
Base model	1	40.55 / 27.49	24.26 / 15.95	85.98 / 74.39	7.28 / 3.93	8.58 / 4.25	31.53 / 24.63
Generic Baseline	1	39.86 / 26.35	25.85 / 18.44	85.98 / 70.53	8.57 / 4.85	9.07 / 5.23	32.37 / 24.76
ADE ( $C=1$ )	1 ∗	45.79 / 31.74	30.39 / 20.11	87.20 / 72.76	7.71 / 4.00	8.82 / 4.66	33.53 / 25.38
ADE ( $C=1$ )	2	42.60 / 28.70	28.80 / 19.27	84.76 / 72.15	8.78 / 5.00	9.31 / 4.98	32.91 / 25.35
ADE ( $C=1$ )	3	43.51 / 30.68	27.66 / 19.58	82.32 / 69.31	7.28 / 4.14	6.86 / 3.84	31.03 / 24.22

Appendix

Table 18: Data Selection–Code run-level results (%).

\ast

marks the Best Run within each ADE setting .

Method	Run	In-loop	Held-out
		MATH-S	MATH-500	AIME24	AIME25	HMMT	Macro Avg.
Base model	1	39.40 / 20.87	36.60 / 19.93	0.00 / 0.00	3.33 / 0.10	0.00 / 0.00	9.98 / 5.01
Generic Baseline	1	49.00 / 34.13	46.80 / 30.13	3.33 / 0.21	0.00 / 0.00	0.00 / 0.00	12.53 / 7.59
ADE ( $C=1$ )	1	50.40 / 34.73	44.60 / 28.60	13.33 / 0.52	0.00 / 0.00	0.00 / 0.00	14.48 / 7.28
ADE ( $C=1$ )	2 ∗	50.20 / 36.33	46.40 / 30.80	3.33 / 0.10	6.67 / 0.21	3.33 / 0.10	14.93 / 7.80
ADE ( $C=1$ )	3	52.60 / 36.20	45.80 / 30.60	6.67 / 0.42	3.33 / 0.10	0.00 / 0.00	13.95 / 7.78

Appendix

Table 19: Reward Design–Math run-level results (%).

\ast

marks the Best Run within each ADE setting .

Method	Run	In-loop	Held-out
		MATH-S	MATH-500	AIME24	AIME25	HMMT	Macro Avg.
Base model	1	39.40 / 20.87	36.60 / 19.93	0.00 / 0.00	3.33 / 0.10	0.00 / 0.00	9.98 / 5.01
Generic Baseline	1	49.00 / 34.13	46.80 / 30.13	3.33 / 0.21	0.00 / 0.00	0.00 / 0.00	12.53 / 7.59
ADE ( $C=1$ )	1	51.00 / 36.13	47.20 / 30.80	10.00 / 0.62	0.00 / 0.00	3.33 / 0.10	15.13 / 7.88
ADE ( $C=1$ )	2	50.80 / 34.73	46.80 / 30.00	6.67 / 0.21	0.00 / 0.00	0.00 / 0.00	13.37 / 7.55
ADE ( $C=1$ )	3 ∗	51.80 / 36.40	45.80 / 30.60	6.67 / 0.31	0.00 / 0.00	3.33 / 0.10	13.95 / 7.75

Appendix

Table 20: Curriculum Learning–Math run-level results (%).

\ast

marks the Best Run within each ADE setting .

Method	AIME24	MATH-500	MATH-S	AIME25	HMMT	Macro Avg.
Unified Agent	50.00 / 14.38	87.00 / 76.33	88.20 / 76.33	43.33 / 12.81	33.33 / 5.52	62.97 / 42.75
Independent Search	50.00 / 15.52	89.20 / 78.60	88.80 / 77.13	36.67 / 13.33	36.67 / 7.29	62.83 / 44.09
Outcome-only Feedback	46.67 / 14.17	85.80 / 76.33	88.00 / 76.93	40.00 / 14.58	26.67 / 6.35	60.12 / 43.55
Raw Evidence Feedback	50.00 / 15.10	88.00 / 78.00	87.60 / 77.47	36.67 / 11.77	30.00 / 5.52	60.57 / 43.19
ADE ( $C=1$ )	56.67 / 17.81	90.20 / 80.00	88.80 / 79.73	46.67 / 14.90	36.67 / 6.67	65.58 / 45.32

Appendix

Table 21: Data Selection–Math comparison details (%). Cells report Pass@

K

/ Average@

K

;

K=32

on AIME24, AIME25, and HMMT, and

K=3

on MATH-500 and MATH-S.

Task	$C$	Runs	GPT (M)	Qwen (M)	GPT ($)	Qwen ($)	Total ($)
Data Selection–Math	1	3	4.67	34.32	49.58	12.55	62.13
Data Selection–Math	3	3	15.58	106.46	175.64	39.91	215.56
Data Selection–Code	1	3	5.07	36.51	57.09	14.92	72.01
Data Selection–Code	3	3	15.19	131.59	181.41	58.49	239.90
Reward Design–Math	1	2	5.71	43.39	70.57	18.57	89.13
Reward Design–Math	3	2	16.16	133.87	204.91	60.29	265.20

Appendix

Table 22: Estimated mean token usage and API-equivalent cost per Run. Token counts are in millions (M); costs are in USD. Total sums the two model-cost columns before rounding.

Model	Input	Cached input	Output
GPT-5.6 Sol	4.00	0.40	20.00
Qwen3.6-35B-A3B	0.248	–	1.485

Appendix

Table 23: Fixed API rates used for cost estimation, dated September 9, 2026 (USD per million tokens ).

Figure 7: Estimated research-stage times at

C=3

. Bars show time shares; labels above each bar give mean hours per Plan, including waiting.

Plan	Strategy	Formal parent(s)	Pass@3	Average@3
C1/P1	Complete targets	—	42.14	29.76
C1/P2	Judge validity	C1/P1	42.60	28.78
C1/P3	Stratified Repetition Control	C2/P1	48.06	33.03
C1/P4	Wrapper closure	C1/P3	43.74	32.95
C1/P5	Literal image-marker removal	C1/P3	45.56	31.66
C2/P1	Code-contract priority	—	44.87	32.50

Appendix

Table 24: All 15 Plans in the Data Selection–Code case-study Run. Scores are in-loop percentages. A dash denotes a new direction with no formal Plan parent. Bold scores mark the in-loop-selected strategy.

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 305 papers

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
Jul 28, 2026Fanqing Meng, Lingxiao Du, Qiguang Chen +4LLM Agent Evaluation LLM Agent Self-Improvement
1Evolvent AI · 2National University of Singapore
Exploring Autonomous Agentic Data Engineering for Model Specialization
May 28, 2026Yujie Luo, Xiangyuan Ru, Jingsheng Zheng +10Large Language Model-Guided Optimization Synthetic Data Generation for Language Models
Zhejiang University · Platform and Content Group, Tencent
AutoData: Agentic Search for Pre-training Data Selection
Sep 17, 2026Yan Meng, Dhruv Srikanth, Bingchen Zhao +2Language Model Pretraining Automated Algorithm Discovery
University of Amsterdam · Weco AI

Jul 28, 2026J/K move · Enter open · S save

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao+3

1Evolvent AI · 2National University of Singapore

LLM Agent Evaluation LLM Agent Self-Improvement AI Agent Benchmarks LLM Post-Training Recursive Self-Improvement

Open paper AskPDF arXiv

Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents' research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.

Abstract

Autonomous Research Agents

Explore similar work

RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

Exploring Autonomous Agentic Data Engineering for Model Specialization

AutoData: Agentic Search for Pre-training Data Selection