TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

Jiarui Yan; Weiwei Sun; Sijie Li; Wenhan Li; Yiming Yang

cs.LGAug 26, 2026

TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

Authors: Jiarui Yan, Weiwei Sun, Sijie Li, Wenhan Li, Yiming Yang

Organizations: Carnegie Mellon University

Abstract

Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their own feedback, yet on most competitions they still finish below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4{,}465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, and the labeling models, together with an open-source toolkit that turns a run from any command-line agent into a TraceML trajectory and reads it against the human cohorts.

Figures & tables

Figure 1: Stylized trajectories with per-edit action ribbons, one panel per cohort. Humans mix actions, pivot when behind, and reopen abandoned lines; Codex maintains one solution through small submission-side edits; MLEvolve mutates its model in place. § 4 quantifies each behavior.

Benchmark / Dataset	Score Traj	Code Traj	Human Traj	# Envs.	Horizon
MLE-bench ( Chan et al., 2024 )	✗	✗	⚫	75	24h
MLAgentBench ( Huang et al., 2024 )	✗	⚫	✗	13	–
AIRA ( Toledo et al., 2025 )	⚫	⚫	✗	22	24h
RE-Bench ( Wijk et al., 2025 )	✓	⚫	✓	7	8h
HCAST ( Rein et al., 2025 )	✗	✗	✓	189	1m–8h
SciCode ( Tian et al., 2024 )	✗	✗	✗	80 / 338	–

Table 1: Long-horizon agent benchmarks and datasets. TraceML is the only entry that keeps scored intermediate versions, their code, and task-matched human development. RE-Bench comes closest, pairing time-budgeted human and agent attempts, but on 7 bespoke environments with partial code; TraceML keeps the full code of every scored version across 134 competitions. Horizon is the per-task agent budget, or for TraceML the span of human development. ✓ , ⚫ , ✗ : full, partial, no coverage.

Figure 2: TraceML reconstruction pipeline: notebook histories, git commits, and search journals become one version-level representation. Appendix B.6 follows one real trajectory through every stage.

Subset	Comps	Trajectories	Snapshots	Avg snap/traj	Avg lines/snap
Human (public Kaggle notebook histories)
Grandmaster	114	423	16,973	40.1	555
Master	121	649	24,138	37.2	601
Expert	130	1,386	50,663	36.6	569
Contributor	133	1,932	54,619	28.3	494
Other / Unknown	40	75	3,090	41.2	295

Table 2: TraceML corpus statistics by source. Human trajectories are broken down by Kaggle author tier. Per-version code lines is total code lines divided by snapshots within the subset.

Figure 3: Action profiles at two resolutions. (a) PCA of per-trajectory coarse-action distributions, large markers are cohort centroids; (b) pairwise JSD between cohort action distributions; (c) coarse-action profiles in pipeline order; (d) fine-action usage relative to pooled humans, Codex against MLEvolve ( $\log_{2}$ axes) — the labeled off-diagonal clusters are the two scaffold loops and the shared gap.

Figure 4: Harness effect on six discipline features. Dashed rule: top human value; frames mark whether Codex

+

skill lands closer to it (blue) or farther from it (red) than prior Codex.

Figure 5: Running best leaderboard percentile over relative time, one panel per paired competition; shaded bands are the human cohort thresholds, markers the final percentile reached. The harness lift is uneven across competitions.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Outcome	Kernels	Character of the set
Retained	4,465	47.3% medalled
Removed: no in-window version	382	99.2% published entirely after the deadline, median 299 days late
Removed: failed content conditions	201	Too few versions, too short a span, or no score
Removed (all)	583	43.3% medalled

Appendix

Table 3: What the retention filter removes, from 5,048 candidate kernels. The removed set is dominated by post-deadline write-ups rather than by weak development trajectories, and its medal rate is comparable to that of the retained set, so the filter does not remove a weak tail.

Reuse measure	Full corpus (4,465)	Paired sample (430)
Has a fork parent	8.2%	8.6%
Near-duplicate code in $\geq 1$ version	23.7%	21.0%
Near-duplicate code in a majority of versions	—	7.9%
Near-duplicate code across the whole trajectory	—	3.3%

Appendix

Table 4: Visible reuse in the retained corpus and in the paired sample used for human-agent comparison. Near-duplicate code is code similarity

\geq 0.9

against another kernel; all observed near-duplicate links fall within the same competition, consistent with shared starter templates.

	Human	Codex	MLEvolve
Median versions per trajectory	20	18	5
Versions carrying a score	45%	$\approx$ 99%	$\approx$ 99%
Saved versions that are syntactically valid code	97.9%	—	—

Appendix

Table 5: How the version unit lines up across sources. Agent versions are submission-producing commits with adjacent identical-code commits collapsed, which is the analogue of a deliberate Kaggle save-version rather than a raw log entry.

				MLEvolve
Scope	Comps	Human traj.	Codex runs	runs	branches
Corpus (§ 3.1 )	134	4,465	—	—	—
Paired subset (§ 3.2 )	7	430	11	13	189
Twelve-hour scope (§ 4 )	7	430	10	3	107
Harness arms (§ 5.1 )	7	—	30	—	—

Appendix

Table 6: The three scopes. Human trajectories are split into a top cohort and the rest by leaderboard rank. MLEvolve branches share tree nodes, so the run count rather than the branch count sets the effective sample size.

Competition	Track	Transitions	Pivot rate
gquest	agent	1,017	2.1%
gquest	llm_v3	226	2.2%
ranzcr	agent	166	4.8%
ranzcr	llm_v3	102	2.0%
hms	agent	97	2.1%
aes2	llm_v3	64	1.6%

Appendix

Table 7: The ten twelve-hour Codex runs. “Pivot rate” is the marginal (unmatched) share of transitions carrying a pivot tag. Runs are ordered by length; the four shortest carry almost no information individually.

Transition	Actions	Intents	Magn.	Score
$v_{13}\to v_{14}$	model, training, housekeeping, infra	optimization + debugging	micro	$0.535\to 0.509$ (improving)
$v_{14}\to v_{15}$	data, augmentation, training, model	exploration + optimization	minor	unscored
$v_{15}\to v_{16}$	data, training, validation, housekeeping	debugging + optimization	micro	$\to 0.4989$ (trajectory best)
$v_{16}\to v_{17}$	training, model, infra	optimization	micro	$0.4989\to 0.5062$ (regressing)

Appendix

Table 8: Four consecutive transitions from one human trajectory, as represented in TraceML. Each transition carries actions, intents, a magnitude, and the score effect read from the leaderboard.

Figure 6: Example schema output for one version and its transition.

Annotation level (#tags)	Self-consistency	Cross-model	Teacher $\to$ Student
State coarse (8)	$\kappa{=}0.872$ / J= $0.969$	$\kappa{=}0.801$ / J= $0.954$	$F_{1}^{\mathrm{macro}}{=}0.978$
Action coarse (10)	J= $0.875$	J= $0.641$	$F_{1}^{\mathrm{macro}}{=}0.733$
Intent (6 classes)	$\kappa{=}0.834$	$\kappa{=}0.611$	$\mathrm{acc}{=}0.772$
Magnitude (4 levels)	$\kappa{=}0.921$	$\kappa{=}0.576$	$\mathrm{acc}{=}0.928$

Appendix

Table 9: Annotation reliability across teacher stability, cross-model agreement, and teacher–student transfer. We report Cohen’s

\kappa

and multi-label Jaccard (J) on held-out state and action annotations; teacher–student scores compare the released Qwen3-1.7B labelers with the gpt-5.4-mini teacher.

Cohort	$n$	6-class agreement	3-class collapse
Human	40	80.0%	82.5%
Codex	40	85.0%	87.5%
MLEvolve	20	75.0%	75.0%
Overall	100	81%	83% ( $\kappa{=}0.68$ )

Appendix

Table 10: Human audit of released intent labels, stratified over the three cohorts. Agreement on agent transitions is not lower than on human ones, so the human-agent behavioral gaps of § 4 are not an artifact of the labels being less reliable on agent code. The

75

–

85\%

spread lies within binomial uncertainty at these sample sizes (roughly

\pm 12

points at

n{=}40

Gap (human $-$ agent)	Orig.	Scored-only	Agents filt.	$+$ New runs	95% cluster CI
Debugging intent, vs Codex	$+11.4$	$+8.0$	$+11.6$	$+10.1$	$[+6.0,+14.0]$
Debugging intent, vs MLEvolve	$+9.6$	$+6.2$	$+9.0$	—	$[+7.7,+12.4]$
Ensemble action mass, vs Codex	$-13.4$	$-12.9$	$-14.8$	$-10.4$	$[-19.9,-7.7]$
Model $+$ train. mass, vs MLEvolve	$-16.5$	$-16.3$	$-15.1$	—	$[-21.3,-10.9]$
Action JSD (bits), vs Codex	$0.077$	$0.080$	$0.095$	$0.041$	$[0.047,0.145]$
Action JSD (bits), vs MLEvolve	$0.054$	$0.061$	$0.047$	—	$[0.048,0.084]$

Appendix

Table 11: Every headline gap survives every fairness treatment, and every interval excludes zero. Gaps are human minus agent, in percentage points except for the divergence rows. Note that agents filtered widens rather than narrows the Codex gaps.

Coarse action	Top-40% humans	All humans	Codex	MLEvolve
data	8.0	9.1	8.5	9.0
features	9.1	10.0	5.6	11.9
augmentation	0.4	0.4	0.2	0.4
model	10.7	10.5	8.2	20.4
training	11.4	11.4	8.6	18.0
ensemble	9.8	6.3	19.7	6.0

Appendix

Table 12: Share of coarse-action mass per cohort, in percent, trajectory-equal. Columns need not sum to

100

exactly because a transition can carry several actions. Codex concentrates in ensemble and inference , MLEvolve in model and training , and both spend roughly a third of the human share on housekeeping .

Cohort	Quintile	data	feat.	aug.	model	train.	ens.	valid.	infer.	infra	house.
Top-40% humans	Q1	9.4	8.7	0.2	9.8	10.3	7.3	4.1	9.6	15.1	25.4
	Q2	7.8	8.7	0.6	10.4	13.5	9.1	4.9	9.2	13.5	22.4
	Q3	7.7	9.0	0.3	10.8	10.6	10.1	5.9	10.3	13.0	22.2
	Q4	6.6	9.1	0.4	9.4	11.1	10.5	5.0	10.8	12.5	24.5
	Q5	6.5	9.3	0.3	10.6	11.3	11.3	5.2	10.9	12.0	22.6
Codex	Q1	8.7	5.4	0.2	6.3	8.6	22.6	9.7	23.9	11.0	3.4

Appendix

Table 13: Per-quintile split of the coarse-action shares of Table 12 , in percent, for the top-40% human cohort and Codex. Q1–Q5 are within-trajectory position quintiles. Human ensembling rises steadily over the run (7.3 to 11.3) while Codex holds its ensemble-and-inference band from the first quintile on.

Solution revisit (state signature)
	Top humans	Other humans	Codex	MLEvolve
% of eligible versions	9.1%	7.2%	0.2%	0.0%
% of trajectories with one	32.8%	33.7%	14.3%	0.0%
eligible versions	5,838	5,648	658	344
revisits observed	531	406	1	0
% that beat the version returned to	78.5%	63.9%	$n$ too small

Appendix

Table 14: Two senses of going back, on the twelve-hour scope. Solution revisits use the released state signature (Jaccard

\geq 0.6

, closer to the earlier version than to the predecessor by

0.1

, with the run reaching something below

0.5

similarity in between); setback recovery uses scores only. Codex’s single revisit in

658

eligible versions and MLEvolve’s zero in

344

are against the

60

and

31

occurrences the top human rate would predict.

Feature	One-line description
Ensemble timing (3)
has_ens	Indicator for whether any ensemble-blending action appears in the trajectory.
ens_late_minus_early	Q5 minus Q1 share of ensemble-blending actions; positive means ensembling concentrates late.
pos_first_ens	Normalised position $[0,1]$ of the first ensemble action; $1$ if no ensemble action ever fires.
Magnitude and quintile-shift (4)
micro_q1	Q1 share of transitions labeled magnitude $=$ micro (small early edits typical of expert iteration).

Appendix

Table 15: The 19 trajectory features used in the predictive measures of § 4 and the per-cohort comparison of § 5.1 . Q1 / Q5 denote the first and fifth normalized-position quintiles; "share" denotes the fraction of transitions or states (whichever applies) within the trajectory.

	paired ( $n=358$ , 7 comps)		humans-only ( $n=3545$ , 127 comps)
#	feature	$\rho$	feature	$\rho$
1	fold_averaging	$-0.59$	pos_first_ens	$+0.35$
2	holdout_split	$+0.50$	fold_averaging	$-0.35$
3	mode.ens	$-0.49$	holdout_split	$+0.34$
4	pos_first_ens	$+0.47$	blending	$-0.33$
5	oof_prediction	$-0.43$	change_weights	$-0.32$

Appendix

Table 16: Feature-rank association:

n

-weighted mean of per-competition Spearman

\rho

with leaderboard percentile. Competition-cluster bootstrap CIs for the top eight exclude zero on both splits. Five features enter the top six on both splits; oof_prediction and blending are near-collinear markers of the same practice, and we carry blending as the representative. change_weights is kept despite also being a Codex mono-loop marker (CIs

[-0.48,-0.35]

paired,

[-0.37,-0.28]

humans-only); it is non-monotone across the full range, which is what makes it the saturated component of § 5.2 . All associations are correlational and co-vary with trajectory length.

Action tag	One-line description
Codex bookkeeping-loop markers
change_weights	Re-weight the contributions of existing ensemble members.
postprocess_change	Adjust a post-hoc step on predictions (rounding, clipping, calibration).
data_loading	Modify how data is read or assembled without changing features or model.
add_member	Add a model to the ensemble.
MLEvolve mutate-in-place markers

Appendix

Table 17: Fine action tags named in the scaffold-signature analysis of § 4.1 , grouped by the loop each characterizes. All are drawn from the 85-tag action vocabulary of the transition schema.

Competition	Metric	Baseline	Harness	Abl-A	Abl-B
commonlit	RMSE $\downarrow$	0.510	0.505 / 0.517	0.520	0.512
equity	C-index $\uparrow$	0.675	0.670 / 0.680	0.672	0.670
gquest	Spearman $\uparrow$	0.371	0.429	0.372	0.388
aes2	QWK $\uparrow$	0.771	0.817 / 0.808	0.806	0.796
hms	KL $\downarrow$	1.050	0.718	0.795	1.377
ranzcr	AUC $\uparrow$	0.545	0.877	0.542	0.583

Appendix

Table 18: Best score reached per run at a matched twelve-hour budget. Arrows give the metric direction. Two entries in a cell are independent runs under the same condition. Abl-A is a single content block; Abl-B is the injection cadence with no planning content.

Explore similar work

Sep 16, 2026cs.AI

TuiML: Machine Learning for AI Agents

Machine-learning libraries such as Weka and scikit-learn were designed for human programmers. Language-model agents now use these same libraries by recalling APIs from memory and writing code, an approach that hides what a library offers, delays errors until runtime, and loses experimental state between turns. We present TuiML, a self-contained machine-learning library built for AI agents, with native algorithms across supervised, unsupervised, time-series, data handling, tuning, and evaluation tasks. Every component describes itself through machine-readable metadata and parameter schemas, so an agent can search the library, inspect components, compose validated workflows, and register new ones that become discoverable in turn. Every call is validated, seeded, and traced, and sessions export as runnable notebooks, making experiments reproducible by construction. One specification layer drives the Model Context Protocol (MCP), agent-framework adapters, a Python API, a CLI, and local model serving, while data and models never leave the machine. Benchmarks show TuiML remains predictively competitive with scikit-learn and Weka. While looking like a conventional library to a human user, TuiML is designed for agents first, allowing them to read, extend, and operate machine learning autonomously. TuiML is open source, with documentation at https://tuiml.ai.

Nilesh Verma, Nick Lim, Albert Bifet +1

AI Institute, University of Waikato Hamilton 3216, New Zealand · LTCI, T´el´ecom Paris, Institut Polytechnique de Paris 19 place Marguerite Perey, 91120 Palaiseau, France

Artificial Intelligence Agents Model Context Protocol

Ask PDF

Sep 30, 2026cs.AI

ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible

When designing machine learning models, desirable properties are often in tension: improving one behavior can impair another, so task progress can depend on alleviating the conflict. LLM-based AutoResearch systems, which iteratively edit model code and retain edits based on scalar task-performance feedback, have largely ignored this trade-off. We find that scalar feedback supports broad exploration early in search, but it does not reveal how edits affect competing behaviors. In matched-budget experiments, introducing competing-behavior feedback as task gains diminish increases the share of proposals that improve both behaviors and sustains progress beyond scalar-only plateaus. Obtaining this feedback for a given model requires identifying its competing behaviors and designing probes to measure them. To make competing-behavior feedback actionable, we introduce ConflictGuide. Its reusable ConflictGuide-Skill combines a literature-grounded taxonomy with model-specific evidence to identify competing behaviors and specify probes for a code agent to implement as metrics. Evolution proceeds in two stages: Stage I explores with task feedback; Stage II uses probe feedback to steer proposals toward conflict alleviation and retains marginal-gain edits only when probes indicate sufficient alleviation. Across five diverse model families, ConflictGuide reduces task and conflict-related errors by up to 28% and 14%, respectively, relative to scalar-only AutoResearch, with gains extending to other code agents.

Binqian Xu, Qiran Zou, Xiangbo Shu +1

National University of Singapore · Nanjing University of Science and Technology

Code Quality Adversarial Evaluation Feedback Agent Behavior Modeling

Ask PDF

May 7, 2026cs.MA

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by

0.81\%

, raises NanoChat-D12 CORE by

38.7\%

, and reduces CIFAR-10 Airbench96 wallclock by

4.59\%

, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.

Jingjie Ning, Xiaochuan Li, Ji Zeng +2

School of Computer Science, Carnegie Mellon University

Auto Research Loop Single Trajectory Lineages

Ask PDF