TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

Jiarui Yan; Weiwei Sun; Sijie Li; Wenhan Li; Yiming Yang

cs.LGAug 26, 2026

TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

Authors: Jiarui Yan, Weiwei Sun, Sijie Li, Wenhan Li, Yiming Yang

Organizations: Carnegie Mellon University

Abstract

Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their own feedback, yet on most competitions they still finish below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4{,}465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, and the labeling models, together with an open-source toolkit that turns a run from any command-line agent into a TraceML trajectory and reads it against the human cohorts.

Figures & tables

Figure 1: Stylized trajectories with per-edit action ribbons, one panel per cohort. Humans mix actions, pivot when behind, and reopen abandoned lines; Codex maintains one solution through small submission-side edits; MLEvolve mutates its model in place. § 4 quantifies each behavior.

Benchmark / Dataset	Score Traj	Code Traj	Human Traj	# Envs.	Horizon
MLE-bench ( Chan et al., 2024 )	✗	✗	⚫	75	24h
MLAgentBench ( Huang et al., 2024 )	✗	⚫	✗	13	–
AIRA ( Toledo et al., 2025 )	⚫	⚫	✗	22	24h
RE-Bench ( Wijk et al., 2025 )	✓	⚫	✓	7	8h
HCAST ( Rein et al., 2025 )	✗	✗	✓	189	1m–8h
SciCode ( Tian et al., 2024 )	✗	✗	✗	80 / 338	–

Table 1: Long-horizon agent benchmarks and datasets. TraceML is the only entry that keeps scored intermediate versions, their code, and task-matched human development. RE-Bench comes closest, pairing time-budgeted human and agent attempts, but on 7 bespoke environments with partial code; TraceML keeps the full code of every scored version across 134 competitions. Horizon is the per-task agent budget, or for TraceML the span of human development. ✓ , ⚫ , ✗ : full, partial, no coverage.

Figure 2: TraceML reconstruction pipeline: notebook histories, git commits, and search journals become one version-level representation. Appendix B.6 follows one real trajectory through every stage.

Subset	Comps	Trajectories	Snapshots	Avg snap/traj	Avg lines/snap
Human (public Kaggle notebook histories)
Grandmaster	114	423	16,973	40.1	555
Master	121	649	24,138	37.2	601
Expert	130	1,386	50,663	36.6	569
Contributor	133	1,932	54,619	28.3	494
Other / Unknown	40	75	3,090	41.2	295

Table 2: TraceML corpus statistics by source. Human trajectories are broken down by Kaggle author tier. Per-version code lines is total code lines divided by snapshots within the subset.

Figure 3: Action profiles at two resolutions. (a) PCA of per-trajectory coarse-action distributions, large markers are cohort centroids; (b) pairwise JSD between cohort action distributions; (c) coarse-action profiles in pipeline order; (d) fine-action usage relative to pooled humans, Codex against MLEvolve ( $\log_{2}$ axes) — the labeled off-diagonal clusters are the two scaffold loops and the shared gap.

Figure 4: Harness effect on six discipline features. Dashed rule: top human value; frames mark whether Codex

+

skill lands closer to it (blue) or farther from it (red) than prior Codex.

Figure 5: Running best leaderboard percentile over relative time, one panel per paired competition; shaded bands are the human cohort thresholds, markers the final percentile reached. The harness lift is uneven across competitions.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Outcome	Kernels	Character of the set
Retained	4,465	47.3% medalled
Removed: no in-window version	382	99.2% published entirely after the deadline, median 299 days late
Removed: failed content conditions	201	Too few versions, too short a span, or no score
Removed (all)	583	43.3% medalled

Appendix

Table 3: What the retention filter removes, from 5,048 candidate kernels. The removed set is dominated by post-deadline write-ups rather than by weak development trajectories, and its medal rate is comparable to that of the retained set, so the filter does not remove a weak tail.

Reuse measure	Full corpus (4,465)	Paired sample (430)
Has a fork parent	8.2%	8.6%
Near-duplicate code in $\geq 1$ version	23.7%	21.0%
Near-duplicate code in a majority of versions	—	7.9%
Near-duplicate code across the whole trajectory	—	3.3%

Appendix

Table 4: Visible reuse in the retained corpus and in the paired sample used for human-agent comparison. Near-duplicate code is code similarity

\geq 0.9

against another kernel; all observed near-duplicate links fall within the same competition, consistent with shared starter templates.

	Human	Codex	MLEvolve
Median versions per trajectory	20	18	5
Versions carrying a score	45%	$\approx$ 99%	$\approx$ 99%
Saved versions that are syntactically valid code	97.9%	—	—

Appendix

Table 5: How the version unit lines up across sources. Agent versions are submission-producing commits with adjacent identical-code commits collapsed, which is the analogue of a deliberate Kaggle save-version rather than a raw log entry.

				MLEvolve
Scope	Comps	Human traj.	Codex runs	runs	branches
Corpus (§ 3.1 )	134	4,465	—	—	—
Paired subset (§ 3.2 )	7	430	11	13	189
Twelve-hour scope (§ 4 )	7	430	10	3	107
Harness arms (§ 5.1 )	7	—	30	—	—

Appendix

Table 6: The three scopes. Human trajectories are split into a top cohort and the rest by leaderboard rank. MLEvolve branches share tree nodes, so the run count rather than the branch count sets the effective sample size.

Competition	Track	Transitions	Pivot rate
gquest	agent	1,017	2.1%
gquest	llm_v3	226	2.2%
ranzcr	agent	166	4.8%
ranzcr	llm_v3	102	2.0%
hms	agent	97	2.1%
aes2	llm_v3	64	1.6%

Appendix

Table 7: The ten twelve-hour Codex runs. “Pivot rate” is the marginal (unmatched) share of transitions carrying a pivot tag. Runs are ordered by length; the four shortest carry almost no information individually.

Transition	Actions	Intents	Magn.	Score
$v_{13}\to v_{14}$	model, training, housekeeping, infra	optimization + debugging	micro	$0.535\to 0.509$ (improving)
$v_{14}\to v_{15}$	data, augmentation, training, model	exploration + optimization	minor	unscored
$v_{15}\to v_{16}$	data, training, validation, housekeeping	debugging + optimization	micro	$\to 0.4989$ (trajectory best)
$v_{16}\to v_{17}$	training, model, infra	optimization	micro	$0.4989\to 0.5062$ (regressing)

Appendix

Table 8: Four consecutive transitions from one human trajectory, as represented in TraceML. Each transition carries actions, intents, a magnitude, and the score effect read from the leaderboard.

Figure 6: Example schema output for one version and its transition.

Annotation level (#tags)	Self-consistency	Cross-model	Teacher $\to$ Student
State coarse (8)	$\kappa{=}0.872$ / J= $0.969$	$\kappa{=}0.801$ / J= $0.954$	$F_{1}^{\mathrm{macro}}{=}0.978$
Action coarse (10)	J= $0.875$	J= $0.641$	$F_{1}^{\mathrm{macro}}{=}0.733$
Intent (6 classes)	$\kappa{=}0.834$	$\kappa{=}0.611$	$\mathrm{acc}{=}0.772$
Magnitude (4 levels)	$\kappa{=}0.921$	$\kappa{=}0.576$	$\mathrm{acc}{=}0.928$

Appendix

Table 9: Annotation reliability across teacher stability, cross-model agreement, and teacher–student transfer. We report Cohen’s

\kappa

and multi-label Jaccard (J) on held-out state and action annotations; teacher–student scores compare the released Qwen3-1.7B labelers with the gpt-5.4-mini teacher.

Cohort	$n$	6-class agreement	3-class collapse
Human	40	80.0%	82.5%
Codex	40	85.0%	87.5%
MLEvolve	20	75.0%	75.0%
Overall	100	81%	83% ( $\kappa{=}0.68$ )

Appendix

Table 10: Human audit of released intent labels, stratified over the three cohorts. Agreement on agent transitions is not lower than on human ones, so the human-agent behavioral gaps of § 4 are not an artifact of the labels being less reliable on agent code. The

75

–

85\%

spread lies within binomial uncertainty at these sample sizes (roughly

\pm 12

points at

n{=}40

Gap (human $-$ agent)	Orig.	Scored-only	Agents filt.	$+$ New runs	95% cluster CI
Debugging intent, vs Codex	$+11.4$	$+8.0$	$+11.6$	$+10.1$	$[+6.0,+14.0]$
Debugging intent, vs MLEvolve	$+9.6$	$+6.2$	$+9.0$	—	$[+7.7,+12.4]$
Ensemble action mass, vs Codex	$-13.4$	$-12.9$	$-14.8$	$-10.4$	$[-19.9,-7.7]$
Model $+$ train. mass, vs MLEvolve	$-16.5$	$-16.3$	$-15.1$	—	$[-21.3,-10.9]$
Action JSD (bits), vs Codex	$0.077$	$0.080$	$0.095$	$0.041$	$[0.047,0.145]$
Action JSD (bits), vs MLEvolve	$0.054$	$0.061$	$0.047$	—	$[0.048,0.084]$

Appendix

Table 11: Every headline gap survives every fairness treatment, and every interval excludes zero. Gaps are human minus agent, in percentage points except for the divergence rows. Note that agents filtered widens rather than narrows the Codex gaps.

Coarse action	Top-40% humans	All humans	Codex	MLEvolve
data	8.0	9.1	8.5	9.0
features	9.1	10.0	5.6	11.9
augmentation	0.4	0.4	0.2	0.4
model	10.7	10.5	8.2	20.4
training	11.4	11.4	8.6	18.0
ensemble	9.8	6.3	19.7	6.0

Appendix

Table 12: Share of coarse-action mass per cohort, in percent, trajectory-equal. Columns need not sum to

100

exactly because a transition can carry several actions. Codex concentrates in ensemble and inference , MLEvolve in model and training , and both spend roughly a third of the human share on housekeeping .

Cohort	Quintile	data	feat.	aug.	model	train.	ens.	valid.	infer.	infra	house.
Top-40% humans	Q1	9.4	8.7	0.2	9.8	10.3	7.3	4.1	9.6	15.1	25.4
	Q2	7.8	8.7	0.6	10.4	13.5	9.1	4.9	9.2	13.5	22.4
	Q3	7.7	9.0	0.3	10.8	10.6	10.1	5.9	10.3	13.0	22.2
	Q4	6.6	9.1	0.4	9.4	11.1	10.5	5.0	10.8	12.5	24.5
	Q5	6.5	9.3	0.3	10.6	11.3	11.3	5.2	10.9	12.0	22.6
Codex	Q1	8.7	5.4	0.2	6.3	8.6	22.6	9.7	23.9	11.0	3.4

Appendix

Table 13: Per-quintile split of the coarse-action shares of Table 12 , in percent, for the top-40% human cohort and Codex. Q1–Q5 are within-trajectory position quintiles. Human ensembling rises steadily over the run (7.3 to 11.3) while Codex holds its ensemble-and-inference band from the first quintile on.

Solution revisit (state signature)
	Top humans	Other humans	Codex	MLEvolve
% of eligible versions	9.1%	7.2%	0.2%	0.0%
% of trajectories with one	32.8%	33.7%	14.3%	0.0%
eligible versions	5,838	5,648	658	344
revisits observed	531	406	1	0
% that beat the version returned to	78.5%	63.9%	$n$ too small

Appendix

Table 14: Two senses of going back, on the twelve-hour scope. Solution revisits use the released state signature (Jaccard

\geq 0.6

, closer to the earlier version than to the predecessor by

0.1

, with the run reaching something below

0.5

similarity in between); setback recovery uses scores only. Codex’s single revisit in

658

eligible versions and MLEvolve’s zero in

344

are against the

60

and

31

occurrences the top human rate would predict.

Feature	One-line description
Ensemble timing (3)
has_ens	Indicator for whether any ensemble-blending action appears in the trajectory.
ens_late_minus_early	Q5 minus Q1 share of ensemble-blending actions; positive means ensembling concentrates late.
pos_first_ens	Normalised position $[0,1]$ of the first ensemble action; $1$ if no ensemble action ever fires.
Magnitude and quintile-shift (4)
micro_q1	Q1 share of transitions labeled magnitude $=$ micro (small early edits typical of expert iteration).

Appendix

Table 15: The 19 trajectory features used in the predictive measures of § 4 and the per-cohort comparison of § 5.1 . Q1 / Q5 denote the first and fifth normalized-position quintiles; "share" denotes the fraction of transitions or states (whichever applies) within the trajectory.

	paired ( $n=358$ , 7 comps)		humans-only ( $n=3545$ , 127 comps)
#	feature	$\rho$	feature	$\rho$
1	fold_averaging	$-0.59$	pos_first_ens	$+0.35$
2	holdout_split	$+0.50$	fold_averaging	$-0.35$
3	mode.ens	$-0.49$	holdout_split	$+0.34$
4	pos_first_ens	$+0.47$	blending	$-0.33$
5	oof_prediction	$-0.43$	change_weights	$-0.32$

Appendix

Table 16: Feature-rank association:

n

-weighted mean of per-competition Spearman

\rho

with leaderboard percentile. Competition-cluster bootstrap CIs for the top eight exclude zero on both splits. Five features enter the top six on both splits; oof_prediction and blending are near-collinear markers of the same practice, and we carry blending as the representative. change_weights is kept despite also being a Codex mono-loop marker (CIs

[-0.48,-0.35]

paired,

[-0.37,-0.28]

humans-only); it is non-monotone across the full range, which is what makes it the saturated component of § 5.2 . All associations are correlational and co-vary with trajectory length.

Action tag	One-line description
Codex bookkeeping-loop markers
change_weights	Re-weight the contributions of existing ensemble members.
postprocess_change	Adjust a post-hoc step on predictions (rounding, clipping, calibration).
data_loading	Modify how data is read or assembled without changing features or model.
add_member	Add a model to the ensemble.
MLEvolve mutate-in-place markers

Appendix

Table 17: Fine action tags named in the scaffold-signature analysis of § 4.1 , grouped by the loop each characterizes. All are drawn from the 85-tag action vocabulary of the transition schema.

Competition	Metric	Baseline	Harness	Abl-A	Abl-B
commonlit	RMSE $\downarrow$	0.510	0.505 / 0.517	0.520	0.512
equity	C-index $\uparrow$	0.675	0.670 / 0.680	0.672	0.670
gquest	Spearman $\uparrow$	0.371	0.429	0.372	0.388
aes2	QWK $\uparrow$	0.771	0.817 / 0.808	0.806	0.796
hms	KL $\downarrow$	1.050	0.718	0.795	1.377
ranzcr	AUC $\uparrow$	0.545	0.877	0.542	0.583

Appendix

Table 18: Best score reached per run at a matched twelve-hour budget. Arrows give the metric direction. Two entries in a cell are independent runs under the same condition. Abl-A is a single content block; Abl-B is the injection cadence with no planning content.

Explore similar work

TuiML: Machine Learning for AI Agents
Sep 16, 2026Nilesh Verma, Nick Lim, Albert Bifet +1Artificial Intelligence Agents Model Context Protocol
AI Institute, University of Waikato Hamilton 3216, New Zealand · LTCI, T´el´ecom Paris, Institut Polytechnique de Paris 19 place Marguerite Perey, 91120 Palaiseau, France
ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible
Sep 30, 2026Binqian Xu, Qiran Zou, Xiangbo Shu +1Code Quality Adversarial Evaluation
National University of Singapore · Nanjing University of Science and Technology
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
May 7, 2026Jingjie Ning, Xiaochuan Li, Ji Zeng +2Auto Research Loop
School of Computer Science, Carnegie Mellon University

Sep 16, 2026, cs.AIJ/K move · Enter open · S save

TuiML: Machine Learning for AI Agents

Nilesh Verma, Nick Lim, Albert Bifet, Bernhard Pfahringer

AI Institute, University of Waikato Hamilton 3216, New Zealand · LTCI, T´el´ecom Paris, Institut Polytechnique de Paris 19 place Marguerite Perey, 91120 Palaiseau, France

Artificial Intelligence Agents Model Context Protocol

Open paper AskPDF arXiv

Machine-learning libraries such as Weka and scikit-learn were designed for human programmers. Language-model agents now use these same libraries by recalling APIs from memory and writing code, an approach that hides what a library offers, delays errors until runtime, and loses experimental state between turns. We present TuiML, a self-contained machine-learning library built for AI agents, with native algorithms across supervised, unsupervised, time-series, data handling, tuning, and evaluation tasks. Every component describes itself through machine-readable metadata and parameter schemas, so an agent can search the library, inspect components, compose validated workflows, and register new ones that become discoverable in turn. Every call is validated, seeded, and traced, and sessions export as runnable notebooks, making experiments reproducible by construction. One specification layer drives the Model Context Protocol (MCP), agent-framework adapters, a Python API, a CLI, and local model serving, while data and models never leave the machine. Benchmarks show TuiML remains predictively competitive with scikit-learn and Weka. While looking like a conventional library to a human user, TuiML is designed for agents first, allowing them to read, extend, and operate machine learning autonomously. TuiML is open source, with documentation at https://tuiml.ai.

Abstract

Explore similar work

TuiML: Machine Learning for AI Agents

ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes