Agentic machine learning engineering (MLE) is an emerging AI4SE application for complex ML tasks and a step toward recursive self-improvement of AI systems. Recent agentic MLE systems show the value of leveraging insights from related MLE tasks: some systems condition code generation on expert domain knowledge, which is implicitly curated from peer MLE tasks; some systems have a loop of solving an MLE task, gathering memory to benefit future tasks. However, insight collection and injection remain ad hoc, and existing evaluations often do not control insight sources, making data leakage a threat to validity. To close these gaps, we systematically study how insights from related MLE tasks affect agentic MLE systems, and how their impact depends on insight type and injection method. We introduce MLE-InsightBench, a benchmark of 160 Kaggle competitions across 12 domains. To prevent leakage, one competition per domain is held out as the target, while the remaining 148 serve as sources only when they cannot have been influenced by the target task, such as through later editions or reused solutions. We also develop MLE-InsightForge, an insight assembly agent that explores source competitions, top human solutions, and writeups to construct three insight types: 12 domain insights summarizing best practices, 148 competition insights describing winning solutions, and 16 improvement insights capturing generic debugging and optimization strategies. These insights are injected either as on-demand skills or as an upfront playbook. In our evaluation, the extracted insights improve the normalized private-test score by 123.9% over a no-insight baseline. Controlled experiments further show that the three insight types are complementary; the better injection mode (skill vs. playbook) varies by tasks, with skills more cost-effective overall; and the gains generalize across backend LLMs.
Figures & tables
Figure 1. Overview of our study. Three side-by-side panels. The benchmark panel lists Kaggle competition collection, domain clustering, the held-out split, leakage control, and private-test grading. The technique panel lists the insight assembly agent, domain, competition, and improvement insights, and insight injection as skills or playbooks. The evaluation panel lists the agentic MLE system, six conditions, the fixed budget, the metrics, and the research questions.
Figure 2. An example Kaggle competition and a simplified ML solution developed by Baseline. Two side-by-side boxes. (a) The task Jigsaw – Agile Community Rules Classification, styled as its Kaggle competition page with the address kaggle.com/competitions/jigsaw-agile-community-rules, the subtitle "Using AI models to help moderators uphold community-specific norms", and the sections Description, Dataset, Metric, and Submission: the description asks for a binary classifier that predicts whether a Reddit comment broke a specific rule, with unseen rules in the test data; the dataset consists of train.csv and test.csv, illustrated by one row with the comment body, the rule, the subreddit, positive and negative example comments, and the binary target rule_violation; the metric is the column-averaged AUC; the submission file contains a probability for each row. (b) A Python solution in four stages: data preparation, which encodes the comment, the rule, and the example comments with a sentence-embedding model and computes their similarities as features; validation, which trains a logistic regression with each rule held out in turn and reports the mean AUC; training on all data; and test, which predicts the test data and writes submission.csv.
Figure 3. UMAP embeddings of the competition descriptions (12 clusters). A scatter plot of UMAP embeddings of the competition descriptions, clustered into twelve clusters plus noise.
Domain
Target task
Metric
Type
Agriculture Recognition
CSIRO Biomass ( Liao et al., 2025 )
R2↑
Simple
Bioinformatics
CAFA 5 Protein Function ( Friedberg et al., 2023 )
F max↑
Simple
Geography Retrieval and Localization
Image Matching 2025 ( Bellavia et al., 2025 )
mAA ↑
Simple
Medical Image Recognition
RSNA Aneurysm Detection ( Rudie et al., 2025 )
AUC ↑
Code
NLP
Jigsaw Community Rules ( Sorensen et al., 2025b )
AUC ↑
Simple
Physiological Signal Processing
PhysioNet ECG Digitization ( Reyna et al., 2025 )
SNR ↑
Simple
Table 1. Target tasks of MLE-InsightBench .
Figure 4. Architecture of MLE-InsightForge : (a) the assembly agent and (b) the four assembly stages. Two panels. Panel (a) shows an assembly task, consisting of instructions and required sources, entering a ReAct loop in which an LLM calls tools (reading files and notebooks, inspecting datasets, running analyses in a sandbox, reading web pages, PDFs, and repositories, and inspecting images and videos) and receives observations; the tools access a read-only workspace with the \SourceCompetitionsand the blacklist-filtered web; the LLM's context is summarized when needed; the LLM submits a report to a validation step that checks that all required sources were read in full and that each insight has an evidence record, returning rejections to the LLM and otherwise producing the insight text and evidence records. Panel (b) shows four stages: competition insight assembly from each \SourceCompetition, domain insight synthesis from the competition insights and descriptions of a domain, improvement insight assembly from writeups reporting a problem and its resolution, and clustering and synthesis of the raw improvement insights into improvement insights.
Figure 5. UMAP embeddings of the raw improvement insights (15 clusters). A scatter plot of UMAP embeddings of the raw improvement insights, clustered into groups plus noise.
Figure 6. Workflow of the agentic MLE system, with the insights highlighted. A sequence diagram with five lifelines: orchestrator, MLE agent, solution tree, submission validator, and insight skills. The orchestrator starts the agent with the task, public data, the domain insight, and the skills or playbook. In a loop, the agent calls the LLM, executes solutions that become nodes of the solution tree, validates submissions, and reads skills in skill mode. After the agent submits a final node, the orchestrator replaces the public-test inputs with the private-test ones, reruns the selected solution, and grades the submission with the private-test labels.
Figure 7. Scope of the insights injected for a target task. A large circle represents the current domain and contains the target task, drawn as a star, and three \SourceCompetitions, drawn as dots. Two smaller circles represent other domains, each containing three \SourceCompetitions. Dashed arrows go from the \SourceCompetitionsin the current domain to a green square labeled domain insight, and from the \SourceCompetitionsof all domains to a red stack of rectangles labeled improvement insights; solid arrows go from the square, from the stack, and from each \SourceCompetitionin the current domain (labeled competition insights) to the target task.
Figure 8. Running example on Energy Behavior of Prosumers. Two parts. Part (a) shows three side-by-side insight files, each with a gray front matter holding its name and description, followed by its content. The Time Series Prediction domain insight has three items under the DATA heading, each stating a condition, an action, and a rationale; the highlighted actions are (1) to use target lags and rolling baselines anchored at least one horizon behind each target, (2) to normalize historical rates to the denominator available for the forecast period, and (3) to align each backtest value to its actual publication vintage. The GoDaddy Microbusiness Density Forecasting competition insight has three items under the DATA heading; the highlighted actions are (2) to normalize all historical rates to the denominator expected at forecast time and (3) to align each value to its real publication vintage, whereas the item on entities with tiny counts is not highlighted. The deployment-contract-temporal-backtesting improvement insight recommends gating every candidate with a deployment-contract backtest replayed across several forecast origins, with the highlighted rationale (3) of matching the as-of information boundary. Part (b) shows a side-by-side diff of two code excerpts with three hunks: features, training, and prediction through the time-series API. The solution without insights, with a validation MAE of 29.24 and a private-test MAE of 234.15, computes rolling target means shifted by only two hours, joins the client data of the same day, trains on the raw target, and at prediction time merges client data from the training period only and fills missing features with zeros. The solution with insights, with a validation MAE of 31.91 and a private-test MAE of 57.12, computes rolling means from target lags of at least 48 hours (change 1), divides the production target by the installed capacity in training and multiplies the predictions by it (change 2), and joins the client data as of two days earlier, both in training and from the data served by the API (change 3).
Mode
Domain
Competition
Improvement
Baseline
–
–
–
Full-skill
playbook
skill
skill
Full-playbook
playbook
playbook
playbook
D-playbook
playbook
–
–
C-playbook
–
playbook
–
I-playbook
–
–
playbook
Table 2. Insight injection modes of each condition.
Figure 9. Validation scores during development; gains are over the first solution of Baseline. Four line charts over development minutes. (a) The best validation score found so far: Full-skill ends highest at about 60, Full-playbook at about 58, and Baseline at about 55; Full-skill and Full-playbook are above Baseline from the start. (b) The gain over Baseline's first solution: Full-skill and Full-playbook start at about 2 and 3 points and end at about 8 and 6 points, and Baseline starts at 0 and ends at about 3 points. (c) All three single-type conditions start above Baseline and end with gains of about 7 to 8 points, compared with about 4 points for Baseline. (d) The difference to Baseline is positive throughout for all three single-type conditions, at about 2 to 3 points in the first 100 minutes; C-playbook ends highest at about 6 points, I-playbook at about 4, and D-playbook at about 3.
Figure 10. Private-test items correctly predicted only by the enclosing conditions. Two Venn diagrams. (a) Baseline, Full-skill, and Full-playbook: 509,717 items are predicted correctly by all three; 3,838 only by Baseline, 5,376 only by Full-skill, 4,918 only by Full-playbook, and 8,786 by Full-skill and Full-playbook but not Baseline. (b) Baseline and the three single-type ablations: 509,377 items are predicted correctly by all four; 1,184 only by Baseline, 1,238 only by D-playbook, 12,345 only by C-playbook, and 4,566 only by I-playbook.
Large language models (LLMs) are expanding from assisting scientific research and engineering to autonomously developing and improving AI systems, advancing AI for AI (AI4AI) as a pathway toward recursive self-improvement (RSI). Workflow-driven machine learning engineering (MLE) systems provide a practical path toward this pursuit through complex orchestration that manages long-horizon experimentation, while recent MLE systems adopt model-directed interaction, allowing models to autonomously organize experiments and adapt optimization strategies based on execution feedback. However, feedback from each experiment reflects several preceding decisions, and a long MLE run contains many such costly experiments. Training must therefore select useful supervision from long interaction trajectories and learn from intermediate outcomes. We propose a post-training method for autonomous long-horizon MLE optimization and apply it to train MLExpert, a 35B Mixture-of-Experts agentic model. We design the Kaggle-MLE Factory to construct executable tasks and an interaction harness for model-directed experimentation. Hierarchical agentic SFT transfers general tool-use and coding capabilities to MLE through outcome-guided supervision. Windowed Interaction RL (WIRL) treats the multi-turn decisions between successive experiment results as an execution window and assigns it a shared outcome-based reward. Asynchronous collection and policy correction support policy updates before the trajectory ends. On MLE-Bench Lite, MLExpert improves the average medal rate of the base model from 37.9% to 72.7%, achieving performance comparable to larger frontier models such as Kimi-K3 and GLM-5.3. Further experiments show that MLExpert improves performance on NatureBench Lite, agent scaffold optimization, and algorithm optimization, suggesting that the learned optimization capability generalizes to other tasks.
Shangheng Du, Jinxin Shi, Shuaiyu Zhang +7
East China Normal University · Shanghai Artificial Intelligence Laboratory
ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition; this is a single-seed campaign result, and multi-seed replication is the priority follow-up. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.
Yongbin Kim, Yashar Talebirad, Osmar R. Zaiane
Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii), Edmonton, Canada
Machine learning engineering (MLE) agents have made substantial progress, but learning through ML experimentation remains costly in time and computation. Synthetic environments reduce these costs while introducing variations in data and experimental settings that require task-specific diagnosis. Access to diagnostic tools alone does not ensure that agents learn when to use them or how to act on their findings. We introduce ToolMLBench, a suite of executable tools for data inspection, code verification, and experiment diagnosis, together with an SFT and RL pipeline for learning their use. Diagnostic calls acquire evidence whose value depends on subsequent decisions, so final outcomes provide limited guidance on which calls to reinforce. We address this challenge with SPICE, which measures how privileged context changes the likelihood of a sampled tool action and uses this difference as a turn-level reward alongside the final outcome. We train on 80 synthetic tasks and evaluate on 25 in-domain and 10 out-of-domain tasks. Providing tool interfaces and descriptions alone yields inconsistent gains across unadapted models. With the same diagnostic interface, our training pipeline raises in-domain success from 24.8% to 52.4% for Qwen3-8B and from 35.6% to 69.2% for Qwen3.5-35B-A3B. The latter also improves from 31% to 48% out-of-domain, supporting learned diagnostic tool use on held-out sources and targets.