Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems. LLM-driven data science agents promise to automate the entire machine learning pipeline, yet their real-world effectiveness remains limited. Existing frameworks depend on rigid, pre-defined workflows and inflexible coding strategies; consequently, they excel only on relatively simple, classical problems and fail to capture the empirical expertise that human practitioners bring to complex, innovative tasks. In this work, we introduce AutoMind, an adaptive, knowledgeable LLM-agent framework that overcomes these deficiencies through three key advances: (1) a curated expert knowledge base that grounds the agent in domain expert knowledge, (2) an agentic knowledgeable tree search algorithm that strategically explores possible solutions, and (3) a self-adaptive coding strategy that dynamically tailors code generation to task complexity. Evaluations on two automated data science benchmarks demonstrate that AutoMind delivers superior performance versus state-of-the-art baselines. Additional analyses confirm favorable effectiveness, efficiency, and qualitative solution quality, highlighting AutoMind as an efficient and robust step toward fully automated data science. Code is at https://github.com/zjunlp/AutoMind.
Figures & tables
Figure 1. The framework of our AutoMind . The Expert Knowledge Base (upper right) supplies domain papers and competition tricks through a retriever. The Agentic Knowledgeable Tree Search (left) explores the solution space by iteratively expanding a tree via three actions: Drafting creates initial solutions from retrieved papers, Improving refines valid nodes using retrieved tricks, and Debugging fixes buggy nodes through plan modification. The Self-Adaptive Coding Strategy (bottom right) routes each plan through a complexity scorer, where high-complexity plans are decomposed into substeps with per-step verification, and low-complexity plans are generated in a single pass.
Method
Backbone
MLE-Bench
Top AI Competitions
Easy
Medium
Hard
Overall
OAG
BELKA
Overall
Best@3
Avg@3
Best@3
Avg@3
Best@3
Avg@3
Best@3
Avg@3
MLAB
GPT-4o ‡
0.22
0.13
0.08
0.04
0.03
0.02
0.11
0.06
-
-
-
OpenHands
GPT-4o ‡
0.48
0.28
0.08
0.07
0.17
0.08
0.24
0.15
-
-
-
AIDE
GPT-4o ‡
0.71
0.53
0.26
0.15
0.13
0.07
0.37
0.25
-
-
-
o3-mini
0.53
0.44
0.22
0.19
0.27
0.20
0.34
0.28
0.56
0.09
0.33
Table 1. Main results. For MLE-Bench, we report both the best@3 and avg@3 win rates against human participants. For the OAG and BELKA competitions, we report the avg@3 task metrics, which are AUC and AP, respectively. The best and suboptimal results for each task are highlighted. ‡ indicates that the results are borrowed from the grading reports of previous work.
Figure 2. Test time scaling results on MLE-Bench. We record hourly snapshots of the percentage of human participants surpassed by the agent’s best solution over a 24-hour time budget in experiments with DeepSeek-V3 .
Figure 3. Abaltion studies on DeepSeek-V3 for AutoMind on the Medium split of MLE-Bench. Win Rate represents the percentage of human participants surpassed by the agent on the official leaderboard. Valid Rate represents the percentage of valid submissions among all solutions the agent makes within a 24-hour time budget.
Method
Precision
Recall
F1
Relevant
Embedding-only
52%
68%
0.59
51/75
Label-only
78%
42%
0.55
32/75
Ours (Hybrid)
84%
74%
0.79
56/75
Table 2. Knowledge Retrieval Strategy Comparison. Evaluation on 15 data science tasks, top-5 solutions retrieved per task (75 total retrievals).
Figure 4. Complexity Score Distribution.
Agents
Input
Output
Total
AIDE (24h)
2.27 ± 0.28
0.22 ± 0.03
2.49 ± 0.31
AutoMind (15h)
2.15 ± 0.24
0.27 ± 0.04
2.25 ± 0.27
Table 3. Token costs across all MLE-Bench tasks. Input, output, and total token costs for experiments with DeepSeek-V3 , reported in millions of tokens.
Figure 5. A running case on the BELKA challenge. We compare the proposed solution plans and corresponding code implementations generated by both AIDE and AutoMind .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Tasks
Task Type
Dataset Size (GB)
Metric Type
Original Complexity
Split
aptos2019-blindness-detection
Image Classification
10.22
Max
Low
Easy
random-acts-of-pizza
Text Classification
0.003
Max
Low
Easy
spooky-author-identification
Text Classification
0.0019
Min
Low
Easy
google-quest-challenge
Training LLMs
0.015
Max
Medium
Easy
stanford-covid-vaccine
Tabular
2.68
Min
High
Easy
predict-volcanic-eruptions-ingv-oe
Signal Processing
31.25
Min
High
Easy
Appendix
Table 4. Full list of tasks in MLE-Bench used for evaluation in our work.
Hyperparameter
Value
agent.retriever.model
gpt-4.1-mini-2025-04-14
agent.analyzer.model
gpt-4.1-mini-2025-04-14
agent.planner.model
&TARGET_MODEL
agent.coder.model
&TARGET_MODEL
agent.improver.model
&TARGET_MODEL
agent.verifier.model
gpt-4.1-mini-2025-04-14
Appendix
Table 5. Hyperparameters for AutoMind . &TARGET_MODEL is the foundation model being evaluated. agent.search.num_drafts is the number of initial draft nodes. agent.search.debug_prob is the probalitity of whether debug a node. agent.search.trick_prob is the probalitity of whether use tricks to improve a node. agent.search.greedy_prob is the probability of whether select the best node.
Hyperparameter
Value
agent.code.model
&TARGET_MODEL
agent.feedback.model
gpt-4.1-mini-2025-04-14
agent.steps
2000
agent.search.num_drafts
5
agent.search.max_debug_depth
20
agent.search.debug_prob
1
Appendix
Table 6. Hyperparameters for AIDE. &TARGET_MODEL is the foundation model being evaluated. agent.search.num_drafts is the number of initial draft nodes. agent.search.debug_prob is the probalitity of whether debug a node.