The fast-growing demands in using Large Language Models (LLMs) to tackle complex multi-step data science tasks create an emergent need for accurate benchmarking. There are two major gaps in existing benchmarks: (i) the lack of standardized, process-aware evaluation that captures instruction adherence and process fidelity, and (ii) the scarcity of accurately labeled training data. To bridge these gaps, we introduce DARE-bench, a benchmark designed for machine learning modeling and data science instruction following. Unlike many existing benchmarks that rely on human- or model-based judges, all tasks in DARE-bench have verifiable ground truth, ensuring objective and reproducible evaluation. To cover a broad range of tasks and support agentic tools, DARE-bench consists of 6,300 Kaggle-derived tasks and provides both large-scale training data and evaluation sets. Extensive evaluations show that even highly capable models such as gpt-o4-mini struggle to achieve good performance, especially in machine learning modeling tasks. Using DARE-bench training tasks for fine-tuning can substantially improve model performance. For example, supervised fine-tuning boosts Qwen3-32B's accuracy by 1.83x and reinforcement learning boosts Qwen3-4B's accuracy by more than 8x. These significant improvements verify the importance of DARE-bench both as an accurate evaluation benchmark and critical training data.
Figures & tables
Figure 1: DARE-bench defines each task by providing a natural-language question and structured files (metadata and train/test splits). An LLM agent executes code within a sandbox to generate predictions, which are compared against ground truth for automatic and reproducible evaluation.
Benchmark
Domain
Data File
Inst-follow
Time Series
Verifiable
Train Tasks
Tasks
MLAgentBench ( Huang et al., 2023 )
Deep Learning
✓
-
-
✓
✗
13
MLE-bench ( Chan et al., 2024 )
Deep Learning
✓
-
-
✓
✗
75
SWE-bench ( Jimenez et al., 2024 )
Software Eng.
✓
-
-
✓
✓
21,294
DS-1000 ( Lai et al., 2023 )
Data Science
✗
✗
✗
✗
✗
1,000
Arcade ( Yin et al., 2022 )
Data Science
✗
✗
✗
✗
✗
1,082
Spider2V ( Cao et al., 2024 )
Data Science
✗
✗
✗
✓
✗
494
Table 1: Comparison between DARE-bench and existing benchmarks.
Task Type
Train Tasks
Test Tasks
Capability Assessed
Classification-IF
1160
74
Instruction following
Classification-MM
1160
74
ML Modeling
Regression-IF
899
45
Instruction following
Regression-MM
899
45
ML Modeling
Time-series-XF
915
57
Predictive ML, forecasting
Time-series-CF
915
57
Predictive ML, forecasting
Table 2: Overview of DARE-bench benchmark composition and the primary capabilities evaluated by each task type. Variants are denoted as IF = Instruction Following, MM = ML Modeling, XF = eXogenous Features, CF = Canonical Forecasting.
Figure 2: Automated pipeline of DARE-bench. The construction process consists of four stages: (1) Dataset Sourcing , where Kaggle datasets are filtered by tags, license, size, and metadata; (2) Task Design , where schema summaries, targets, features, and feasibility are analyzed with the help of LLM; (3) Post-Process , including splitting, noise injection for IF tasks or resampling or entity checks for time-series-CF tasks; and (4) Finalization , which validates solvability in a sandbox for IF tasks and produces standardized benchmark artifacts.
Dataset
Finance
Health
Business
Technology
Automotive
Education
Environment
Others
Train
16.9%
10.2%
7.3%
4.0%
4.5%
2.8%
6.8%
47.5%
Test
17.1%
8.4%
8.2%
5.6%
3.3%
3.1%
2.4%
51.9%
Table 3: Distribution of task domains across the DARE-bench train and test sets.
turns
time
class-IF
class-MM
reg-IF
reg-MM
time-XF
time-CF
3
300
37.16
55.44
29.71
51.69
37.99
6.67
5
200
67.56
57.89
53.62
57.60
42.29
9.67
6
180
73.42
61.07
63.76
60.92
41.59
9.79
8
120
73.87
61.42
65.21
61.05
42.11
8.82
10
100
75.22
63.36
62.31
62.07
42.03
10.97
15
100
76.80
65.88
66.66
62.41
40.03
9.92
Table 4: Hyperparameter sensitivity analysis for o4-mini across different turns and sandbox maximum execution time limit configurations.
Model
class-IF
class-MM
reg-IF
reg-MM
time-XF
time-CF
gpt-4o
32.88
40.45
20.28
40.60
35.54
4.77
gpt-4.1
55.82
57.83
52.17
58.62
40.78
6.60
gpt-5
69.81
43.40
57.24
56.29
36.83
10.13
gpt-o4-mini
67.56
57.89
53.62
57.60
42.29
9.67
Claude-Sonnet-3.7
61.48
61.03
46.37
63.20
49.88
13.70
Claude-Sonnet-4
16.21
18.27
15.21
11.33
4.80
0.01
Table 5: Main evaluation results on our benchmark (test tasks) under the configuration where turns set as 5 and sandbox maximum execution time set as 200 s. The best score in each column is bolded. 2 2 footnotemark: 2
Figure 3: Example of an instruction-following task where the agent fails to respect explicit constraints. Despite being asked to fix the random seed, the model omitted the required argument, leading to incorrect predictions and an evaluation failure.
Model
Setting
class-IF
class-MM
reg-IF
reg-MM
time-XF
time-CF
Total
Model-Perf
Qwen3-32B
Baseline
17.11
30.71
15.21
35.86
26.96
0.00
23.25
65.03
Qwen3-32B
SFT-FV
40.54 +23.43
44.71 +13.99
42.75 +27.54
49.21 +13.35
39.95 +12.99
0.07 +0.07
42.42 +19.17
72.32 +7.29
Qwen3-32B
SFT-AV
40.54 +23.43
47.20 +16.49
42.02 +26.81
55.56 +19.70
33.56 +6.60
0.00 +0.00
42.91 +19.72
70.27 +5.24
Qwen3-32B
SFT-BV
38.06 +20.95
48.91 +18.20
42.75 +27.54
54.55 +18.69
35.91 +8.95
0.00 +0.00
42.83 +19.58
71.01 +5.98
Qwen3-32B
SFT-DV
38.58 +21.47
43.82 +13.11
39.13 +23.92
51.00 +15.14
38.92 +11.96
0.00 +0.00
41.12 +17.18
71.68 +6.65
Qwen3-4B
Baseline
3.60
5.23
0.72
3.29
6.97
0.00
4.39
54.18
Table 6: Fine-tuning and RL improve performance over baselines. Superscripts denote absolute gains compared to the baseline of the same model.
Train Data
class-IF
class-MM
reg-IF
reg-MM
time-XF
time-CF
baseline
17.11
30.71
15.21
35.86
26.96
0.00
IF
40.99 +23.88
22.38 -8.33
47.82 +32.61
27.85 -8.01
23.83 -3.13
0.00 +0.00
MM
11.71 -5.40
45.69 +14.98
18.84 +3.63
45.38 +9.52
34.12 +7.16
0.00 +0.00
IF+MM
38.58 +21.47
43.82 +13.11
39.13 +23.92
51.00 +15.14
38.92 +11.96
0.00 +0.00
Table 7: Ablation study on the impact of Instruction Following (IF) and ML Modeling (MM) data with SFT-DV rejection sampling data.
Model
Setting
Competition-level Accuracy
Qwen3-32B
Baseline
32.38
Qwen3-32B
SFT-FV
37.82 +5.44
Qwen3-32B
SFT-AV
41.08 +8.70
Qwen3-32B
SFT-BV
40.06 +7.68
Qwen3-32B
SFT-DV
42.41 +10.03
Qwen3-4B
Baseline
18.23
Table 8: External validation on DSBench ( Jing et al., 2024 ) after converting tasks into our format. Superscripts denote absolute gains over the baseline of the same model.
Model
Inst Adhere
Code Error
Code Exec Limit
Max Token Limit
gpt-5
126
333
0
0
Claude-Sonnet-3.7
158
262
0
0
Qwen3-32B
48
106
257
372
Qwen3-32B + SFT-DV
43
80
236
256
Qwen3-4B
79
174
661
102
Qwen3-4B + RL
49
91
331
119
Table 9: Failure mode analysis across different models.
Framework
Model
class-IF
class-MM
reg-IF
reg-MM
time-XF
time-CF
DSBench
Native Function Call
Qwen3-32B
17.11
30.71
15.21
35.86
26.96
0.0
32.38
Native Function Call
Qwen3-32B + SFT-DV
38.58
43.82
39.13
51.00
38.92
0.0
42.41
DataWiseAgent
Qwen3-32B
21.62
29.63
34.78
34.40
30.45
0.0
29.17
Table 10: Comparison between Native Function Call 4 4 4 https://qwen.readthedocs.io/en/latest/framework/function_call.html and DataWiseAgent on DARE-bench and DSBench.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
class-IF
class-MM
reg-IF
reg-MM
time-XF
time-CF
gpt-4o
31.86
41.10
20.63
39.74
37.09
5.24
gpt-4.1
55.39
57.75
50.00
57.68
41.19
6.81
gpt-5
70.10
43.18
55.56
55.21
36.78
7.84
gpt-o4-mini
68.14
59.14
51.59
57.48
42.15
8.15
Claude-Sonnet-3.7
61.52
61.22
46.03
61.36
51.21
12.08
Claude-Sonnet-4
14.71
17.70
14.29
10.50
5.27
0.02
Appendix
Table 11: Performance on the Strictly Open-Sourceable Subset
Model
IF Tokens
IF Tool Calls
MM Tokens
MM Tool Calls
Overall Tokens
Overall Tool Calls
prompt
596.7
-
224.6
-
350.7
-
gpt-5
609.5
2.2
582.5
2.4
591.2
2.4
Claude-Sonnet-3.7
675.2
3.6
894.3
4.8
830.0
4.4
Qwen3-32B
2093.3
3.1
1693.0
3.5
1816.8
3.4
Qwen3-32B-SFT-DV
1778.3
3.3
1572.0
3.6
1638.7
3.5
Qwen3-4B
1691.1
3.7
1151.7
3.9
1328.1
3.8
Appendix
Table 12: Average number of tokens and tool calls for completions of different models and prompts. All token counts are calculated using the Qwen3 tokenizer.
Hyper-parameter
Value
RL algorithm
GRPO ( Shao et al., 2024 )
Loss aggregation
Sequence level
Group normalization
False
Learning rate
1×10−6
Training mini-batch size
64
KL regularization
False
Appendix
Table 13: Hyper-parameters used for reinforcement learning experiments.