DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

Fan Shu; Yite Wang; Ruofan Wu; Boyi Liu; Zhewei Yao; Yuxiong He; Feng Yan

cs.AIFeb 27, 2026

DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

Authors: Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu, Zhewei Yao, Yuxiong He, Feng Yan

Organizations: University of Houston · Snowflake AI Research

Abstract

The fast-growing demands in using Large Language Models (LLMs) to tackle complex multi-step data science tasks create an emergent need for accurate benchmarking. There are two major gaps in existing benchmarks: (i) the lack of standardized, process-aware evaluation that captures instruction adherence and process fidelity, and (ii) the scarcity of accurately labeled training data. To bridge these gaps, we introduce DARE-bench, a benchmark designed for machine learning modeling and data science instruction following. Unlike many existing benchmarks that rely on human- or model-based judges, all tasks in DARE-bench have verifiable ground truth, ensuring objective and reproducible evaluation. To cover a broad range of tasks and support agentic tools, DARE-bench consists of 6,300 Kaggle-derived tasks and provides both large-scale training data and evaluation sets. Extensive evaluations show that even highly capable models such as gpt-o4-mini struggle to achieve good performance, especially in machine learning modeling tasks. Using DARE-bench training tasks for fine-tuning can substantially improve model performance. For example, supervised fine-tuning boosts Qwen3-32B's accuracy by 1.83x and reinforcement learning boosts Qwen3-4B's accuracy by more than 8x. These significant improvements verify the importance of DARE-bench both as an accurate evaluation benchmark and critical training data.

Figures & tables

Figure 1: DARE-bench defines each task by providing a natural-language question and structured files (metadata and train/test splits). An LLM agent executes code within a sandbox to generate predictions, which are compared against ground truth for automatic and reproducible evaluation.

Benchmark	Domain	Data File	Inst-follow	Time Series	Verifiable	Train Tasks	Tasks
MLAgentBench ( Huang et al., 2023 )	Deep Learning	✓	-	-	✓	✗	13
MLE-bench ( Chan et al., 2024 )	Deep Learning	✓	-	-	✓	✗	75
SWE-bench ( Jimenez et al., 2024 )	Software Eng.	✓	-	-	✓	✓	21,294
DS-1000 ( Lai et al., 2023 )	Data Science	✗	✗	✗	✗	✗	1,000
Arcade ( Yin et al., 2022 )	Data Science	✗	✗	✗	✗	✗	1,082
Spider2V ( Cao et al., 2024 )	Data Science	✗	✗	✗	✓	✗	494

Table 1: Comparison between DARE-bench and existing benchmarks.

Task Type	Train Tasks	Test Tasks	Capability Assessed
Classification-IF	1160	74	Instruction following
Classification-MM	1160	74	ML Modeling
Regression-IF	899	45	Instruction following
Regression-MM	899	45	ML Modeling
Time-series-XF	915	57	Predictive ML, forecasting
Time-series-CF	915	57	Predictive ML, forecasting

Table 2: Overview of DARE-bench benchmark composition and the primary capabilities evaluated by each task type. Variants are denoted as IF = Instruction Following, MM = ML Modeling, XF = eXogenous Features, CF = Canonical Forecasting.

Figure 2: Automated pipeline of DARE-bench. The construction process consists of four stages: (1) Dataset Sourcing , where Kaggle datasets are filtered by tags, license, size, and metadata; (2) Task Design , where schema summaries, targets, features, and feasibility are analyzed with the help of LLM; (3) Post-Process , including splitting, noise injection for IF tasks or resampling or entity checks for time-series-CF tasks; and (4) Finalization , which validates solvability in a sandbox for IF tasks and produces standardized benchmark artifacts.

Dataset	Finance	Health	Business	Technology	Automotive	Education	Environment	Others
Train	16.9%	10.2%	7.3%	4.0%	4.5%	2.8%	6.8%	47.5%
Test	17.1%	8.4%	8.2%	5.6%	3.3%	3.1%	2.4%	51.9%

Table 3: Distribution of task domains across the DARE-bench train and test sets.

turns	time	class-IF	class-MM	reg-IF	reg-MM	time-XF	time-CF
3	300	37.16	55.44	29.71	51.69	37.99	6.67
5	200	67.56	57.89	53.62	57.60	42.29	9.67
6	180	73.42	61.07	63.76	60.92	41.59	9.79
8	120	73.87	61.42	65.21	61.05	42.11	8.82
10	100	75.22	63.36	62.31	62.07	42.03	10.97
15	100	76.80	65.88	66.66	62.41	40.03	9.92

Table 4: Hyperparameter sensitivity analysis for o4-mini across different turns and sandbox maximum execution time limit configurations.

Model	class-IF	class-MM	reg-IF	reg-MM	time-XF	time-CF
gpt-4o	32.88	40.45	20.28	40.60	35.54	4.77
gpt-4.1	55.82	57.83	52.17	58.62	40.78	6.60
gpt-5	69.81	43.40	57.24	56.29	36.83	10.13
gpt-o4-mini	67.56	57.89	53.62	57.60	42.29	9.67
Claude-Sonnet-3.7	61.48	61.03	46.37	63.20	49.88	13.70
Claude-Sonnet-4	16.21	18.27	15.21	11.33	4.80	0.01

Table 5: Main evaluation results on our benchmark (test tasks) under the configuration where turns set as 5 and sandbox maximum execution time set as 200 s. The best score in each column is bolded. 2 2 footnotemark: 2

Figure 3: Example of an instruction-following task where the agent fails to respect explicit constraints. Despite being asked to fix the random seed, the model omitted the required argument, leading to incorrect predictions and an evaluation failure.

Model	Setting	class-IF	class-MM	reg-IF	reg-MM	time-XF	time-CF	Total	Model-Perf
Qwen3-32B	Baseline	17.11	30.71	15.21	35.86	26.96	0.00	23.25	65.03
Qwen3-32B	SFT-FV	40.54 +23.43	44.71 +13.99	42.75 +27.54	49.21 +13.35	39.95 +12.99	0.07 +0.07	42.42 +19.17	72.32 +7.29
Qwen3-32B	SFT-AV	40.54 +23.43	47.20 +16.49	42.02 +26.81	55.56 +19.70	33.56 +6.60	0.00 +0.00	42.91 +19.72	70.27 +5.24
Qwen3-32B	SFT-BV	38.06 +20.95	48.91 +18.20	42.75 +27.54	54.55 +18.69	35.91 +8.95	0.00 +0.00	42.83 +19.58	71.01 +5.98
Qwen3-32B	SFT-DV	38.58 +21.47	43.82 +13.11	39.13 +23.92	51.00 +15.14	38.92 +11.96	0.00 +0.00	41.12 +17.18	71.68 +6.65
Qwen3-4B	Baseline	3.60	5.23	0.72	3.29	6.97	0.00	4.39	54.18

Table 6: Fine-tuning and RL improve performance over baselines. Superscripts denote absolute gains compared to the baseline of the same model.

Train Data	class-IF	class-MM	reg-IF	reg-MM	time-XF	time-CF
baseline	17.11	30.71	15.21	35.86	26.96	0.00
IF	40.99 +23.88	22.38 -8.33	47.82 +32.61	27.85 -8.01	23.83 -3.13	0.00 +0.00
MM	11.71 -5.40	45.69 +14.98	18.84 +3.63	45.38 +9.52	34.12 +7.16	0.00 +0.00
IF+MM	38.58 +21.47	43.82 +13.11	39.13 +23.92	51.00 +15.14	38.92 +11.96	0.00 +0.00

Table 7: Ablation study on the impact of Instruction Following (IF) and ML Modeling (MM) data with SFT-DV rejection sampling data.

Model	Setting	Competition-level Accuracy
Qwen3-32B	Baseline	32.38
Qwen3-32B	SFT-FV	37.82 +5.44
Qwen3-32B	SFT-AV	41.08 +8.70
Qwen3-32B	SFT-BV	40.06 +7.68
Qwen3-32B	SFT-DV	42.41 +10.03
Qwen3-4B	Baseline	18.23

Table 8: External validation on DSBench ( Jing et al., 2024 ) after converting tasks into our format. Superscripts denote absolute gains over the baseline of the same model.

Model	Inst Adhere	Code Error	Code Exec Limit	Max Token Limit
gpt-5	126	333	0	0
Claude-Sonnet-3.7	158	262	0	0
Qwen3-32B	48	106	257	372
Qwen3-32B + SFT-DV	43	80	236	256
Qwen3-4B	79	174	661	102
Qwen3-4B + RL	49	91	331	119

Table 9: Failure mode analysis across different models.

Framework	Model	class-IF	class-MM	reg-IF	reg-MM	time-XF	time-CF	DSBench
Native Function Call	Qwen3-32B	17.11	30.71	15.21	35.86	26.96	0.0	32.38
Native Function Call	Qwen3-32B + SFT-DV	38.58	43.82	39.13	51.00	38.92	0.0	42.41
DataWiseAgent	Qwen3-32B	21.62	29.63	34.78	34.40	30.45	0.0	29.17

Table 10: Comparison between Native Function Call 4 4 4 https://qwen.readthedocs.io/en/latest/framework/function_call.html and DataWiseAgent on DARE-bench and DSBench.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	class-IF	class-MM	reg-IF	reg-MM	time-XF	time-CF
gpt-4o	31.86	41.10	20.63	39.74	37.09	5.24
gpt-4.1	55.39	57.75	50.00	57.68	41.19	6.81
gpt-5	70.10	43.18	55.56	55.21	36.78	7.84
gpt-o4-mini	68.14	59.14	51.59	57.48	42.15	8.15
Claude-Sonnet-3.7	61.52	61.22	46.03	61.36	51.21	12.08
Claude-Sonnet-4	14.71	17.70	14.29	10.50	5.27	0.02

Appendix

Table 11: Performance on the Strictly Open-Sourceable Subset

Model	IF Tokens	IF Tool Calls	MM Tokens	MM Tool Calls	Overall Tokens	Overall Tool Calls
prompt	596.7	-	224.6	-	350.7	-
gpt-5	609.5	2.2	582.5	2.4	591.2	2.4
Claude-Sonnet-3.7	675.2	3.6	894.3	4.8	830.0	4.4
Qwen3-32B	2093.3	3.1	1693.0	3.5	1816.8	3.4
Qwen3-32B-SFT-DV	1778.3	3.3	1572.0	3.6	1638.7	3.5
Qwen3-4B	1691.1	3.7	1151.7	3.9	1328.1	3.8

Appendix

Table 12: Average number of tokens and tool calls for completions of different models and prompts. All token counts are calculated using the Qwen3 tokenizer.

Hyper-parameter	Value
RL algorithm	GRPO ( Shao et al., 2024 )
Loss aggregation	Sequence level
Group normalization	False
Learning rate	$1\times 10^{-6}$
Training mini-batch size	64
KL regularization	False

Appendix

Table 13: Hyper-parameters used for reinforcement learning experiments.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Jul 30, 2026

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.

Philipp D. Siedler, Jordan Sassoon

Aleph Alpha Research, Heidelberg, Germany.

LLM Evaluation Benchmark Auditing Benchmark Construction LLM Auditing

Ask PDF

Jul 13, 2026

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.

Chuhan Shi, Xiaoquan Ren, Sicheng Song +3

Southeast University · East China Normal University · Hong Kong University of Science and Technology

AI for Science LLM Evaluation Scientific Reasoning

Ask PDF

Jul 7, 2026

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.

So Hasegawa, Shailaja Keyur Sampat, Lei Liu +1

Fujitsu Research of America, Santa Clara CA 95054, USA

Table QA LLM Evaluation

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities