TML-Bench: Benchmark for Data Science Agents on Tabular ML Tasks

Mykola Pinchuk

cs.LGMar 5, 2026

TML-Bench: Benchmark for Data Science Agents on Tabular ML Tasks

Authors: Mykola Pinchuk

Organizations: Independent Researcher, San Jose, USA

Abstract

Autonomous coding agents can produce strong tabular baselines quickly on Kaggle-style tasks. Practical value depends on end-to-end correctness and reliability under time limits. This paper introduces TML-Bench, a tabular benchmark for data science agents on Kaggle-style tasks. This paper evaluates 10 OSS LLMs on four Kaggle competitions and three time budgets (240s, 600s, and 1200s). Each model is run five times per task and budget. A run is successful if it produces a valid submission and a private-holdout score on hidden labels that are not accessible to the agent. This paper reports median performance, success rates, and run-to-run variability. MiniMax-M2.1 model achieves the best aggregate performance score on all four competitions under the paper's primary aggregation. Average performance improves with larger time budgets. Scaling is noisy for some individual models at the current run count. Code and materials are available at https://github.com/MykolaPinchuk/TML-bench/tree/master.

Figures & tables

Figure 1: Aggregate performance leaderboard (primary aggregation: best budget per competition).

Figure 2: Per-competition ranks (1=best).

Figure 3: Performance vs stability (each dot is one model; fill color indicates success rate).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	Type	Parameters	Release date	Knowledge cutoff	License
Qwen3-Coder-480B-A35B	chutes	open weights	480B total, 35B active	2025-07-23	2025-01-22	Apache-2.0
gpt-oss-120b	chutes	open weights	120B	2025-08-05	2024-06-01	Apache-2.0
GLM-4.7-FP8	chutes	open weights	358B	2025-12-22	2025-04	MIT
GLM-4.7-Flash	chutes	open weights	30B	2025-12-22	2025-04	MIT
MiniMax-M2.1-TEE	chutes	open weights	230B total, 10B active	2025-12-23	2025-06-25	unknown
GLM-4.6-FP8	chutes	open weights	357B	2025-09-30	2025-04	MIT

Appendix

Table 1: Model inventory for the 10 models evaluated in this paper.

Figure 5: Overall aggregation (all competitions and all budgets).

Figure 7: Rank-based aggregation (best budget per competition).

Figure 8: Rank-based aggregation (all competitions and all budgets).

Figure 9: Rank-based aggregation (1200s only).

Figure 10: Rank standard deviation across competitions.

Figure 13: Marginal gains with increasing budget.

Figure 14: Monotonicity rate across budgets.

Competition	Problem and target	Metric	Train rows	Test rows	Features
bank-customer-churn-ict-u-ai	Binary classification: predict customer churn ( Exited )	AUC	12,000	3,000	12
foot-traffic-wuerzburg-retail-forecasting-2-0	Regression: forecast foot traffic ( target )	RMSE	41,155	10,289	4
playground-series-s5e10	Regression: predict road accident risk ( accident_risk )	RMSE	414,203	103,551	12
playground-series-s6e1	Regression: predict exam scores ( exam_score )	RMSE	504,000	126,000	11

Appendix

Table 2: Competition summary for the four tasks used in TML-bench. Feature counts exclude the id column and target column.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories
Aug 20, 2026Josias Moukpe, Priyanka Aryal, Matthew KenneyAI Agent Benchmarks LLM Agent Evaluation
Algorithmic Research Group
TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development
Aug 26, 2026Jiarui Yan, Weiwei Sun, Sijie Li +2Long-Horizon Agent Evaluation AI Agent Benchmarks
Carnegie Mellon University
DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Aug 11, 2026Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub +3Computer-Use Agent Benchmarks AI Agent Evaluation
York University · Nanyang Technological University · Salesforce AI Research

Aug 20, 2026J/K move · Enter open · S save

DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories

Josias Moukpe, Priyanka Aryal, Matthew Kenney

Algorithmic Research Group

AI Agent Benchmarks LLM Agent Evaluation Autonomous Experimentation Autonomous Research Agents AI Agents for Scientific Discovery

Open paper AskPDF arXiv

Autonomous agents for machine learning experimentation must navigate heterogeneous repositories, repair training pipelines, and evaluate candidate improvements under realistic compute constraints. Existing benchmarks only partially capture these conditions. We introduce DeltaML-Bench, a benchmark comprising 48 tasks sourced from research papers that require agents to improve published baselines within imperfect, open-source repositories. We evaluate GPT-5 and Claude Sonnet 4 with a standard Modular agent and a search-based ARG scaffolding. In the 4 x 6h allocation, ARG raises GPT-5's per-run success rate from 9.4% to 33.9%; in the 2 x 12h allocation, GPT-5 ARG reaches 49.0%. Modular configurations exhibit specification gaming rates as high as 47.9%, while no gaming is observed in the evaluated ARG configurations. These results indicate that scaffolding design and integrity checks are important considerations when deploying agents for autonomous ML experimentation.

Abstract

AI Agent Benchmarks

Explore similar work

DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories

TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?