cs.LGMar 19, 2026
SaveAgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
Organizations: School of Statistics, University of Minnesota · AIScientists, Inc. · Data Science Institute, University of Chicago · Carlson School of Management, University of Minnesota · Cisco Research · Department of Electrical and Computer Engineering, University of Minnesota · Division of Computational Health Sciences, University of Minnesota
Abstract
Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .
Figures & tables
Figure 1: Overview of AgentDS. The framework comprises three stages: (1) data preparation, where domain-specific challenges are built using realistic synthetic multimodal data and validated by experts; (2) a time-limited competition supporting global participation and human–AI collaboration; and (3) results, including submissions, shared insights, awards, and community engagement, which provide empirical evidence on human, AI, and collaborative performance.
| Domain | Challenge | Problem | Metric | Additional Modalities |
| Commerce | Demand Forecasting | Regression | RMSE | Text, CSV |
| Product Recommendation | Ranking | NDCG@10 | Image, CSV | |
| Coupon Redemption | Classification | Macro-F1 | JSON | |
| Food Production | Shelf Life Prediction | Regression | MAE | JSON |
| Quality Control | Classification | Macro-F1 | Image, JSON | |
| Demand Forecasting | Regression | RMSE | Text, CSV |
Table 1: An Overview of Challenges in AgentDS Across Six Domains
Figure 2: Overall quantile score comparison between all AI baselines and competition teams (n=29). Among agentic baselines, MorphMind (dark green, score: 0.510) ranks 9th, Claude Code (orange-red, score: 0.458) ranks 10th, and Julius AI (rose, score: 0.382) ranks 11th, all above the participant median of 0.156 (dashed line). Among direct prompting LLM baselines, GPT-5.5 (dark gray, score: 0.415) ranks 11th, Claude Opus 4.7 (tan, score: 0.391) ranks 11th, DeepSeek V4 Pro (blue, score: 0.325) ranks 11th, Gemini 3.1 Pro (purple, score: 0.216) ranks 14th, and GPT-4o (light gray, score: 0.183) ranks 14th. Bars are sorted descending by score (Team 1 = best); all AI baselines are inserted at their rank positions. The overall quantile score is the mean of domain scores, each of which is itself the mean of per-challenge quantiles within that domain, with 1.0 indicating best performance and 0.0 indicating non-participation. The result shows that current AI-only baselines, whether using direct prompting or agentic coding, do not match the performance of the top human teams in the competition, highlighting a substantial gap between AI automation and human-AI collaboration.
Figure 3: Distribution of domain-level quantile scores across all participants (teal dots), with agentic baselines indicated by filled circles (MorphMind: dark green; Claude Code: orange-red; Julius AI: rose) and direct prompting LLM baselines by filled triangles (Claude Opus 4.7: tan; Gemini 3.1 Pro: purple; DeepSeek V4 Pro: blue; GPT-5.5: dark gray; GPT-4o: light gray). Agentic baselines outperform all direct prompting LLMs across most domains: MorphMind has its highest domain scores in Healthcare (0.699) and Manufacturing (0.647), Claude Code in Retail Banking (0.553), and Julius AI in Healthcare (0.584) and Manufacturing (0.500). Among LLM baselines, GPT-5.5 and Claude Opus 4.7 are competitive in several domains (e.g., Food Production: 0.573 and 0.603 respectively). Nonetheless, all baselines remain well below the top-performing participants in each domain, confirming that current agentic coding tools cannot yet replicate the domain-specific strategies of expert human data scientists.
Figure 4: Challenge-specific quantile score distributions across six domains. Teal dots represent participants who submitted for each challenge (zero-score non-participants excluded from display); gray dashed lines indicate per-challenge participant medians among submitters. Agentic baselines are shown as filled circles (MorphMind: dark green; Claude Code: orange-red; Julius AI: rose) and direct prompting LLM baselines as filled triangles (Claude Opus 4.7: tan; Gemini 3.1 Pro: purple; DeepSeek V4 Pro: blue; GPT-5.5: dark gray; GPT-4o: light gray). MorphMind obtains high challenge-level scores on several tasks, including Manufacturing Ch. 1 (0.786), Healthcare Ch. 1 (0.750), and Healthcare Ch. 3 (0.783). Claude Code records its largest gains over GPT-4o in Retail Banking Ch. 1 (0.737 vs. 0.316) and Commerce Ch. 1 (0.625 vs. 0.000). Julius AI peaks on Insurance Ch. 1 (0.786), Food Production Ch. 2 (0.714), and Healthcare Ch. 1 (0.708), but scores zero on Food Production Ch. 1, Commerce Ch. 2, Insurance Ch. 3, and Retail Banking Ch. 1. Among direct prompting LLMs, GPT-5.5 peaks on Food Production Ch. 1 (0.933) and Commerce Ch. 3 (0.800), while Claude Opus 4.7 leads in Retail Banking Ch. 1 (0.789). No baseline achieves top-quartile performance on every challenge, suggesting that current AI-only approaches still fall short of the best human-AI collaboration teams under this evaluation protocol.
Figure 5: Performance of five direct-prompting LLM baselines (GPT-5.5, Claude Opus 4.7, DeepSeek V4 Pro, Gemini 3.1 Pro, GPT-4o) on the AgentDS benchmark, compared with the best human–AI collaboration team (dashed teal). Each radial axis is a quantile in : the fraction of non-zero human submissions a baseline outperforms on that challenge, averaged across challenges per domain or modality. Left (By domain): mean per-challenge quantile within each of six domains. Right (By modality): mean per-challenge quantile over all challenges tagged with each additional data modality. GPT-5.5 ranks highest in this cohort (overall 0.415), with its strongest domain results in Food Production (0.573) and Manufacturing (0.526), but near-floor modality scores on PDF (0.040) and Text (0.130). Claude Opus 4.7 (overall 0.391) leads direct-prompting models on Retail Banking (0.605) and performs moderately on JSON (0.440). DeepSeek V4 Pro (overall 0.325) is middling and uneven across domains. Gemini 3.1 Pro (overall 0.216) and GPT-4o (overall 0.183) trail substantially. No direct-prompting model approaches the best human–AI collaboration team (overall 0.860; domain quantiles 0.740–1.000; modality quantiles 0.830–0.920). Structured modalities (JSON, CSV) represent the relative strength of LLMs, occasionally reaching 0.4–0.6, while Image and PDF remain near 0.1–0.3 for most models. LLMs underperform the best human–AI collaboration on every domain and modality, with particularly sharp deficits on multimodal and document-heavy tasks.
Figure 6: Performance of agentic coding baselines (MorphMind, Julius AI, Claude Code) on the AgentDS benchmark, compared with the best human–AI collaboration team (dashed teal). Each radial axis is a quantile in : the fraction of non-zero human submissions a baseline outperforms on that challenge, averaged across challenges per domain or modality. Left (By domain): for each of six domains, the plotted value is the mean of per-challenge quantiles within that domain. Right (By modality): for each additional data modality (JSON, CSV, Image, PDF, Text), the value is the mean quantile over all challenges whose additional modality set contains that modality. Among the three agentic coding baselines, MorphMind has an overall mean domain quantile of 0.510, with domain scores of 0.699 in Healthcare and 0.647 in Manufacturing, and modality scores of 0.590 on PDF and 0.570 on JSON. Claude Code has an overall mean domain quantile of 0.458, with higher scores in Retail Banking (0.553) and Commerce (0.450). Julius AI has an overall mean domain quantile of 0.382 , with relatively higher performance on Text (0.580) and lower scores on PDF (0.240) and Retail Banking (0.130). However, all agentic baselines occupy only the inner half of the radar, indicating a large gap between current agentic coding tools and the best human-AI collaboration performance across every domain and modality.