Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses

Huachi Zhou; Yujing Zhang; Jiahe Du; Jiacheng Cai; Zijin Hong; Chuang Zhou; Zheng Yuan; Qinggang Zhang

cs.AISep 28, 2026

Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses

Authors: Huachi Zhou, Yujing Zhang, Jiahe Du, Jiacheng Cai, Zijin Hong, Chuang Zhou, Zheng Yuan, Qinggang Zhang, +2 more

Organizations: The Hong Kong Polytechnic University, Hong Kong, China · Jilin University, Changchun, China

Abstract

Large language model agents are increasingly deployed for data-intensive work, yet reliable data analysis requires more than general-purpose reasoning and ad hoc tool augmentation. Data Agents, equipped with workflow harnesses, offer a promising paradigm for automating the end-to-end data science lifecycle. This paper examines Data Agents from a harness-centric perspective. First, we introduce a taxonomy of Data Agents and associated data environments, organizing the literature around five functional stages: perception, planning, execution, verification, and repair. Second, we analyze the key technical routes within each stage, identifying 15 distinct approaches ranging from data structure probing to data state reconstruction. Third, we identify four open reliability problems: inactive semantic calibration, missing clarification, missing experience transfer, and the missing verification-repair repository. These problems explain why silent failures can persist even when individual components function correctly, highlighting the need for rigorous workflow harnesses and shared reliability resources. Finally, we summarize the horizontal task families of Data Agents, examine their vertical application settings, and benchmarks for evaluation, while maintaining a companion repository at https://github.com/DEEP-PolyU/Awesome-Data-Agents.

Figures & tables

Figure 1. The workflow harness of AI Data Scientists. The harness governs Data Agent workflows through five functional stages, including perception, planning, execution, verification, and repair. These stages are invoked iteratively as needed to maintain the evolving data state across data tasks.

Figure 3. Technical routes for perception across heterogeneous data environments: Data Structure Probing, Cross-Source Data Alignment, and Confidence-Calibrated State Scoring.

Figure 4. Technical routes for planning: Tree Search, Heuristic Search, and Strategy Routing.

Figure 5. Technical routes for execution: Tool Capability Boundary Learning, Compositional Tool Reasoning, and Feedback-Adaptive Tool Calling.

Figure 6. Technical routes for verification: Process Score Estimation, Rule-based Constraint Checking, and Trace-Level Explainable Attribution.

Figure 7. Technical routes for repair: Data State Reconstruction, Search-Guided Intervention-Based Repair, and Reusable Memory Skills.

Figure 8. We identify four open reliability problems that prevent Data Agents from producing reliable analytical outcomes: (1) the Inactive Semantic Calibration Problem, (2) the Missing Clarification Problem, (3) the Missing Experience Transfer Problem, and (4) the Missing Verification-Repair Repository Problem.

Benchmark	Focus	Task & Process Coverage	Evaluation
StockGQL ( Liang et al., 2024 )	Natural-language-to-GQL translation over structured financial knowledge.	Data Querying; Execution, Verification	Query correctness and retrieval of the required information.
TableBench ( Wu et al., 2025 )	Table question answering covering fact checking, numerical reasoning, data analysis, and visualization.	Data Querying; Verification	TableQA accuracy across multiple reasoning categories.
Visual-TableQA ( Lompo and Haraoui, 2026 )	Visual reasoning over rendered tables, including structure understanding and multi-step reasoning.	Visualization and Multimodal Analysis; Verification	Question-answering and reasoning accuracy on table images.
TopBench ( Ji et al., 2026 )	Implicit predictive reasoning over tabular data, including prediction, decision making, and treatment-effect analysis.	Analysis and Prediction; Verification	Performance on analytical objectives beyond direct table lookup.
PrepBench ( Xu et al., 2026a )	Natural-language-driven data preparation involving cleaning, restructuring, and table transformation.	Data Preparation; Execution, Verification	Correctness of generated output tables across preparation settings.
InfiAgent-DABench ( Hu et al., 2024 )	End-to-end data analysis over CSV files requiring agents to interact with an execution environment.	Analysis and Prediction; Planning, Execution	Automatically evaluated answers across diverse analytical questions.

Table 1. Representative benchmarks for evaluating autonomous Data Agents across analytical tasks.

Survey / Study	Capability	Data	Lifecycle	Evaluation	Stage-wise	Reliability
Survey of Data Agents ( Zhu et al., 2025b )	✓	✓	$\circ$	$\circ$	–	$\circ$
Autonomous Data Agents ( Fu et al., 2025b )	✓	$\circ$	✓	$\circ$	–	$\circ$
LLM $\times$ DATA ( Zhou et al., 2025 )	✓	✓	$\circ$	$\circ$	–	–
LLM/Agent-as-Data-Analyst ( Tang et al., 2025b )	✓	✓	$\circ$	$\circ$	–	$\circ$
LLM-Based Data Science Agents ( Rahman et al., 2025 )	✓	✓	✓	✓	$\circ$	$\circ$
Clean Up Your Mess ( Zhou et al., 2026a )	✓	✓	$\circ$	✓	–	$\circ$

Table 2. Comparison on Data Agents and related LLM-based data systems across capabilities, data environments, lifecycle organization, evaluation, stage-wise technical methods, and workflow reliability. ✓ indicates systematic coverage,

\circ

indicates partial or secondary coverage, and – indicates no systematic coverage.

Explore similar work

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Aug 11, 2026Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub +3Data Science Agents Memoryagentbench
York University · Nanyang Technological University · Salesforce AI Research
Data Agents: Agentic Data Systems
Sep 21, 2026Guoliang Li, Peiyao Zhou, Xuanhe Zhou +3Data Science Agents Production Agentic Systems
Department of Computer Science, Tsinghua University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China · The Hong Kong University of Science and Technology (Guangzhou), China +1
Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI
Sep 8, 2026Arun Vignesh Malarkkan, Xinyuan Wang, Yanjie FuAgent Loop Trustworthy Artificial Intelligence
School of Computing and Augmented Intelligence, Arizona State University, USA

Aug 11, 2026, cs.AIJ/K move · Enter open · S save

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar+2

York University · Nanyang Technological University · Salesforce AI Research

Data Science Agents Memoryagentbench Scientific Workflows First Comprehensive Benchmark

Open paper AskPDF arXiv

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.

Abstract

Explore similar work

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

Data Agents: Agentic Data Systems

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI