cs.DBNov 4, 2026

BobFlow: Towards Human-Agent Collaborative Data Science Using Dataflows

Authors: Jiadong Bai, Yicong Huang, Chen Li

Organizations: University of California, Irvine Irvine, California, USA · University of Massachusetts Amherst Amherst, Massachusetts, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent advances in large language models (LLMs) have established human-agent collaboration as a promising paradigm for data science. Effective collaboration requires a shared computational abstraction that enables agents to reason about processing logic while allowing human analysts to understand, guide, and refine the analysis. Although scripts written in Python or SQL provide precise control, their syntax and execution details can be difficult for analysts to understand and modify. In this paper, we argue that dataflows are an effective abstraction for human-agent collaboration in data science. We introduce BobFlow, a system that combines the dataflow abstraction with the ReAct paradigm. It enables agents to iteratively construct and execute dataflows for data science tasks while allowing analysts to inspect the analysis and provide feedback. Our user study shows that BobFlow helps analysts understand, inspect, and verify agent-generated analyses. A thorough experimental evaluation across proprietary and open-weight LLMs shows accuracy score gains of up to 18.6 points at just over 1/3 of a script-based ReAct agent's inference cost.

Figures & tables

Figure 1. Three abstractions of the same movie-analysis task. Natural language leaves decisions unspecified; Python is precise but harder to understand and modify; dataflows expose operators, dependencies, and intermediate results for targeted feedback.
Figure 2. Human–agent collaboration on data science tasks.
AbstractionPrecise [-2pt] SemanticsIntuitive [-2pt] RepresentationStep-Level [-2pt] Transparency
Natural Language✗✓✗
Programming Language✓✗✓
SQL✓✗✗
Dataflow✓✓✓
Table 1. Abstractions for human–agent collaboration.
Declarative operatorPandasPySpark
scan({ path } , { options })pd.read_csv(...)spark.read.csv(...)
filter({ condition })query(...)filter(...)
distinct({ columns })drop_duplicates(...)dropDuplicates(...)
sort({ columns } , { orders })sort_values(...)orderBy(...)
limit({ offset } , { count })iloc[...]offset(...).limit(...)
project({ mapping })[[...]].rename(...)select(...)
Table 2. Declarative operators and example implementations.
Figure 3. A dataflow for ML model training and inference, with a corresponding physical plan.
Figure 4. System architecture of BobFlow.
Figure 5. Dataflow ReAct life cycle for a data science task.
ActionSpecifiesApplication
DescribeReasoningReasoningDisplay reasoning in chat.
AnswerTaskAnswerDisplay answer in chat.
GetDataflowOperators, edgesInspect operators, schemas, and edges.
SetDataflowDataflow changesAdd, delete, or update operators and edges.
GetCodev∈Vv\in VRetrieve registered physical implementation.
SetCodeCodeRegister implementation for custom operators.
Table 3. Actions for solving tasks using dataflows.
Figure 6. Assemble, Think, and Act in Dataflow ReAct.
Figure 7. Human interactions in Dataflow ReAct.
Figure 8. Full output versus samples and result statistics.
Figure 9. Pruning irrelevant observations for uu at v3v_{3} .
Figure 10. User study results for Scenarios 1 (S1) and 2 (S2).
DimensionQuestionnaire itemScript ReActBobFlowΔ\Delta
IntuitivenessEasy to collaborate with the agent3.94.0+0.1\mathbf{+0.1}
TransparencyEasy to grasp processed/produced tables3.54.3+0.8\mathbf{+0.8}
Confident about what each step does3.33.8+0.5\mathbf{+0.5}
Easy to view each step’s output tables2.64.2+1.6\mathbf{+1.6}
Easy to follow data lineage2.34.2+1.9\mathbf{+1.9}
Clear final-answer vs. exploratory steps2.94.0+1.1\mathbf{+1.1}
Table 4. Participant ratings of intuitiveness, transparency, and control (1 = strongly disagree; 5 = strongly agree). Means are shown; bold indicates the more favorable score; Δ\Delta = BobFlow −- Script ReAct.
BenchmarkAgentGPT-5.6 LunaGPT-5.6 TerraGPT-5.6 SolGPT-6.1 Sol
Acc.CostAcc.CostAcc.CostAcc.Cost
KramaBenchScript ReAct62.30.4064.36.9968.910.2278.93.22
BobFlow76.40.3072.32.7676.45.0978.41.67
PrepBenchScript ReAct69.91.2171.924.5679.445.9790.517.33
BobFlow83.30.9986.38.6788.616.5090.86.33
BenchmarkAgentGemini 3.8 FlashGemini 4 ArgonDeepSeek V4.1-FlashClaude Opus 5.5
Table 5. Task accuracy scores and inference cost ($); bold indicates the better value (higher accuracy, lower cost).
Figure 11. Input and output tokens per benchmark run.
ModelAccuracyΔ\Delta Acc.Cost ($)Δ\Delta Cost
GPT-5.6 Luna82.4 →\to 83.3+0.9+0.91.21 →\to 0.99−18.2\mathbf{-18.2} %
GPT-5.6 Terra85.9 →\to 86.3+0.4+0.410.66 →\to 8.67−18.7\mathbf{-18.7} %
GPT-5.6 Sol88.2 →\to 88.6+0.4+0.421.35 →\to 16.50−22.7\mathbf{-22.7} %
GPT-6.1 Sol88.6 →\to 90.8+2.2+2.28.39 →\to 6.33−24.6\mathbf{-24.6} %
Gemini 3.8 Flash90.2 →\to 89.5−0.7-0.716.19 →\to 13.22−18.3\mathbf{-18.3} %
Gemini 4 Argon90.2 →\to 91.5+1.3+1.317.97 →\to 15.42−14.2\mathbf{-14.2} %
Table 6. Data-property optimization on PrepBench: BobFlow without opt1 →\to BobFlow. Accuracy changes are absolute; cost changes are relative.
Figure 12. Simplified traces of four KramaBench cases comparing BobFlow and Script ReAct.
Figure 13. Median context size (top) and cache-miss rate (bottom) versus the number of preceding interactions.
ModelConfigurationAccuracyInputOutputCost ($)
(K)(K)ColdWarm
GPT-5.6 LunaBobFlow without opt242.91261.49.733.54
BobFlow42.9541.95.434.85
GPT-5.6 TerraBobFlow without opt254.6991.288.3329.52
BobFlow58.4451.549.3743.47
DeepSeek V4.1-FlashBobFlow without opt258.02394.09.872.57
Table 7. LongDS-Bench v1.1 Lite results; mean tokens in thousands per interaction.
Figure 14. Memory growth from interaction 100 onward.
Figure 15. Jump latency (left) and memory with and without tracking (right) in replays of over 5,000 snapshots.
More in this topic

Human-AI Collaboration

Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.

See all 463 papers

Explore similar work

CardsList
Jul 18, 2026

DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines

Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts. We call this disconnect the NL2Pipeline gap. To bridge it, we introduce DataFlow-Harness, a platform that guides an LLM agent to construct platform-native directed acyclic graphs (DAGs) through typed, incremental mutations rather than free-form scripts. The platform combines DataFlow-Skills for procedural guidance, a Model Context Protocol (MCP) layer that exposes the live operator registry and current pipeline state, and DataFlow-WebUI, which synchronizes conversational authoring with a visual DAG editor. On a 12-task data-engineering benchmark, DataFlow-Harness achieves a 93.3% observed end-to-end pass rate. Relative to Vanilla Claude Code, it reduces measured monetary cost by 72.5% and generation latency by 49.9%; its observed pass rate is within 0.9 percentage points of the Context-Aware Claude Code baseline while its cost is 42.8% lower. Per-task analysis indicates that Skills are most useful when construction depends on implicit procedural knowledge. These results show that live platform grounding can produce persistent, editable workflow artifacts with an observed reliability close to script-generation baselines and with lower measured construction cost and latency.

Runming He, Zhen Hao Wong, Hao Liang +4

Peking University · Institute for Advanced Algorithms Research, Shanghai · Zhongguancun Academy

LLM GroundingCode GenerationAgentic Code GenerationLLM Agent Harnesses
AskPDF
Mar 19, 2026

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .

An Luo, Jin Du, Xun Xian +12

School of Statistics, University of Minnesota · AIScientists, Inc. · Data Science Institute, University of Chicago +4

AI Agent EvaluationAI Agent BenchmarksHuman-AI CollaborationData Analysis AgentsBenchmark Design
AskPDF
Aug 11, 2026

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.

Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub +3

York University · Nanyang Technological University · Salesforce AI Research

Computer-Use Agent BenchmarksAI Agent EvaluationWorkflow AutomationAgent BenchmarksData Analysis Agents
AskPDF