Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.
Figures & tables
Task Types
Construction Method
SQL Querying
Document QA
Numeric Computation (e.g., point query, comparison)
Analytical Reasoning (e.g., trend, stats)
Document Retrieval
Document Alters Answer
Q&A Generation
Quality Check
Spider
✓
✗
✓
✗
✗
✗
M
M
BIRD
✓
✗
✓
✗
✗
✗
M
M
Spider 2.0
✓
✗
✓
✗
✗
✓
M>L
M
τ -bench
✗
✓
✓
✗
✗
✓
M>L
M
InfiAgent-DABench ‡
✗
✗
✓
✓
✗
✗
L>M
M, L
Table 1: Comparison of DI-Bench against benchmarks for analytical / tool-using agents. Symbols on benchmark names indicate the released artifact: † pipeline only; ‡ both pipeline and fixed dataset; no symbol means dataset only. The document retrieval column indicates whether the agent must retrieve the relevant document. The document alters answer column specifies if the answer correctness relies on the document. The pair ✗ / ✓ for these two columns (no retrieval, document needed) appears when the document affects the answer but is given to the agent up front, rather than retrieved. Construction method: M = manual, L = LLM, T = template, G = graph; “ > ” orders components by contribution.
Figure 1: The DI-Bench pipeline. From a database and knowledge base, the pipeline (1) constructs an artifact linkage graph connecting metrics to required tables and governing rule documents, sampling connected subgraphs as task skeletons, (2) computes gold answers via query execution, (3) generates natural language questions backward from the answers, and (4) applies automated validation filters to verify each task.
Edge
Meaning
metric → table
Metric formula uses columns from table
table → dimension
Table has column usable as a filter/grouping dimension
table → table
Tables can be joined
rule → metric
Rule document changes this metric’s computed answer
metric → metric
Composite metric uses another as a component
rule → dimension
Rule applies only when a dimension takes a specific value
Table 2: Edge types in the artifact linkage graph. The rule → metric edge is key: it connects unstructured documents to the structured computations they modify.
Model
BR
CZ
All
Turns
Claude Opus 4.6
59.2
55.7
57.9
5.5
DeepSeek V3.2
56.1
42.6
50.9
11.5
Claude Haiku 4.5
52.8
42.9
49.0
5.5
Qwen3-80B
53.0
39.7
47.9
5.2
Table 3: Agent accuracy (%) and efficiency on DI-Bench. BR = Brazilian E-Commerce (449 tasks), CZ = Czech Financial (282 tasks), 731 total. Avg. Turns = mean LLM calls per task (max 20).
Figure 2: Knowledge vs. Knowledge-grounded analytical tasks across models. Models score highest on Knowledge tasks, lower once Analytics is added, and lowest when a retrieved rule must further modify the computation. The resulting performance drop reveals a persistent knowledge-to-computation gap: retrieving business knowledge is largely solved, but correctly applying that knowledge during computation remains a major challenge.
Task Type
BR (449)
CZ (282)
N
Acc
N
Acc
Ranking
92
14
53
29
Knowledge retrieval
69
87
68
78
Threshold crossing
81
78
42
43
Point query
54
28
48
20
Trend analysis
70
78
8
47
Table 4: Task volume and accuracy ( % ) by types, averaged across the four models. Knowledge retrieval, comparison, threshold, and trend are easier (binary or single-fact match); point query and ranking require exact match across multiple rows; counterfactual and decomposition are hardest, requiring multi-rule reasoning.
Model
Rule
Oracle
No rule
Opus 4.6
35.0
37.3
53.6
DeepSeek V3.2
34.0
31.4
54.3
Haiku 4.5
30.0
29.4
49.3
Qwen3-80B
31.0
32.7
54.3
Average
32.5
32.7
52.9
Table 5: Accuracy (%) on 175 rule-grounded tasks. Task content and answer format are fixed across conditions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Operators
Example Question
Gold Answer (example)
Knowledge retrieval
Ret
What constitutes an outlier that should be excluded when calculating the average delivery time?
‘‘Delivery duration exceeding sixty (60) calendar days is excluded as a data anomaly.’’
Point query
Ret → Fil → Agg
For our Q2 2018 performance review, what percentage of transactions were paid with credit cards?
[{payment_type: credit_card, pct: 80.08}]
Trend analysis
Ret → Tmp → Agg
Show me the monthly trend of our average delivery time throughout 2017.
Are the average installment counts for orders paid by credit card and boleto in the first quarter of 2018 statistically significantly different from each other?
Table 6: Analysis type definitions with operator compositions, example questions, and a representative gold answer (Ret = Retrieve, Fil = Filter, Agg = Aggregate, Tmp = Temporal, Cmp = Compare, Rnk = Rank, Tst = Test, Grp = Group). Gold answers range from a single value or pass/fail verdict to multi-row breakdowns that must match exactly, which explains the accuracy ordering in Table 4 .
Brazilian
Czech
Number of rule documents
17
17
Mean word count
821
756
Min / max word count
654 / 1132
592 / 928
Median word count
847
721
Rule types represented
6
6
Appendix
Table 7: Descriptive statistics for the LLM-generated rule documents. “Rule types represented” shows the count out of 6 possible types (definitional, exclusionary, temporal, conditional, privacy, override).
Pairing strategy
Discriminative validity
Graph
82%
Semantic
29%
Random
22%
Appendix
Table 8: Discriminative validity under three rule-pairing strategies on identical task skeletons ( N=175 rule tasks, both domains pooled). A pairing is discriminative if applying the rule changes the computed answer. The graph selects rules that genuinely modify the computation ∼ 3 × more often than random pairing; semantic similarity helps only marginally, since a topically related rule need not alter the metric’s value.
Table 9: Representative selected subgraph for each analysis type from the Brazilian E-commerce domain. Scores reflect the composite quality metric q(S) from Equation 1 .
Analysis Type
Metric(s)
Tables
dimension
Rules
Score
Point query
Transaction Volume
trans
operation (categorical)
Account Opening Date Rule
0.866
Trend analysis
Balance Volatility (CV)
trans
trans.date (temporal)
Small Sample Suppression
0.801
Comparison
Credit Risk Score
loan, trans, account, district, client
district region (spatial)
Credit Risk Score Weights
1.000
Ranking
Average Account Balance
trans, account
k_symbol (categorical)
Fee Structure Change 1996, Inter-Bank Transaction Flag
0.863
Significance testing
Standing Order Value by Type
order
k_symbol (categorical)
Standing Order Amount Validation
0.858
Threshold crossing
Debt Service Coverage Ratio
trans, loan, account
k_symbol (categorical)
Cohort Maturity Threshold, NPL Denominator
0.849
Appendix
Table 10: Representative selected subgraph for each analysis type from the Czech Financial domain.
Dimension
Human
Validator
Admitted
(100)
(100)
(25)
Obj.
87%
64%
100%
Leakage
100%
100%
100%
Natural.
100%
100%
100%
Relevance
92%
83%
100%
Format
98%
77%
96%
Appendix
Table 11: Human-in-the-loop validation (100-task stratified sample). Human : expert-annotator pass rate (ground truth); Validator : LLM validator pass rate on the same tasks; Admitted : human-verified pass rate of the 25 tasks the validator admits (all eight dimensions pass from the validator). The validator is stricter than the human across dimensions, so what it admits is near-perfect (99% average). Its conservatism reduces yield (admits 25% of candidates) but indicates high precision among validator-admitted tasks.
Brazilian
Czech
Table nodes
9
8
Metric nodes
29
29
Rule nodes
17
17
Dimension nodes
21
17
Total nodes
76
71
metric → table edges
61
45
Appendix
Table 12: Artifact linkage graph statistics for both domains.
Analysis Type
N
Opus
Haiku
Qwen
DeepSeek
Ranking
145
23
20
11
23
Knowledge retrieval
137
89
85
78
76
Threshold crossing
123
72
61
65
67
Point query
102
34
22
18
23
Trend analysis
78
78
63
78
78
Comparison
63
79
59
65
65
Appendix
Table 13: Per-model accuracy (%) by analysis type on the full 731-task benchmark (Brazilian e-commerce and Czech financial combined). N is the number of tasks of each type.
Figure 3: Five-panel reference implementation: graph construction, weighted candidate selection, SQL grounding and backward generation, validation and export, and executable stage commands. Policy-bearing tasks are retained only when rule removal changes the executed answer.
Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results. Established text-to-SQL benchmarks evaluate query generation alone, and audits have found their answer keys frequently wrong. Because real enterprise warehouses are too sensitive to release, these benchmarks are built on public datasets where a business event fits in a single table. We introduce Argo-Bench, an evaluation framework comprising 210 data science and analytics tasks. Drawing on public data, peer-reviewed industry literature, and regulatory filings, we simulate a food delivery platform in New York City at true scale, with 81 million orders in 2024, grounded economics, fraud patterns, and marketplace incentives. We export this world to an ERP warehouse of 235 tables and 7.5 billion rows, modeled on the Oracle E-Business Suite schema. The simulator's ground-truth state is withheld from the warehouse the agent sees, so tasks require reconstructing facts by navigating the warehouse before acting on them. Argo-Bench goes beyond text-to-SQL: the agent files actions such as banning fraudulent accounts, allocating courier incentive budgets, or issuing back pay, and the grader scores each by its consequences in the simulator. Every task has an executable reference solution that demonstrates solvability using only the warehouse. The strongest of 14 frontier and open-weight models scores 95 or higher on only 34.8% of tasks and averages 59.5 points. We hope Argo-Bench drives progress toward agents that understand, navigate, and act within real data environments.
Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing +2
Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance. We introduce DRBENCHER, a synthetic benchmark generator for questions that require both browsing and computation, synthesized answer-first from knowledge-graph chains with no seed passages or provenance text. It enforces four criteria: verifiability (gold answers are computed by executing parameterized code over knowledge-graph values), complexity (multi-hop entity identification, property retrieval, and domain-specific computation), difficulty (a two-stage verification cascade filters out questions solvable by the generating model), and diversity (a greedy max-min embedding filter maximizes coverage). These criteria are realized via a unified answer-first pipeline spanning five domains: biochemistry, financial, geophysical, security, and history. Human evaluation shows 76% validity, with 35% of errors due to outdated knowledge-graph entries, highlighting an inherent limitation of systems that reason over evolving data. Automatic evaluation shows that the strongest frontier model achieves only 20% answer accuracy. Compared to manually constructed benchmarks (BrowseComp+, MATH-500, GPQA), DRBENCHER achieves the highest semantic diversity. We release DrBencher pipeline and the human-verified benchmark data at https://github.com/IBM/DrBencher.
Young-Suk Lee, Ramon Fernandez Astudillo, Radu Florian
Production data integration is bottlenecked by repeated, lossy handoffs between data owners, engineers, and analysts who must collaboratively discover, structure, and query enterprise data. We present Data Intelligence Agents (DIA), a system of three agents (Data Interpreter, Schema Creator, and Query Generator) that compresses this workflow by treating autonomous coding agents (ACAs) as a first-class abstraction: rather than emitting text, the agents generate, execute, validate, and repair concrete artifacts, draw on a shared memory for experience reuse, and surface each for review by domain experts. DIA is deployed in production for enterprise customers. We study the Query Generator in depth and evaluate it in fully autonomous mode across seven SQL benchmarks spanning four task categories and four dialects. It matches or surpasses the best published results on all seven, demonstrating that an architecture grounded in execution, built on ACAs and a shared memory, generalizes across the data intelligence workload with adaptation confined to natural-language instructions.
Anoushka Vyas, Aarushi Dhanuka, Sina Khoshfetrat Pakazad +1