cs.AISep 4, 2026
SaveDI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
Abstract
Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.
Figures & tables
| Task Types | Construction Method | |||||||
| SQL Querying | Document QA | Numeric Computation (e.g., point query, comparison) | Analytical Reasoning (e.g., trend, stats) | Document Retrieval | Document Alters Answer | Q&A Generation | Quality Check | |
| Spider | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | M | M |
| BIRD | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | M | M |
| Spider 2.0 | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | M>L | M |
| -bench | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | M>L | M |
| InfiAgent-DABench ‡ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | L>M | M, L |
Table 1: Comparison of DI-Bench against benchmarks for analytical / tool-using agents. Symbols on benchmark names indicate the released artifact: pipeline only; both pipeline and fixed dataset; no symbol means dataset only. The document retrieval column indicates whether the agent must retrieve the relevant document. The document alters answer column specifies if the answer correctness relies on the document. The pair ✗ / ✓ for these two columns (no retrieval, document needed) appears when the document affects the answer but is given to the agent up front, rather than retrieved. Construction method: M = manual, L = LLM, T = template, G = graph; “ ” orders components by contribution.
Figure 1: The DI-Bench pipeline. From a database and knowledge base, the pipeline (1) constructs an artifact linkage graph connecting metrics to required tables and governing rule documents, sampling connected subgraphs as task skeletons, (2) computes gold answers via query execution, (3) generates natural language questions backward from the answers, and (4) applies automated validation filters to verify each task.
| Edge | Meaning |
| metric table | Metric formula uses columns from table |
| table dimension | Table has column usable as a filter/grouping dimension |
| table table | Tables can be joined |
| rule metric | Rule document changes this metric’s computed answer |
| metric metric | Composite metric uses another as a component |
| rule dimension | Rule applies only when a dimension takes a specific value |
Table 2: Edge types in the artifact linkage graph. The rule metric edge is key: it connects unstructured documents to the structured computations they modify.
| Model | BR | CZ | All | Turns |
| Claude Opus 4.6 | 59.2 | 55.7 | 57.9 | 5.5 |
| DeepSeek V3.2 | 56.1 | 42.6 | 50.9 | 11.5 |
| Claude Haiku 4.5 | 52.8 | 42.9 | 49.0 | 5.5 |
| Qwen3-80B | 53.0 | 39.7 | 47.9 | 5.2 |
Table 3: Agent accuracy (%) and efficiency on DI-Bench. BR = Brazilian E-Commerce (449 tasks), CZ = Czech Financial (282 tasks), 731 total. Avg. Turns = mean LLM calls per task (max 20).
Figure 2: Knowledge vs. Knowledge-grounded analytical tasks across models. Models score highest on Knowledge tasks, lower once Analytics is added, and lowest when a retrieved rule must further modify the computation. The resulting performance drop reveals a persistent knowledge-to-computation gap: retrieving business knowledge is largely solved, but correctly applying that knowledge during computation remains a major challenge.
| Task Type | BR (449) | CZ (282) | |||
| N | Acc | N | Acc | ||
| Ranking | 92 | 14 | 53 | 29 | |
| Knowledge retrieval | 69 | 87 | 68 | 78 | |
| Threshold crossing | 81 | 78 | 42 | 43 | |
| Point query | 54 | 28 | 48 | 20 | |
| Trend analysis | 70 | 78 | 8 | 47 | |
Table 4: Task volume and accuracy ( ) by types, averaged across the four models. Knowledge retrieval, comparison, threshold, and trend are easier (binary or single-fact match); point query and ranking require exact match across multiple rows; counterfactual and decomposition are hardest, requiring multi-rule reasoning.
| Model | Rule | Oracle | No rule |
| Opus 4.6 | 35.0 | 37.3 | 53.6 |
| DeepSeek V3.2 | 34.0 | 31.4 | 54.3 |
| Haiku 4.5 | 30.0 | 29.4 | 49.3 |
| Qwen3-80B | 31.0 | 32.7 | 54.3 |
| Average | 32.5 | 32.7 | 52.9 |
Table 5: Accuracy (%) on 175 rule-grounded tasks. Task content and answer format are fixed across conditions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Operators | Example Question | Gold Answer (example) |
| Knowledge retrieval | Ret | What constitutes an outlier that should be excluded when calculating the average delivery time? | ‘‘Delivery duration exceeding sixty (60) calendar days is excluded as a data anomaly.’’ |
| Point query | Ret Fil Agg | For our Q2 2018 performance review, what percentage of transactions were paid with credit cards? | [{payment_type: credit_card, pct: 80.08}] |
| Trend analysis | Ret Tmp Agg | Show me the monthly trend of our average delivery time throughout 2017. | {direction: increasing, total_change_pct: 32.1, peak_period: 2018-01} |
| Comparison | Ret Fil Agg Cmp | Which of these two states — PR or AL — had a faster average delivery time for shipped and delivered orders during Q3 2017? | {group_a: PR, group_b: AL, value_a: 10.7, value_b: 28.52, higher: AL} |
| Ranking | Ret Fil Agg Rnk | Which 5 states have the highest number of active sellers? | [{SP: 1638}, {PR: 316}, {MG: 220}, {SC: 172}, {RJ: 175}] |
| Significance testing | Ret Fil Agg Tst | Are the average installment counts for orders paid by credit card and boleto in the first quarter of 2018 statistically significantly different from each other? | {significant: true, p_value: 0.0, test_name: Welch’s t-test (two-tailed)} |
Table 6: Analysis type definitions with operator compositions, example questions, and a representative gold answer (Ret = Retrieve, Fil = Filter, Agg = Aggregate, Tmp = Temporal, Cmp = Compare, Rnk = Rank, Tst = Test, Grp = Group). Gold answers range from a single value or pass/fail verdict to multi-row breakdowns that must match exactly, which explains the accuracy ordering in Table 4 .
| Brazilian | Czech | |
| Number of rule documents | 17 | 17 |
| Mean word count | 821 | 756 |
| Min / max word count | 654 / 1132 | 592 / 928 |
| Median word count | 847 | 721 |
| Rule types represented | 6 | 6 |
Table 7: Descriptive statistics for the LLM-generated rule documents. “Rule types represented” shows the count out of 6 possible types (definitional, exclusionary, temporal, conditional, privacy, override).
| Pairing strategy | Discriminative validity |
| Graph | 82% |
| Semantic | 29% |
| Random | 22% |
Table 8: Discriminative validity under three rule-pairing strategies on identical task skeletons ( rule tasks, both domains pooled). A pairing is discriminative if applying the rule changes the computed answer. The graph selects rules that genuinely modify the computation 3 more often than random pairing; semantic similarity helps only marginally, since a topically related rule need not alter the metric’s value.
| Analysis Type | Metric(s) | Tables | dimension | Rules | Score |
| Point query | Late Delivery Rate by Carrier Segment | orders, customers, sellers, order_items | customer_state (spatial) | Active Seller Definition | 0.958 |
| Trend analysis | Average Delivery Time | orders | order_purchase _timestamp (temporal) | Outlier Exclusion (Delivery) | 0.807 |
| Comparison | CLV, Revenue per Order, CSAT | customers, orders, order_items | order_status (categorical) | CLV Lifespan, CSAT Calculation, Revenue Recognition, …(11 total) | 0.898 |
| Ranking | Seller Fulfillment Score | order_items, orders, order_reviews | order_status (categorical) | Seller Fulfillment Score Weights | 0.884 |
| Significance testing | Payment Method Distribution | order_payments | payment_type (categorical) | Undefined Payment Exclusion | 0.879 |
| Threshold crossing | Unique Buyer Count | orders, customers | order_status (categorical) | Customer Deduplication | 0.846 |
Table 9: Representative selected subgraph for each analysis type from the Brazilian E-commerce domain. Scores reflect the composite quality metric from Equation 1 .
| Analysis Type | Metric(s) | Tables | dimension | Rules | Score |
| Point query | Transaction Volume | trans | operation (categorical) | Account Opening Date Rule | 0.866 |
| Trend analysis | Balance Volatility (CV) | trans | trans.date (temporal) | Small Sample Suppression | 0.801 |
| Comparison | Credit Risk Score | loan, trans, account, district, client | district region (spatial) | Credit Risk Score Weights | 1.000 |
| Ranking | Average Account Balance | trans, account | k_symbol (categorical) | Fee Structure Change 1996, Inter-Bank Transaction Flag | 0.863 |
| Significance testing | Standing Order Value by Type | order | k_symbol (categorical) | Standing Order Amount Validation | 0.858 |
| Threshold crossing | Debt Service Coverage Ratio | trans, loan, account | k_symbol (categorical) | Cohort Maturity Threshold, NPL Denominator | 0.849 |
Table 10: Representative selected subgraph for each analysis type from the Czech Financial domain.
| Dimension | Human | Validator | Admitted |
| (100) | (100) | (25) | |
| Obj. | 87% | 64% | 100% |
| Leakage | 100% | 100% | 100% |
| Natural. | 100% | 100% | 100% |
| Relevance | 92% | 83% | 100% |
| Format | 98% | 77% | 96% |
Table 11: Human-in-the-loop validation (100-task stratified sample). Human : expert-annotator pass rate (ground truth); Validator : LLM validator pass rate on the same tasks; Admitted : human-verified pass rate of the 25 tasks the validator admits (all eight dimensions pass from the validator). The validator is stricter than the human across dimensions, so what it admits is near-perfect (99% average). Its conservatism reduces yield (admits 25% of candidates) but indicates high precision among validator-admitted tasks.
| Brazilian | Czech | |
| Table nodes | 9 | 8 |
| Metric nodes | 29 | 29 |
| Rule nodes | 17 | 17 |
| Dimension nodes | 21 | 17 |
| Total nodes | 76 | 71 |
| metric table edges | 61 | 45 |
Table 12: Artifact linkage graph statistics for both domains.
| Analysis Type | N | Opus | Haiku | Qwen | DeepSeek |
| Ranking | 145 | 23 | 20 | 11 | 23 |
| Knowledge retrieval | 137 | 89 | 85 | 78 | 76 |
| Threshold crossing | 123 | 72 | 61 | 65 | 67 |
| Point query | 102 | 34 | 22 | 18 | 23 |
| Trend analysis | 78 | 78 | 63 | 78 | 78 |
| Comparison | 63 | 79 | 59 | 65 | 65 |
Table 13: Per-model accuracy (%) by analysis type on the full 731-task benchmark (Brazilian e-commerce and Czech financial combined). N is the number of tasks of each type.
Figure 3: Five-panel reference implementation: graph construction, weighted candidate selection, SQL grounding and backward generation, validation and export, and executable stage commands. Policy-bearing tasks are retained only when rule removal changes the executed answer.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,494 papers