Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.
Figures & tables
Figure 1. Quality and cost comparison of BI-Agent vs. other methods, including frontier models (e.g., GPT-5.5) and SOTA NL2SQL systems (e.g., Kwai-AutoSQL and ktx), on BI-Bench using SQL. Complete results on 24 models and systems can be found in Tables 3 and 4 . BI-Agent using data-management tools significantly improves frontier models by over 10 percentage points on average (shown by teal arrows); while BI-Agent with post-training (shown by purple arrows) enables Qwen3-8B to achieve quality comparable to much larger frontier models, at substantially lower costs (up to 50 × cheaper).
Figure 2. ( a ): An example BI dashboard built by users in the wild, extracted from a real Power BI project file (.pbix). For BI-Bench , we manually select a visualization whose title can accurately describe user intent: e.g., the visualization in the dashed pink box corresponds to a user question Q = ‘‘Sales amount, budget, and forecast comparison by month and year’’ . ( b ): We directly retrieve the answer A(Q) for Q by exporting the underlying result table of the selected visualization as a CSV file through the “Export data” menu option (in the green box). This (Q,A(Q)) pair becomes the basis of a test case in BI-Bench . ( c ): The input data tables of this BI project in a visual data model view. Note that this schema graph follows a “multi-snowflake” pattern common in BI, with the “Budget & Forecast” and “Sales” tables at the center serving as “fact tables” (in orange boxes), and the remaining 7 tables serving as “dimension tables”.
Figure 3. ( Left ): a raw ‘‘Sales’’ table imported from spreadsheets. This is a “pivot table” in a cross-tabulation form and not properly relationalized, with different months as column headers, which makes it hard to perform relational analysis, such as joining on or aggregating over months. ( Right ): A properly relationalized version of the same ‘‘Sales’’ table, using the unpivot transformation, where both aggregating over and joining on ‘‘Month’’ become straightforward.
Figure 4. BI-Bench : distribution by domains.
Figure 5. Architecture overview of BI-Agent . The top layer shows BI-Agent uses different LLMs (vanilla vs. post-trained) as the orchestrator, to plan and execute a reasoning/tool-call loop. The middle layer shows tools such as general coding and data-management tools that can be invoked by BI-Agent , which BI-Agent uses to interact with raw data files in the bottom.
Figure 6. Data generation pipeline to automatically synthesize training trajectories for model post-training.
Python No Tool
Python Tool
SQL No Tool
SQL Tool
#SFT Turns
19,563
28,698
23,100
42,686
#RL Tasks
2,975
4,586
2,049
3,426
Table 1. Training data statistics.
SQL
Python
Qwen3-8B Variants
no-tools
with-tools
no-tools
with-tools
SFT
(11) Qwen3-8B-SFT (SQL)
(12) Qwen3-8B-SFT-Tool (SQL)
(13) Qwen3-8B-SFT (Python)
(14) Qwen3-8B-SFT-Tool (Python)
SFT + RL
(15) Qwen3-8B-RL (SQL)
(16) Qwen3-8B-RL-Tool (SQL)
(17) Qwen3-8B-RL (Python)
(18) Qwen3-8B-RL-Tool (Python)
Table 2. A total of 8 post-trained model variants of Qwen3-8B, using (SFT, SFT+RL) × (No tools, Tools) × (SQL, Python).
SQL
Python
Models
no-tools
with-tools
no-tools
with-tools
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
o4-mini
48.2
10.3
61.9 ( ↑+13.7 ) ∗∗∗ [+7.7,+19.7]
7.64
57.4
3.95
66.3 ( ↑+8.9 ) ∗ [+1.8,+15.8]
5.61
GPT-5.5
46.7
7.17
60.2 ( ↑+13.5 ) ∗∗∗ [+8.1,+19.4]
14.91
59.5
8.66
65.4 ( ↑+5.9 ) ∗ [+1.0,+11.0]
16.97
GPT-5.2
46.1
3.03
55.2 ( ↑+9.1 ) ∗∗∗ [+4.4,+14.2]
5.77
48.6
3.08
60.2 ( ↑+11.6 ) ∗∗∗ [+6.1,+17.1]
5.58
Mistral-Large-3
36.2
4.81
44.8 ( ↑+8.6 ) ∗∗ [+2.7,+14.7]
6.06
35.0
3.23
51.1 ( ↑+16.1 ) ∗∗∗ [+10.0,+22.5]
6.20
Table 3. BI-Agent ’s data management tools can greatly improve quality for end-to-end BI: Accuracy (%) and cost per query (0.01)comparisonacrossdifferentmodelsonBI−Bench.Weannotatequalitygainswith(\bm{\uparrow}\bm{+x}).Gainsmarked∗∗∗(p{<}0.001),∗∗(p{<}0.01),and∗(p{<}0.05)arestatisticallysignificant(pairedt−test,n{=}100$ ); 95% CIs [l,r] are obtained from paired bootstrap.
SQL
Python
Models and Agents
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
BI-Agent
Qwen3-8B-SFT
22.6 ( ↑+17.4 ) ∗∗∗
0.17
17.2 ( ↑+11.8 ) ∗∗∗
1.89
[+11.9,+23.3]
[+7.3,+16.8]
Qwen3-8B-SFT-Tool
27.2 ( ↑+22.0 ) ∗∗∗
0.31
22.6 ( ↑+17.2 ) ∗∗∗
1.61
[+15.6,+29.1]
[+11.4,+23.3]
Table 4. BI-Agent ’s post-training framework significantly improves quality: Accuracy (%) and cost per query (0.01)comparisonsacrossdifferentpost−trainedQwen3−8BvariantsandSOTANL2SQLmethodsandcodingagentsonBI−Bench.Weannotateoverallgains(\bm{\uparrow}\bm{+x}),andwhenapplicable,breakthemdownintotoolgain(\bm{\uparrow}{\bm{Tool}},\bm{+x})andpost−traininggain(\bm{\uparrow}{\bm{Train}},\bm{+x}).Allgainsmarked∗∗∗(p{<}0.001),∗∗(p{<}0.01),and∗(p{<}0.05)arestatisticallysignificant(pairedt−test,n{=}100$ ); 95% CIs [l,r] are obtained from paired bootstrap.
Figure 11
Figure 9. Post-training quality gains (with and without tools), by task complexity. We group tasks by (left) number of tables, (middle) number of joins, and (right) transformations. Error bars are one standard error of the mean across BI-Bench tasks.
Figure 10. Error distributions for (a) tool vs. no-tool settings averaged over all large models and (b) base vs. post-trained Qwen3-8B.
SQL
Python
Models
no-tools
with-tools
no-tools
with-tools
Qwen3-8B
0.8
4.9
0
1.6
Qwen3-8B-SFT
10.7
13.1
2.5
4.9
Qwen3-8B-RL
13.1
16.4
3.3
8.2
Table 5. BI-Agent ’s data management tools and post-training improve performance on Spider 2.0, an unseen NL2SQL benchmark (best results are in bold and underlined).
Figure 11. Number of interaction rounds (a) and total output tokens (b) required to complete a task on BI-Bench under different settings, across all models (overall, correct, and incorrect executions); and interaction rounds across post-training stages of Qwen3-8B (c).
Business intelligence (BI) increasingly combines dashboard interaction with LLM-based assistance, but these two modes often fall out of sync during multi-step analysis. As users switch between direct dashboard manipulation and natural-language queries, it becomes difficult to preserve a consistent analytical state across filters, hierarchies, metrics, and chart context. We present TwinBI, an agentic digital-twin framework that couples an LLM-based agent system with an executable BI dashboard state. TwinBI unifies conversational interaction, dashboard manipulation, semantic grounding, and provenance tracking through a shared analytical state reconstructed from a unified interaction log. It also exposes artifacts such as schema views, SQL, logs, and an /insights command for state-grounded analytical summaries. We evaluate TwinBI in two complementary ways. In a controlled A/B benchmark with the same backbone agent, TwinBI improves exact-match accuracy from 43.3% to 63.3%, partial-credit accuracy from 48.3% to 70.8%, and substantially reduces timeout rate from 40.0% to 10.0% relative to Dashboard alone. In a usability study, participants benefited from the integrated dashboard-and-chat workflow, with high task accuracy, moderate workload, and favorable ratings for state-aware interaction mechanisms. These results suggest that TwinBI improves both agent-level analytical reliability and user-facing analytical support by turning visible dashboard state into richer actionable context. Our dataset and source code are available at: https://github.com/simonjisu/TwinBI
Jisoo Jang Wen-Syan Li
Graduate School of Data Science, Seoul National University
Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.
Jiangyun Zhang, Kristen Surrao, Torpong Nitayanont +10
Transforming fragmented enterprise data into actionable insights remains a significant challenge for LLMs, constrained by complex database schemas, limitations in dynamic SQL generation, and the need for deep multi-dimensional analysis.In this paper, we propose AIDA(Autonomous Insight Discovery Agent), the first end-to-end framework designed for autonomous exploration in complex business environments. We establish a highly flexible instant retail environment encompassing 200+ metrics and 100+ dimensions, and integrates a proprietary Domain-Specific Language (DSL) that bridges semantic reasoning with precise SQL execution. Our reinforcement learning system subsequently formulates business analysis as a Pareto Principle-guided cumulative reasoning process. Experimental results demonstrate that AIDA significantly outperforms workflow-based agents, and extensive evaluations further reveal that AIDA achieves superior environmental perception and more in-depth analysis from diverse perspectives. Our work ultimately establishes the transformative potential of autonomous intelligence for industrial-scale business intelligence systems.
Dongming Wu, Junwen Li, Ming Lu +2
1Rajax Network Technology(Taobao Shangou of Alibaba)