Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.
Figures & tables
Figure 1. Quality and cost comparison of BI-Agent vs. other methods, including frontier models (e.g., GPT-5.5) and SOTA NL2SQL systems (e.g., Kwai-AutoSQL and ktx), on BI-Bench using SQL. Complete results on 24 models and systems can be found in Tables 3 and 4 . BI-Agent using data-management tools significantly improves frontier models by over 10 percentage points on average (shown by teal arrows); while BI-Agent with post-training (shown by purple arrows) enables Qwen3-8B to achieve quality comparable to much larger frontier models, at substantially lower costs (up to 50 × cheaper).
Figure 2. ( a ): An example BI dashboard built by users in the wild, extracted from a real Power BI project file (.pbix). For BI-Bench , we manually select a visualization whose title can accurately describe user intent: e.g., the visualization in the dashed pink box corresponds to a user question Q = ‘‘Sales amount, budget, and forecast comparison by month and year’’ . ( b ): We directly retrieve the answer A(Q) for Q by exporting the underlying result table of the selected visualization as a CSV file through the “Export data” menu option (in the green box). This (Q,A(Q)) pair becomes the basis of a test case in BI-Bench . ( c ): The input data tables of this BI project in a visual data model view. Note that this schema graph follows a “multi-snowflake” pattern common in BI, with the “Budget & Forecast” and “Sales” tables at the center serving as “fact tables” (in orange boxes), and the remaining 7 tables serving as “dimension tables”.
Figure 3. ( Left ): a raw ‘‘Sales’’ table imported from spreadsheets. This is a “pivot table” in a cross-tabulation form and not properly relationalized, with different months as column headers, which makes it hard to perform relational analysis, such as joining on or aggregating over months. ( Right ): A properly relationalized version of the same ‘‘Sales’’ table, using the unpivot transformation, where both aggregating over and joining on ‘‘Month’’ become straightforward.
Figure 4. BI-Bench : distribution by domains.
Figure 5. Architecture overview of BI-Agent . The top layer shows BI-Agent uses different LLMs (vanilla vs. post-trained) as the orchestrator, to plan and execute a reasoning/tool-call loop. The middle layer shows tools such as general coding and data-management tools that can be invoked by BI-Agent , which BI-Agent uses to interact with raw data files in the bottom.
Figure 6. Data generation pipeline to automatically synthesize training trajectories for model post-training.
Python No Tool
Python Tool
SQL No Tool
SQL Tool
#SFT Turns
19,563
28,698
23,100
42,686
#RL Tasks
2,975
4,586
2,049
3,426
Table 1. Training data statistics.
SQL
Python
Qwen3-8B Variants
no-tools
with-tools
no-tools
with-tools
SFT
(11) Qwen3-8B-SFT (SQL)
(12) Qwen3-8B-SFT-Tool (SQL)
(13) Qwen3-8B-SFT (Python)
(14) Qwen3-8B-SFT-Tool (Python)
SFT + RL
(15) Qwen3-8B-RL (SQL)
(16) Qwen3-8B-RL-Tool (SQL)
(17) Qwen3-8B-RL (Python)
(18) Qwen3-8B-RL-Tool (Python)
Table 2. A total of 8 post-trained model variants of Qwen3-8B, using (SFT, SFT+RL) × (No tools, Tools) × (SQL, Python).
SQL
Python
Models
no-tools
with-tools
no-tools
with-tools
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
o4-mini
48.2
10.3
61.9 ( ↑+13.7 ) ∗∗∗ [+7.7,+19.7]
7.64
57.4
3.95
66.3 ( ↑+8.9 ) ∗ [+1.8,+15.8]
5.61
GPT-5.5
46.7
7.17
60.2 ( ↑+13.5 ) ∗∗∗ [+8.1,+19.4]
14.91
59.5
8.66
65.4 ( ↑+5.9 ) ∗ [+1.0,+11.0]
16.97
GPT-5.2
46.1
3.03
55.2 ( ↑+9.1 ) ∗∗∗ [+4.4,+14.2]
5.77
48.6
3.08
60.2 ( ↑+11.6 ) ∗∗∗ [+6.1,+17.1]
5.58
Mistral-Large-3
36.2
4.81
44.8 ( ↑+8.6 ) ∗∗ [+2.7,+14.7]
6.06
35.0
3.23
51.1 ( ↑+16.1 ) ∗∗∗ [+10.0,+22.5]
6.20
Table 3. BI-Agent ’s data management tools can greatly improve quality for end-to-end BI: Accuracy (%) and cost per query (0.01)comparisonacrossdifferentmodelsonBI−Bench.Weannotatequalitygainswith(\bm{\uparrow}\bm{+x}).Gainsmarked∗∗∗(p{<}0.001),∗∗(p{<}0.01),and∗(p{<}0.05)arestatisticallysignificant(pairedt−test,n{=}100$ ); 95% CIs [l,r] are obtained from paired bootstrap.
SQL
Python
Models and Agents
Accuracy ↑
Cost ↓
Accuracy ↑
Cost ↓
BI-Agent
Qwen3-8B-SFT
22.6 ( ↑+17.4 ) ∗∗∗
0.17
17.2 ( ↑+11.8 ) ∗∗∗
1.89
[+11.9,+23.3]
[+7.3,+16.8]
Qwen3-8B-SFT-Tool
27.2 ( ↑+22.0 ) ∗∗∗
0.31
22.6 ( ↑+17.2 ) ∗∗∗
1.61
[+15.6,+29.1]
[+11.4,+23.3]
Table 4. BI-Agent ’s post-training framework significantly improves quality: Accuracy (%) and cost per query (0.01)comparisonsacrossdifferentpost−trainedQwen3−8BvariantsandSOTANL2SQLmethodsandcodingagentsonBI−Bench.Weannotateoverallgains(\bm{\uparrow}\bm{+x}),andwhenapplicable,breakthemdownintotoolgain(\bm{\uparrow}{\bm{Tool}},\bm{+x})andpost−traininggain(\bm{\uparrow}{\bm{Train}},\bm{+x}).Allgainsmarked∗∗∗(p{<}0.001),∗∗(p{<}0.01),and∗(p{<}0.05)arestatisticallysignificant(pairedt−test,n{=}100$ ); 95% CIs [l,r] are obtained from paired bootstrap.
Figure 11
Figure 9. Post-training quality gains (with and without tools), by task complexity. We group tasks by (left) number of tables, (middle) number of joins, and (right) transformations. Error bars are one standard error of the mean across BI-Bench tasks.
Figure 10. Error distributions for (a) tool vs. no-tool settings averaged over all large models and (b) base vs. post-trained Qwen3-8B.
SQL
Python
Models
no-tools
with-tools
no-tools
with-tools
Qwen3-8B
0.8
4.9
0
1.6
Qwen3-8B-SFT
10.7
13.1
2.5
4.9
Qwen3-8B-RL
13.1
16.4
3.3
8.2
Table 5. BI-Agent ’s data management tools and post-training improve performance on Spider 2.0, an unseen NL2SQL benchmark (best results are in bold and underlined).
Figure 11. Number of interaction rounds (a) and total output tokens (b) required to complete a task on BI-Bench under different settings, across all models (overall, correct, and incorrect executions); and interaction rounds across post-training stages of Qwen3-8B (c).