As AI-driven document understanding and processing tools become increasingly prevalent in real-world applications, the need for rigorous evaluation standards has grown increasingly urgent. Existing benchmarks and evaluations often focus on isolated capabilities or simplified scenarios, failing to capture the end-to-end task effectiveness required in practical settings. To address this gap, we introduce AIDABench, a comprehensive benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner. AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation. These tasks are grounded in realistic scenarios involving heterogeneous data types, including spreadsheets, databases, financial reports, and operational records, and reflect analytical demands across diverse industries and job functions. Notably, the tasks in AIDABench are sufficiently challenging that even human experts require 1-2 hours per question when assisted by AI tools, underscoring the benchmark's difficulty and real-world complexity. We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families. Our results reveal that complex, real-world data analytics tasks remain a significant challenge for current AI systems, with the best-performing model achieving only 59.43% pass-at-1. We provide a detailed analysis of failure modes across each capability dimension and identify key challenges for future research. AIDABench offers a principled reference for enterprise procurement, tool selection, and model optimization, and is publicly available at https://github.com/MichaelYang-lyx/AIDABench.
Figures & tables
Figure 1: Overview of the AIDABench evaluation framework. The workflow illustrates the pipeline from multi-format Raw Data ingestion, through Intermediate Processing (encompassing the capability dimensions of data editing/transformation and numerical/statistical reasoning in a non-sequential flow), to the final Delivery Result (corresponding to the QA, data visualization, and file generation dimensions).
Figure 2: Three example evaluation scenarios in the benchmark. (a) QA: answer users’ data analysis questions based on the provided data; (b) Data Visualization: create visualizations based on users’ questions and the provided data. Each scenario shows the corresponding Query and the expected Reference output format; (c) File Generation: generate spreadsheets according to users’ requirements.
Figure 3: The design of three types of evaluators in AIDABench.
pass@3
pass@1
Model
Model Size
Overall
QA
Data Vis.
File Gen.
Overall
QA
Data Vis.
File Gen.
claude-sonnet-4-5
Undisclosed
70.78
72.12
78.28
66.28
59.43
68.58
67.71
49.43
gemini-3-pro-preview
Undisclosed
69.25
71.24
78.97
63.22
57.39
64.60
63.87
48.28
qwen3-max-2026-01-23-thinking
Undisclosed
66.34
71.68
75.05
57.85
56.67
66.81
64.41
44.83
deepseek-v3.2-thinking
685B
65.72
69.47
71.83
59.77
53.81
62.39
60.83
43.30
deepseek-v3.2
685B
65.47
69.91
75.68
57.09
51.35
60.18
62.51
40.61
Table 1: Performance on AIDABench under pass@3 and pass@1. Data Vis. denotes Data Visualization , and File Gen. denotes File Generation . Model sizes are reported when available; otherwise marked as Undisclosed . Best results in each column are highlighted in bold.
Model
Overall
Δ
QA
Δ
Data Vis.
Δ
File Gen.
Δ
claude-sonnet-4-5
58.62
↓0.84
68.14
↓3.39
65.63
↑1.67
47.25
↑0.26
gemini-3-pro-preview
56.37
↑0.38
61.95
↑2.01
64.45
↓0.36
47.95
↓0.70
qwen3-max-2026-01-23-thinking
55.95
↓1.41
65.63
↓1.32
62.68
↑2.73
44.57
↓3.32
deepseek-v3.2-thinking
52.12
↓0.51
60.32
↓0.14
58.62
↓0.12
42.15
↓1.03
deepseek-v3.2
50.99
↑0.25
58.7
↑0.59
62.17
↑2.75
39.34
↓1.15
qwen3-max-2026-01-23
50.26
↓0.85
59.23
↓2.45
58.51
↑0.62
38.83
↓0.13
Table 2: Ablation on auxiliary spreadsheet summary ( w/ vs. w/o ). For each setting, all scores are reported as avg@3 (mean over three independent runs). The table shows the w/ scores, and Δ denotes the change when removing the summary, i.e., Δ=score(w/o summary)−score(w/ summary) . ↑ denotes improvement and ↓ denotes degradation. Data Vis. = Data Visualization; File Gen. = File Generation.
Figure 4: Error composition within bad cases across scenarios. Heatmaps report, for each model and scenario (QA, Data Visualization, and File Generation), the percentage share of each error type among the bad cases. Redder colors indicate higher shares. Model names are shown on the left panel, and a single color scale is shared across all panels.
Table 3: Recommended decoding hyperparameters per model. We follow each provider’s recommended temperature and adopt the associated sampling parameters (Top- p , Top- k ) when specified. Unspecified parameters are left to provider defaults.
Figure 5: Example of auxiliary spreadsheet summarization input: screenshot of the workbook Revenue_Calculation_Details.xlsx (active sheet excerpt).
Model
Pass@3 (Easy)
Pass@3 (Medium)
Pass@3 (Hard)
claude-sonnet-4-5
84.34
71.31
50.54
gemini-3-pro-preview
86.97
68.30
46.65
qwen3-max-2026-01-23-thinking
82.31
67.15
42.08
deepseek-v3.2-thinking
83.67
66.78
38.10
deepseek-v3.2
83.25
65.92
39.51
qwen3-max-2026-01-23
79.84
64.95
36.60
Appendix
Table 4: Pass@3 performance stratified by task difficulty. Difficulty is defined by operation-chain length (number of key expert operations/steps): easy ( ≤6 ), medium ( 7 – 12 ), and hard ( ≥13 ), covering 29.5 %, 49.4 %, and 21.1 % of tasks, respectively.
Figure 6: Token consumption across models. Bars show the average tokens used in each scenario (QA, Data Visualization, and File Generation), and the line shows the overall average tokens per task. Models are sorted by overall average token consumption (descending).
Figure 7: Round consumption across models. Bars show the average number of interaction rounds in each scenario (QA, Data Visualization, and File Generation), and the line shows the overall average rounds per task. Models are sorted by overall average round consumption (descending).
Figure 8: Illustration of structural misalignment caused by semantic ambiguity in tabular data instructions. The diagram contrasts the intended execution (top right), which adheres to the user’s explicit "three-column" constraint using a selection and sort operation, against the model’s erroneous interpretation (bottom right). The model hallucinated an unrequested pivot operation, transforming the data into a sparse wide-format matrix instead of the requested long-format tidy data.
Figure 9: Garbled non-English text in generated visualizations. Left: the model fails to select a font with proper CJK support, causing Chinese labels to appear garbled or as missing-glyph placeholders. Right: with a language-compatible font, the same labels are rendered correctly, improving readability while keeping the chart content unchanged.
Figure 10: Numerical calculation error due to hierarchical layout parsing failure. The figure demonstrates a failure mode where the model is unable to correctly interpret the multi-row header structure of the spreadsheet. While the reference approach accurately locates the target data (highlighted in the red box), the model misaligns the table structure and erroneously retrieves a value from an unrelated row.
Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.
So Hasegawa, Shailaja Keyur Sampat, Lei Liu +1
Fujitsu Research of America, Santa Clara CA 95054, USA
Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.
Jiangyun Zhang, Kristen Surrao, Torpong Nitayanont +10
General-purpose agents are increasingly expected to perform professional analytics work, which requires sustained reasoning, iterative investigation, and reliable evidence. Training these capabilities requires diverse and challenging tasks. However, generating tasks directly from raw data often overlooks deeper questions that emerge through analysis and provides limited evidence for verification. To address these limitations, we introduce DataPraxis, an analytics task synthesis framework based on verified workspace evolution. The framework progressively enriches workspaces with validated findings and artifacts, then inherits and combines these workspaces to uncover deeper questions. It constructs tasks from accumulated findings and their dependencies, verifying analytical results both during evolution and after task assembly. Using this framework, we construct DataPraxis-5K, a dataset of 5,000 verified trajectories spanning diverse domains and output formats, to train DataPraxis-27B and DataPraxis-35B-A3B. DataPraxis-27B achieves 55.11 on DABStep and 51.9 on JobBench, matching or surpassing frontier models such as GLM-5.2. DataPraxis-5K is available at huggingface.co/datasets/stargazerhust/DataPraxis-5k
Nan Jia, Wenxuan Shi, Jingqing Ruan +11
Huazhong University of Science and Technology · Institute of Computing Technology, Chinese Academy of Sciences · Meituan +3