TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

Boshen Shi; Yize Liu; Chen Zhao; Ce Chi; Zhendong Wang; Xing Wang; Junlan Feng

cs.CLAug 25, 2026

TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

Authors: Boshen Shi, Yize Liu, Chen Zhao, Ce Chi, Zhendong Wang, Xing Wang, Junlan Feng

Organizations: China Mobile Jiutian Artificial Intelligence Technology (Beijing) Co., Ltd. · School of Computer and Cyberspace Security, Communication University of China

Abstract

LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant reasoning are still needed for reliable structured-data analysis.

Figures & tables

Figure 1: Motivating example of assumption-driven analysis on an unanswerable table task.

Figure 2: Overview of the generation framework.

Dimension	Dataset	#Instances	Sources (coverage)	Avg./source	Excel / CSV	#Perturb. types	#Table files	Rows (avg./max)	Cols (avg./max)
							Table profile
Reliability	AIDABench-QA	562	201/226 (88.94%)	2.8	538 / 24	7	274	1,869.8 / 261,726	20.2 / 238
	DABench	643	242/257 (94.16%)	2.7	2 / 641	6	244	1,578.3 / 16,683	10.0 / 19
Robustness	AIDABench-QA	672	152/226 (67.26%)	4.4	643 / 29	12	205	1,624.3 / 44,648	14.6 / 100
	DABench	463	221/257 (85.99%)	2.1	0 / 463	6	405	5,016.9 / 133,464	8.8 / 251

Table 1: Benchmark Dataset statistics.

	AIDABench-QA				DABench				Avg.
Model	FRR ( $\uparrow$ )	PRR ( $\downarrow$ )	NRR ( $\downarrow$ )	MRS ( $\uparrow$ )	FRR ( $\uparrow$ )	PRR ( $\downarrow$ )	NRR ( $\downarrow$ )	MRS ( $\uparrow$ )	MRS ( $\uparrow$ )
Qwen3.7-Max	9.61	10.14	80.25	14.68	13.37	3.58	83.05	15.16	14.92
DeepSeek-V4-Pro	6.76	11.92	81.32	12.72	9.64	3.89	86.47	11.59	12.16
GLM-5.2	3.57	9.82	86.61	8.48	6.69	6.07	87.25	9.72	9.10
Gemini-3.1-Pro	4.27	8.01	87.72	8.27	4.98	1.40	93.62	5.68	6.98
Claude-Sonnet-5	11.03	9.07	79.89	15.57	6.69	9.80	83.51	11.59	13.58

Table 2: Reliability results. Avg. MRS is the mean across the two datasets.

Figure 3: Model-level MRS distributions across reliability operators.

Figure 4: Response modes across reliability operators.

	AIDABench-QA		DABench		Avg.
Model	ASR( $\downarrow$ )	RAD( $\downarrow$ )	ASR( $\downarrow$ )	RAD( $\downarrow$ )	ASR( $\downarrow$ )
Qwen3.7-Max	15.78	7.18	7.56	4.83	11.67
GLM-5.2	11.27	5.62	9.85	7.13	10.56
DeepSeek-V4-Pro	16.42	7.72	9.93	6.87	13.17
Gemini-3.1-Pro	16.90	7.68	7.18	4.83	12.04
Claude-Sonnet-5	11.76	6.73	6.44	4.51	9.10

Table 3: Robustness results.

Figure 5: Model-level ASR distributions across robustness operators.

Figure 6: Operator-level ASR across robustness difficulty.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Task Setting	Reliability	Robustness	# Perturb. / Failure Types	Workflow
RobuT ( Zhao et al. 2023 )	TQA	No	Yes	10 perturbation types	No
FREB-TQA ( Zhou et al. 2024 )	TQA	No	Yes	8 perturbations	No
ToRR ( Ashury-Tahan et al. 2026 )	TQA	No	Yes	11 prompt configurations: 7 serializations + 4 structural perturbations	No
( Wolff and Hulsebos 2025 )	Data Analysis	No	Yes	3 variations	No
RADAR ( Gu et al. 2025 )	Data Analysis	No	Yes	5 data artifact types	Code-based analysis
Spider-Syn ( Gan et al. 2021 )	SQL	No	Yes	1 synonym-substitution strategy	No

Appendix

Table 4: Comparison with benchmarks on table QA, data analysis, and SQL reliability or robustness. Reliability denotes explicit evaluation of unanswerability, abstention, ambiguity, calibration, or error detection. Robustness denotes explicit evaluation under input, schema, table, or data perturbations.

Field	Track	Meaning
sample_id	Both	Identifier of the source instance being screened.
eligible_attacks	Both	Operators judged applicable enough to enter construction.
attack_type	Both	Closed-set operator name copied from the enabled catalog.
confidence	Both	Selector confidence that the operator has a concrete feasible target.
reason	Both	Short justification for why the operator is applicable or rejected.
required_edit	Reliability	Whether the perturbation changes only the question, modifies files, or removes files from the delivered package.

Appendix

Table 5: Selection protocol fields. Reliability and robustness share the same purpose but differ slightly in target granularity and invariance checks.

Field	Track	Meaning
status	Both	Whether the candidate was successfully constructed or rejected.
attack_type	Both	Operator executed by the constructor.
new_question	Both	Final question text; unchanged for all robustness perturbations and most reliability perturbations.
expected_answer	Reliability	Gold refusal or clarification target, including the specific missing or conflicting evidence.
file_edit_required	Both	Whether the final package differs from the original files.
output_files input_file	Both	Delivered files and the file list exposed to evaluated LLMs.

Appendix

Table 6: Construction protocol fields. The constructor output records the actual intervention and the evidence needed for later rule-based and blind validation.

Field	Track	Meaning
verdict	Both	Final pass/fail decision for accepting the generated instance.
independent_check_completed	Reliability	Whether the validator independently inspected the evidence rather than trusting the construction record.
original_answerable original_answer	Both	Whether the original task is answerable, or the independently recomputed original answer.
attack_rule_valid attack_effective	Both	Whether the perturbation satisfies the intended operator rule and has a real effect.
question_preservation_valid	Reliability	Whether the question-change policy for the reliability operator is obeyed.
file_edit_scope_valid	Reliability	Whether only allowed files, fields, or package membership were modified.

Appendix

Table 7: Validation and blind-judge protocol fields. These fields record an independent decision rather than the constructor’s claimed success.

Operator	Track	AIDABench-QA	DABench	Total
FDM	Reliability	175	221	396
DM	Reliability	109	173	282
HC	Reliability	92	116	208
EC	Reliability	78	70	148
DAM	Reliability	60	61	121
FLM	Reliability	32	0	32

Appendix

Table 8: Operator distribution in the final benchmark. Counts are accepted perturbations used for evaluation.

Failure stage	Operator	Share
Schema/evidence binding	FDM	27.1%
Schema ambiguity detection	HC	19.6%
Value-sufficiency check	DM	18.4%
Evidence consistency check	EC	16.7%
Multi-step dependency propagation	DAM	10.3%
Input-completeness check	FLM	4.5%

Appendix

Table 9: Failure stages among AIDABench-QA no-refusal cases.

Panel of Figure 7 — Figure 7: The model detects a header conflict but proceeds by treating a parser-generated column name as evidence. Red text marks the unsupported reasoning, tool choice, and final answer; each annotation explains why that step fails.

Panel of Figure 8 — Figure 8: GLM-5.2 notices duplicate evidence but reduces validity to a numeric/non-missing check and continues the regression.

Panel of Figure 9 — Figure 9: Qwen3.6-27B correctly detects the absent file but converts unavailable evidence into a zero-valued comparison set.

Panel of Figure 10 — Figure 10: Gemini-3 Pro substitutes an invented mean for a missing target field, using scale compatibility as if it were a scoring specification.

Panel of Figure 11 — Figure 11: Gemini-3 Pro notices unequal valid counts but continues with two different denominators instead of triggering a stop condition.

Panel of Figure 12 — Figure 12: DeepSeek-V4-Pro completes the downstream pipeline by relabeling a nearby field as the missing denominator.

Panel of Figure 13 — Figure 13: GPT-5.5 notices that monthly snapshot context is absent but substitutes hire month, yielding a coherent table for the wrong grouping semantics.

Panel of Figure 14 — Figure 14: Qwen3.7-Max performs extensive cleaning after an unsupported same-header disambiguation; downstream care cannot repair the missing schema evidence.

Panel of Figure 15 — Figure 15: Claude Sonnet 5 follows the executable aggregation path without testing whether same-entity records provide consistent evidence.

Panel of Figure 16 — Figure 16: Qwen3.7-Max completes an early ranking step, then bridges the missing late-stage evidence by conflating battery type with battery brand.

Explore similar work

Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
Aug 26, 2026Srimonti Dutta, Akshata Kishore MoharirLLM Agent Evaluation LLM Answer Verification
WAI USA Research Labs
Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents
Aug 17, 2026Yike Yuan, Virum Ranka, Tina Lasisi +1Data Analysis Agents Agent Benchmarks
University of Michigan Ann Arbor, Michigan, USA
Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction
Feb 24, 2026Hui Wen Goh, Jonas MuellerLLM Answer Verification LLM Uncertainty Estimation
Cleanlab

Aug 26, 2026J/K move · Enter open · S save

Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems

Srimonti Dutta, Akshata Kishore Moharir

WAI USA Research Labs

LLM Agent Evaluation LLM Answer Verification LLM Auditing Text-to-SQL

Open paper AskPDF arXiv

Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We identify the Structure Gap as the deployment failure mode that makes Trace Integrity necessary: natural-language reasoning and free-form rationales do not reliably specify the operator-level programs required by real-world systems. We operationalize Trace Integrity with execution contracts, structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes. In an empirical demonstration on BIRD Mini-Dev, Direct SQL, Operation Summary + SQL, and Contract-First SQL achieve answer accuracies of 20%, 22%, and 24%, while their Trace Integrity Pass Rates are 39%, 43%, and 40% and their CAIT Rates remain high at 55%, 59.1%, and 45.8%, showing that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals. Real-world LLM data agents should, therefore, be evaluated not only by whether their outputs match a reference answer, but by whether those outputs are backed by auditable computation.

Abstract

Explore similar work

Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems

Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents

Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction