SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

Sungho Park; Jueun Kim; Wook-Shin Han

cs.CLFeb 26, 2026

SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

Authors: Sungho Park, Jueun Kim, Wook-Shin Han

Organizations: POSTECH, Pohang, Republic of Korea

Abstract

Real-world Table-Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated - and therefore error-prone - and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table-Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question-answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. Our benchmark, construction code, and baseline models are available at https://github.com/pshlego/SPARTA/tree/main.

Figures & tables

Figure 1: Representative examples of our SPARTA benchmark (see Appendix M for more examples).

Benchmark	Table Size		Question Generation	Grouping/Having	Query Shape Supported		Multi-hop Reasoning		Annotation Error Rate (over 100 sampled queries)
Benchmark	#Col	#Row	Question Generation	Grouping/Having	Chain ( $\!>\!3$ -Hop)	Star	Cross-modal	Uni-modal	Annotation Error Rate (over 100 sampled queries)
TAT-QA Zhu et al. (2021)	4.0	9.4	Manual	✗	✗	✗	✓	✗	30%
FinQA Chen et al. (2021)	–	6.4	Manual	✗	✗	✗	✓	✗	27%
MULTIHIERITT Zhao et al. (2022)	5.0	10.8	Manual	✗	✗	✗	✓	✓	26%
HybridQA Chen et al. (2020b)	4.4	15.7	Manual	✗	✗	✗	✓	✗	21%
OTT-QA Chen et al. (2020a)	4.4	15.7	Manual	✗	✗	✗	✓	✗	21%

Table 1: Comparison of Table–Text QA benchmarks (see Appendix A for detailed annotation audit results).

Figure 2: Overview of SPARTA : (1) Reference Fact Database Construction, (2) Query Generation, (3) Question Verbalisation. ST and GT denote a source table and a grounding table, respectively.

Figure 3: Overview of provenance-based refinement.

Metric	Definition
Success-Q	# of non-nested queries that execute without error and return at least one row.
Exec-Err	# of statements that fail at parse or runtime, revealing schema or logic errors.
Empty-Q	# of syntactically valid queries that return zero rows because predicates are too restrictive.
Duplicate-Q	# of queries whose result duplicates a previously generated query, reducing diversity.
Ideal Calls	# of LLM invocations required if every step succeeds on the first attempt (baseline cost).
Total Calls	# of actual LLM invocations, i.e., Ideal Calls plus extra calls for provenance-guided fixes or other retries.

Table 2: Cost metrics used for benchmark generation.

Method	Success-Q	Empty-Q	Duplicate-Q	Exec-Err	Ideal Calls	Total Calls	Wall Time (s)
Non-nested Query Generation
One-Shot	500	60	1265	5	500	1830	4256.96
Clause	500	51	78	0	1053	1316	3218.83
Execution-Guided	500	0	27	0	1058	1134	2466.47
Nested Query Generation
One-Shot– $k$	600	0	0	0	2664	13962	115316.67

Table 3: Generation Cost Comparison of Query Generation Methods.

Figure 4: Comparison of Query Naturalness for Different Generation Methods.

Method	SPARTA																HybridQA
	NBA				Movie				Medical				Avg.
	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R
ODYSSEY w/ GPT-3.5-turbo	9.0	15.1	26.8	14.8	20.2	23.9	33.6	24.7	6.7	22.9	33.2	21.3	12.0	20.6	31.2	20.3	32.7	42.2	42.6	44.2
HProPro w/ GPT-3.5-turbo	11.0	13.6	16.4	13.8	22.2	27.8	29.1	29.2	15.5	19.5	20.2	19.7	16.2	20.3	21.9	20.9	21.4	25.3	25.7	26.1
ODYSSEY w/ GPT-5	21.2	28.4	38.4	28.1	20.4	24.2	32.9	24.3	47.5	54.2	60.3	54.2	29.7	35.6	43.9	35.5	55.3	69.5	69.3	73.5
HProPro w/ GPT-5	23.6	33.1	36.2	34.0	36.6	47.1	49.2	48.8	28.1	41.0	43.2	41.6	29.5	40.4	42.9	41.5	59.7	70.5	71.1	73.1

Table 4: Table-Text QA Accuracy on the SPARTA (Oracle) across multiple domains.

Method	SPARTA																OTT-QA
	NBA				Movie				Medical				Avg.
	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R
HELIOS+FiE Reader	4.6	6.9	17.6	6.4	8.6	11.9	23.0	11.6	6.6	16.0	33.0	12.9	6.6	11.6	24.5	10.3	58.6	65.2	66.7	65.2
HELIOS+HProPro w/ GPT-5	14.5	19.0	24.2	18.6	17.4	21.6	28.6	21.7	13.7	27.3	31.3	27.1	15.2	22.6	28.0	22.5	47.7	56.0	57.4	56.5

Table 5: Table-Text QA Accuracy on the SPARTA (Retrieval) across multiple domains.

Panel of Figure 5 — Figure 5: Comparison of F1 scores across different configurations.

Method	Setting	EM	F1	P	R
ODYSSEY w/ GPT-5	Table-Text Cross Reasoning	23.9	28.6	36.3	28.4
ODYSSEY w/ GPT-5	Table-only Reasoning	32.0	39.2	49.5	38.8
HProPro w/ GPT-5	Table-Text Cross Reasoning	11.9	16.3	17.2	16.7
HProPro w/ GPT-5	Table-only Reasoning	29.2	45.2	46.9	46.4

Table 6: Performance Comparison With and Without Text Data in Table-Text QA.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Any error	Error breakdown (within erroneous samples)
Dataset	% of all samples	Redundant modality	Incomplete answer set	Incorrect / unanswerable
HybridQA	21%	52.4%	23.8%	23.8%
MULTIHIERITT	26%	15.4%	30.8%	53.8%
TAT-QA	30%	96.7%	0.0%	3.3%
FinQA	17%	41.2%	0.0%	58.8%
SPARTA	0%	0.0%	0.0%	0.0%

Appendix

Table 7: Audit of 100 randomly sampled dev examples from each dataset. “Any error” shows the fraction of all samples containing at least one error. The breakdown columns report the relative distribution among erroneous samples.

Type	Inner aggreg.	Correlation	Typical intent / example
Type–N	✗	✗	Pure set membership. Outer block tests whether a value belongs to the set returned by a non-correlated subquery (e.g., WHERE x IN (SELECT ...) ).
Type–A	✓	✗	Aggregate comparison. Inner block computes an aggregate such as AVG or MAX and the result is compared with each outer tuple (e.g., salary > (SELECT AVG(salary) FROM ...) ).
Type–J	✗	✓	Correlated filtering. Inner query references attributes of the outer block without aggregation (e.g., EXISTS (SELECT 1 FROM Items i WHERE i.order_id = o.id) ).
Type–JA	✓	✓	Correlated aggregate comparison. Inner query both correlates with the outer block and aggregates its own rows before the comparison (e.g., EXISTS (SELECT 1 FROM Items i WHERE i.order_id = o.id GROUP BY ... HAVING SUM(i.qty) > o.limit) ).

Appendix

Table 8: Nested–query patterns.

Figure 6: Four primary nesting patterns—type (N, A, J, JA) queries of depth 1. Each consists of an outer block (

T_{i}

) and an inner block (

T_{j}

). Arcs labeled ‘A’ indicate aggregation in the inner SELECT ; straight arcs ‘N’ denote set-inclusion predicates; curved arcs denote join predicates.

Panel of Figure 7 — Figure 7: Schemas of the reference-fact databases used in SPARTA across three domains. Each database consists of two complementary types of tables: source tables $S_{T}$ (orange) from public datasets (e.g., NBA player salaries, movie metadata, medical records) and grounding tables $G_{T}$ (green) encoding atomic facts extracted from textual passages.

Domain	Table Name	# Columns	# Rows
nba	nba_draft_combine_stats	35	772
	nba_player_information	14	4596
	nba_player_award	3	236
	nba_champion_history	8	69
	nba_player_affiliation	4	13980
	nba_team_information	9	30

Appendix

Table 9: Row/Column Statistics of All Tables in the Sparta Benchmark

Query Shape and Size Distribution (%)
Non-nested	(Depth 1, Breadth 1)	(Depth 1, Breadth 2)	(Depth 1, Breadth 3)	(Depth 2, Breadth 1)	(Depth 2, Breadth 2)	(Depth 3, Breadth 1)	Total
45.5	9.1	9.1	9.1	9.1	9.1	9.1	100.0
SQL Operator Presence (%)
WHERE	GROUP BY	HAVING	ORDER BY	LIMIT	AGGREGATION
100.0	15.3	3.4	7.7	4.5	50.0
Nested Predicate Type Presence in Nested Query (%)

Appendix

Table 10: Benchmark Configuration: SQL Operator Coverage, Query-Shape/Size Distribution.

LLM	Method	Ideal Calls	Total Calls
gpt-oss-20B	One-Shot–k	2661	13736
	Post-Order (no provenance)	3073	5977
	Post-Order + Prov	3146	4854
Llama-3.1-70B-Instruct	One-Shot–k	2664	13962
	Post-Order (no provenance)	3104	8253
	Post-Order + Prov	3074	4722

Appendix

Table 11: Generation cost comparison across LLM sizes for three refinement strategies.

Panel of Figure 8 — Figure 8: Generation cost for varying (a) star-query size and (b) chain-query size. Sky-blue bars mark ideal LLM calls, and the labels above each bar represent the actual excess percentage.

# of Accessed Tables	Average LLM Calls per Query
1	2.3
2	4.9 (+ 2.6)
3	8.8 (+ 3.9)
4	11.0 (+ 2.2)

Appendix

Table 12: Average LLM calls per query as the number of accessed tables increases.

Nesting Type	F1 (HProPro w/ GPT-5)
Type-N	40.0 (+15.9%)
Type-A	33.7 ( $-$ 2.3%)
Type-J	30.8 ( $-$ 10.7%)
Type-JA	25.6 ( $-$ 25.8%)
Total	34.5

Appendix

Table 13: F1 scores of HProPro (w/ GPT-5) across different nesting types. Percentage values indicate relative change from the overall average (34.5).

Query Category	F1 (HProPro w/ GPT-5)
Negation ( NOT LIKE , NOT EXISTS , NOT IN , <> )	28.7 ( $-$ 28.3%)
Range ( > , < , >= , <= )	32.9 ( $-$ 18.6%)
SPARTA (Oracle)	40.4

Appendix

Table 14: F1 scores of HProPro (w/ GPT-5) on negation and range queries.

Figure 9: Illustration of representative error cases where the model fails to correctly answer. (a) Negation reasoning error. (b) Range reasoning error.

Query Set	F1 (HProPro w/ GPT-5)
Original Questions	45.22
Rephrased Questions	45.02 ( $-$ 0.44%)

Appendix

Table 15: F1 scores of HProPro (w/ GPT-5) on original and human-verified rephrased queries.

Panel of Figure 10 — Figure 10: Statistics of errors. For detailed descriptions and examples of each error category, see Appendix J .

Figure 11: Illustration of a representative error case where the model fails to correctly answer.

Row Type	Content
Domain	NBA
Question	Which player won the NBA MVP award in the 1986 season?
SQL	SELECT player_name FROM nba_player_award WHERE season = 1986 AND award = ’nba mvp’
Answer	Larry Bird
Domain	NBA
Question	What are the names of the players who scored more than 15 points and rebounded more than 5 times in a game?

Appendix

Table 16: 20 representative examples from SPARTA , each consisting of a domain, a natural language question, its corresponding SQL query, and the answer.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 66 papers

Sep 3, 2026

TabScope: Question-Adaptive Scope Selection for Table Question Answering

Large Language Models (LLMs) have shown strong performance on table question answering, yet their accuracy often degrades as table size increases. We find that this degradation is not uniform across question types. Localization-sensitive questions are particularly affected by irrelevant table content, while questions requiring broader evidence may still benefit from full-table reasoning. Based on this observation, we propose a question-adaptive framework that dynamically selects between localized and full-table reasoning. The framework constructs question-specific sub-tables through operation-aware table decomposition and uses the predicted question type to determine the appropriate reasoning mode. We further introduce silver reference sub-tables for evaluating evidence selection and construct SLQA, a benchmark based on real-world long tables. Experiments on WikiTQ and SLQA show that localization is particularly effective for lookup and local reasoning questions, while adaptive selection between localized and full-table reasoning achieves the best overall performance. These results highlight that long-table QA requires deciding not only how to localize, but also when to localize. Our code and datasets will be made available upon publication of the paper.

Yuxiang Wang, Junhao Gan, Jianzhong Qi

The University of Melbourne

Table QA

Ask PDF

Jun 20, 2026

Evolving from Lessons: Skill-Augmented Table Graph Reasoning for Operation-wise Table Question Answering

Table Question Answering (TableQA) aims to reason over tables to answer user queries. Existing research treats all questions uniformly and evaluates solely through overall accuracy, obscuring a critical reality that LLMs excel at simple lookups yet struggle with complex operations like aggregation and arithmetic. To reveal this disparity, we introduce a novel Operation-wise TableQA task with a fine-grained question taxonomy and release two datasets named WikiTQ-ow and TabFact-ow for evaluation. As for modeling bottlenecks, existing methods flatten tables into linearized texts, disrupting inherent structures and inducing the ``lost-in-the-middle'' issue, which poses a primary barrier to complex cross-row reasoning. Moreover, they typically reason from scratch, neglecting reusable patterns shared across similar operations. To address these limitations, we propose a Skill-augmented Table Graph Reasoning (SkillTGR) framework for self-evolving structured reasoning. Specifically, SkillTGR represents tables as attributed graphs with explicit row-column-cell structures, where LLMs plan and execute dynamic chains to retrieve evidence subgraphs for graph traversal reasoning. Based on this, SkillTGR builds a hierarchical SkillBank to distill reason trajectories into abstract skills under cognitive heuristics, then hybrid retrieves both successful and failed skills for contrastive augmented table graph reasoning, thereby enabling the continual self-evolution. Extensive experiments demonstrate that SkillTGR achieves superior performance with an average of 5.91% overall and 5.06% operation-wise improvement, also reducing 19.76% token consumption and 27.64% inference latency. Our codes and data will be released upon publication.

Guixin Su, Qiankun Pi, Mayi Xu +5

School of Computer Science, Wuhan University, Wuhan, China

Table QA

Ask PDF

Jul 14, 2026

LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes

While modern question answering (QA) systems excel on clean, schema-aligned corpora, real-world knowledge is rarely so neatly packaged. Answering questions over enterprise and scientific data lakes requires systems to navigate heterogeneous, weakly structured collections of tables, passages, and linked metadata. Current benchmarks abstract away this noisy discovery process, failing to evaluate end-to-end performance. To bridge this gap, we introduce LakeQuest, a human-validated benchmark of 9,846 QA pairs designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes. LakeQuest spans three diverse domains (AI/ML metadata, retail banking, and multimodal biomedical drug information) and pairs every question with exact, modality-aware evidence pointers. By isolating source discovery from cross-modal synthesis, LakeQuest exposes critical failure modes in modern QA systems. Our baseline evaluations, including standard Retrieval-Augmented Generation (RAG) and agentic tool-use methods, reveal that high-quality retrieval does not guarantee correct reasoning. Systems consistently struggle with relation chaining in metadata graphs, policy grounding in bank ledgers, and joint tabular QA in biomedical contexts, highlighting the need for robust discovery and faithful cross-file composition mechanisms in future agentic QA systems.

Michael Solodko, Steven Gong, Guangwei Yu +3

University of Waterloo · Layer 6 AI

Multimodal QA Agentic Retrieval RAG Evaluation

Ask PDF

Abstract

Table QA

Explore similar work

TabScope: Question-Adaptive Scope Selection for Table Question Answering

Evolving from Lessons: Skill-Augmented Table Graph Reasoning for Operation-wise Table Question Answering

LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes