SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

Sungho Park; Jueun Kim; Wook-Shin Han

cs.CLFeb 26, 2026

SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

Authors: Sungho Park, Jueun Kim, Wook-Shin Han

Organizations: POSTECH, Pohang, Republic of Korea

Abstract

Real-world Table-Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated - and therefore error-prone - and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table-Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question-answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. Our benchmark, construction code, and baseline models are available at https://github.com/pshlego/SPARTA/tree/main.

Figures & tables

Figure 1: Representative examples of our SPARTA benchmark (see Appendix M for more examples).

Benchmark	Table Size		Question Generation	Grouping/Having	Query Shape Supported		Multi-hop Reasoning		Annotation Error Rate (over 100 sampled queries)
Benchmark	#Col	#Row	Question Generation	Grouping/Having	Chain ( $\!>\!3$ -Hop)	Star	Cross-modal	Uni-modal	Annotation Error Rate (over 100 sampled queries)
TAT-QA Zhu et al. (2021)	4.0	9.4	Manual	✗	✗	✗	✓	✗	30%
FinQA Chen et al. (2021)	–	6.4	Manual	✗	✗	✗	✓	✗	27%
MULTIHIERITT Zhao et al. (2022)	5.0	10.8	Manual	✗	✗	✗	✓	✓	26%
HybridQA Chen et al. (2020b)	4.4	15.7	Manual	✗	✗	✗	✓	✗	21%
OTT-QA Chen et al. (2020a)	4.4	15.7	Manual	✗	✗	✗	✓	✗	21%

Table 1: Comparison of Table–Text QA benchmarks (see Appendix A for detailed annotation audit results).

Figure 2: Overview of SPARTA : (1) Reference Fact Database Construction, (2) Query Generation, (3) Question Verbalisation. ST and GT denote a source table and a grounding table, respectively.

Figure 3: Overview of provenance-based refinement.

Metric	Definition
Success-Q	# of non-nested queries that execute without error and return at least one row.
Exec-Err	# of statements that fail at parse or runtime, revealing schema or logic errors.
Empty-Q	# of syntactically valid queries that return zero rows because predicates are too restrictive.
Duplicate-Q	# of queries whose result duplicates a previously generated query, reducing diversity.
Ideal Calls	# of LLM invocations required if every step succeeds on the first attempt (baseline cost).
Total Calls	# of actual LLM invocations, i.e., Ideal Calls plus extra calls for provenance-guided fixes or other retries.

Table 2: Cost metrics used for benchmark generation.

Method	Success-Q	Empty-Q	Duplicate-Q	Exec-Err	Ideal Calls	Total Calls	Wall Time (s)
Non-nested Query Generation
One-Shot	500	60	1265	5	500	1830	4256.96
Clause	500	51	78	0	1053	1316	3218.83
Execution-Guided	500	0	27	0	1058	1134	2466.47
Nested Query Generation
One-Shot– $k$	600	0	0	0	2664	13962	115316.67

Table 3: Generation Cost Comparison of Query Generation Methods.

Figure 4: Comparison of Query Naturalness for Different Generation Methods.

Method	SPARTA																HybridQA
	NBA				Movie				Medical				Avg.
	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R
ODYSSEY w/ GPT-3.5-turbo	9.0	15.1	26.8	14.8	20.2	23.9	33.6	24.7	6.7	22.9	33.2	21.3	12.0	20.6	31.2	20.3	32.7	42.2	42.6	44.2
HProPro w/ GPT-3.5-turbo	11.0	13.6	16.4	13.8	22.2	27.8	29.1	29.2	15.5	19.5	20.2	19.7	16.2	20.3	21.9	20.9	21.4	25.3	25.7	26.1
ODYSSEY w/ GPT-5	21.2	28.4	38.4	28.1	20.4	24.2	32.9	24.3	47.5	54.2	60.3	54.2	29.7	35.6	43.9	35.5	55.3	69.5	69.3	73.5
HProPro w/ GPT-5	23.6	33.1	36.2	34.0	36.6	47.1	49.2	48.8	28.1	41.0	43.2	41.6	29.5	40.4	42.9	41.5	59.7	70.5	71.1	73.1

Table 4: Table-Text QA Accuracy on the SPARTA (Oracle) across multiple domains.

Method	SPARTA																OTT-QA
	NBA				Movie				Medical				Avg.
	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R	EM	F1	P	R
HELIOS+FiE Reader	4.6	6.9	17.6	6.4	8.6	11.9	23.0	11.6	6.6	16.0	33.0	12.9	6.6	11.6	24.5	10.3	58.6	65.2	66.7	65.2
HELIOS+HProPro w/ GPT-5	14.5	19.0	24.2	18.6	17.4	21.6	28.6	21.7	13.7	27.3	31.3	27.1	15.2	22.6	28.0	22.5	47.7	56.0	57.4	56.5

Table 5: Table-Text QA Accuracy on the SPARTA (Retrieval) across multiple domains.

Panel of Figure 5 — Figure 5: Comparison of F1 scores across different configurations.

Method	Setting	EM	F1	P	R
ODYSSEY w/ GPT-5	Table-Text Cross Reasoning	23.9	28.6	36.3	28.4
ODYSSEY w/ GPT-5	Table-only Reasoning	32.0	39.2	49.5	38.8
HProPro w/ GPT-5	Table-Text Cross Reasoning	11.9	16.3	17.2	16.7
HProPro w/ GPT-5	Table-only Reasoning	29.2	45.2	46.9	46.4

Table 6: Performance Comparison With and Without Text Data in Table-Text QA.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Any error	Error breakdown (within erroneous samples)
Dataset	% of all samples	Redundant modality	Incomplete answer set	Incorrect / unanswerable
HybridQA	21%	52.4%	23.8%	23.8%
MULTIHIERITT	26%	15.4%	30.8%	53.8%
TAT-QA	30%	96.7%	0.0%	3.3%
FinQA	17%	41.2%	0.0%	58.8%
SPARTA	0%	0.0%	0.0%	0.0%

Appendix

Table 7: Audit of 100 randomly sampled dev examples from each dataset. “Any error” shows the fraction of all samples containing at least one error. The breakdown columns report the relative distribution among erroneous samples.

Type	Inner aggreg.	Correlation	Typical intent / example
Type–N	✗	✗	Pure set membership. Outer block tests whether a value belongs to the set returned by a non-correlated subquery (e.g., WHERE x IN (SELECT ...) ).
Type–A	✓	✗	Aggregate comparison. Inner block computes an aggregate such as AVG or MAX and the result is compared with each outer tuple (e.g., salary > (SELECT AVG(salary) FROM ...) ).
Type–J	✗	✓	Correlated filtering. Inner query references attributes of the outer block without aggregation (e.g., EXISTS (SELECT 1 FROM Items i WHERE i.order_id = o.id) ).
Type–JA	✓	✓	Correlated aggregate comparison. Inner query both correlates with the outer block and aggregates its own rows before the comparison (e.g., EXISTS (SELECT 1 FROM Items i WHERE i.order_id = o.id GROUP BY ... HAVING SUM(i.qty) > o.limit) ).

Appendix

Table 8: Nested–query patterns.

Figure 6: Four primary nesting patterns—type (N, A, J, JA) queries of depth 1. Each consists of an outer block (

T_{i}

) and an inner block (

T_{j}

). Arcs labeled ‘A’ indicate aggregation in the inner SELECT ; straight arcs ‘N’ denote set-inclusion predicates; curved arcs denote join predicates.

Panel of Figure 7 — Figure 7: Schemas of the reference-fact databases used in SPARTA across three domains. Each database consists of two complementary types of tables: source tables $S_{T}$ (orange) from public datasets (e.g., NBA player salaries, movie metadata, medical records) and grounding tables $G_{T}$ (green) encoding atomic facts extracted from textual passages.

Domain	Table Name	# Columns	# Rows
nba	nba_draft_combine_stats	35	772
	nba_player_information	14	4596
	nba_player_award	3	236
	nba_champion_history	8	69
	nba_player_affiliation	4	13980
	nba_team_information	9	30

Appendix

Table 9: Row/Column Statistics of All Tables in the Sparta Benchmark

Query Shape and Size Distribution (%)
Non-nested	(Depth 1, Breadth 1)	(Depth 1, Breadth 2)	(Depth 1, Breadth 3)	(Depth 2, Breadth 1)	(Depth 2, Breadth 2)	(Depth 3, Breadth 1)	Total
45.5	9.1	9.1	9.1	9.1	9.1	9.1	100.0
SQL Operator Presence (%)
WHERE	GROUP BY	HAVING	ORDER BY	LIMIT	AGGREGATION
100.0	15.3	3.4	7.7	4.5	50.0
Nested Predicate Type Presence in Nested Query (%)

Appendix

Table 10: Benchmark Configuration: SQL Operator Coverage, Query-Shape/Size Distribution.

LLM	Method	Ideal Calls	Total Calls
gpt-oss-20B	One-Shot–k	2661	13736
	Post-Order (no provenance)	3073	5977
	Post-Order + Prov	3146	4854
Llama-3.1-70B-Instruct	One-Shot–k	2664	13962
	Post-Order (no provenance)	3104	8253
	Post-Order + Prov	3074	4722

Appendix

Table 11: Generation cost comparison across LLM sizes for three refinement strategies.

Panel of Figure 8 — Figure 8: Generation cost for varying (a) star-query size and (b) chain-query size. Sky-blue bars mark ideal LLM calls, and the labels above each bar represent the actual excess percentage.

# of Accessed Tables	Average LLM Calls per Query
1	2.3
2	4.9 (+ 2.6)
3	8.8 (+ 3.9)
4	11.0 (+ 2.2)

Appendix

Table 12: Average LLM calls per query as the number of accessed tables increases.

Nesting Type	F1 (HProPro w/ GPT-5)
Type-N	40.0 (+15.9%)
Type-A	33.7 ( $-$ 2.3%)
Type-J	30.8 ( $-$ 10.7%)
Type-JA	25.6 ( $-$ 25.8%)
Total	34.5

Appendix

Table 13: F1 scores of HProPro (w/ GPT-5) across different nesting types. Percentage values indicate relative change from the overall average (34.5).

Query Category	F1 (HProPro w/ GPT-5)
Negation ( NOT LIKE , NOT EXISTS , NOT IN , <> )	28.7 ( $-$ 28.3%)
Range ( > , < , >= , <= )	32.9 ( $-$ 18.6%)
SPARTA (Oracle)	40.4

Appendix

Table 14: F1 scores of HProPro (w/ GPT-5) on negation and range queries.

Figure 9: Illustration of representative error cases where the model fails to correctly answer. (a) Negation reasoning error. (b) Range reasoning error.

Query Set	F1 (HProPro w/ GPT-5)
Original Questions	45.22
Rephrased Questions	45.02 ( $-$ 0.44%)

Appendix

Table 15: F1 scores of HProPro (w/ GPT-5) on original and human-verified rephrased queries.

Panel of Figure 10 — Figure 10: Statistics of errors. For detailed descriptions and examples of each error category, see Appendix J .

Figure 11: Illustration of a representative error case where the model fails to correctly answer.

Row Type	Content
Domain	NBA
Question	Which player won the NBA MVP award in the 1986 season?
SQL	SELECT player_name FROM nba_player_award WHERE season = 1986 AND award = ’nba mvp’
Answer	Larry Bird
Domain	NBA
Question	What are the names of the players who scored more than 15 points and rebounded more than 5 times in a game?

Appendix

Table 16: 20 representative examples from SPARTA , each consisting of a domain, a natural language question, its corresponding SQL query, and the answer.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 66 papers

TabScope: Question-Adaptive Scope Selection for Table Question Answering
Sep 3, 2026Yuxiang Wang, Junhao Gan, Jianzhong QiTable QA
The University of Melbourne
Evolving from Lessons: Skill-Augmented Table Graph Reasoning for Operation-wise Table Question Answering
Jun 20, 2026Guixin Su, Qiankun Pi, Mayi Xu +5Table QA
School of Computer Science, Wuhan University, Wuhan, China
LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes
Jul 14, 2026Michael Solodko, Steven Gong, Guangwei Yu +3Multimodal QA Agentic Retrieval
University of Waterloo · Layer 6 AI

Sep 3, 2026J/K move · Enter open · S save

TabScope: Question-Adaptive Scope Selection for Table Question Answering

Yuxiang Wang, Junhao Gan, Jianzhong Qi

The University of Melbourne

Table QA

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown strong performance on table question answering, yet their accuracy often degrades as table size increases. We find that this degradation is not uniform across question types. Localization-sensitive questions are particularly affected by irrelevant table content, while questions requiring broader evidence may still benefit from full-table reasoning. Based on this observation, we propose a question-adaptive framework that dynamically selects between localized and full-table reasoning. The framework constructs question-specific sub-tables through operation-aware table decomposition and uses the predicted question type to determine the appropriate reasoning mode. We further introduce silver reference sub-tables for evaluating evidence selection and construct SLQA, a benchmark based on real-world long tables. Experiments on WikiTQ and SLQA show that localization is particularly effective for lookup and local reasoning questions, while adaptive selection between localized and full-table reasoning achieves the best overall performance. These results highlight that long-table QA requires deciding not only how to localize, but also when to localize. Our code and datasets will be made available upon publication of the paper.

Abstract

Table QA

Explore similar work

TabScope: Question-Adaptive Scope Selection for Table Question Answering

Evolving from Lessons: Skill-Augmented Table Graph Reasoning for Operation-wise Table Question Answering

LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes