SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computation

Hexuan Wang; Yaxuan Ren; Srikar Bommireddypalli; Shuxian Chen; Adarsh Prabhudesai; Rongkun Zhou; Elina Baral; Philipp Koehn

cs.CLMar 9, 2026

SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computation

Authors: Hexuan Wang, Yaxuan Ren, Srikar Bommireddypalli, Shuxian Chen, Adarsh Prabhudesai, Rongkun Zhou, Elina Baral, Philipp Koehn

Organizations: Center for Language and Speech Processing, Johns Hopkins University

Abstract

We introduce SciTaRC, an expert-authored benchmark for question answering over scientific tables that targets composite, multi-step reasoning. To enable fine-grained diagnostic analysis beyond end-task accuracy, SciTaRC pairs each question with a manually constructed reasoning plan and explicit complexity metrics. State-of-the-art models fail on at least 23% of these questions, while highly capable open-weight models like Llama-3.3-70B collapse on 65.5% of the benchmark. Error analysis shows that, in zero-shot settings, failures are driven primarily by question comprehension, where models misinterpret the scientific query and derive the wrong reasoning objective. To determine whether overcoming this gap is sufficient, we use the structured plans to decouple strategy formulation from execution. Surprisingly, providing oracle step-by-step plans yields only limited gains and fails to eliminate the performance gap. This reveals a substantial execution bottleneck: both natural language and code-based methods struggle to reliably carry out long-horizon computational chains over structured data. Ultimately, SciTaRC serves as a rigorous diagnostic testbed for studying both planning and execution in scientific table reasoning.

Figures & tables

Figure 1: A representative SciTaRC task. Instead of simple fact retrieval, the models must execute a multi-step procedure that combines targeted lookup, computation, aggregation, and conditional comparison to derive the final answer.

	Input Complexity			Reasoning Complexity
Metric	$S_{tab}$	$S_{cell}$	$S_{tok}$	$L_{plan}$	$I_{retr}$	$I_{calc}$	$C_{flow}$
Mean	1.3	287.0	2,527	9.8	3.4	4.0	3.0
Max	3	3,338	22,671	27	18	18	12

Table 1: SciTaRC Dataset Statistics. These metrics illustrate the composite reasoning focus over large scientific tables. High average values for plan horizon (

L_{plan}

) and table size (

S_{cell}

) highlight the requirement for high-fidelity execution across long-horizon scientific contexts.

Figure 2: Model Agreement Matrix. Rows represent questions sorted by difficulty, defined as the total number of models that answered correctly (easiest at top). Columns represent models sorted by accuracy. Blue indicates a correct prediction.

Proprietary Models
Model Family	Model Name	Method	Size	LLM-Judge (%)	EM (%)
OpenAI	GPT-5	CoT	-	76.8	22.1
xAI	Grok-4.1-fast-reasoning	CoT	-	76.8	2.7
xAI	Grok-4	CoT	-	74.4	34.0
Moonshot	Kimi-K2-Thinking	CoT	-	73.3	24.8
Anthropic	Claude-Opus-4.5	CoT	-	72.0	20.2

Table 2: SciTaRC Main Leaderboard. Models are grouped by access level and sorted by primary accuracy. All evaluations are zero-shot, using CoT by default and PoT for code-optimized models. Our primary metric, LLM-Judge (%) , evaluates semantic and logical correctness; EM (%) measures strict string equivalence as a traditional TableQA baseline. Best results per category are bolded .

Figure 3: Impact of Reasoning Complexity. We measure accuracy against four metrics: Calculation Intensity (

I_{calc}

), Retrieval Demand (

I_{retr}

), Plan Horizon (

L_{plan}

), and Control Flow (

C_{flow}

). Across all dimensions, increased complexity correlates with lower performance.

Category	Ratio
Comprehension	73%
Calculation	17%
Localization	7%
Memory	3%

Table 3: Error distribution.

Model	Accuracy (%)			Diagnostics
Model	Direct	Auto	Oracle	$\Delta_{\text{Auto}}$	$\Delta_{\text{Max}}$	$\Delta_{\text{Plan}}$
Qwen2.5-Coder-32B	21.3	23.2	27.5	+1.9	+6.2	4.3
DS-R1-Dist-32B	41.5	36.1	45.8	-5.4	+4.3	9.7
Qwen3-30B	44.7	34.8	45.3	-10.0	+0.6	10.5
Qwen3-80B	49.3	41.8	49.1	-7.5	-0.2	7.3

Table 4: Impact of Planning and Upper Bounds. Diagnostics :

\Delta_{\text{Auto}}

denotes the gain from autonomous planning (Auto

-

Direct).

\Delta_{\text{Max}}

denotes the absolute potential gain when the comprehension bottleneck is eliminated (Oracle

-

Direct).

\Delta_{\text{Plan}}

represents the performance gap caused by flawed strategy formulation (Oracle

-

Auto). Even with perfect Oracle guidance, models hit a hard ceiling at

\sim

49% accuracy.

Figure 4: Decomposition of Failure Modes. We classify failure modes by testing if the Oracle plan rescues autonomous (

S_{auto}

) failures. Plan Failure (Yellow) : Failed autonomously but succeeded with an Oracle plan (flawed strategy). Execution Failure (Red) : Failed under both autonomous and Oracle conditions (unrecoverable by planning). The dominant red bars confirm that execution capacity, not strategy, is the primary bottleneck.

Figure 5: Performance Gain Curves. We plot success probability across the difficulty spectrum (see Appendix F for question-level changes). Code Models (Left) show consistent gains from planning on hard tasks. Generalist Models (Right) often regress on easy tasks (Red Shading) and gain on original unsolved tasks (Green Shading)

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

System Prompt
You are evaluating a table QA system. Compare the predicted answer against the ground truth answer. SCORING RULES (follow exactly):
- 1.0: CORRECT - Prediction contains the same core factual information as ground truth
- 0.5: PARTIALLY CORRECT - Prediction has some correct information but missing key parts
- 0.0: INCORRECT - Prediction is factually wrong or completely unrelated CRITICAL REQUIREMENTS:
- ALL components in ground truth MUST be mentioned in prediction
- If ground truth has multiple parts, prediction must include ALL parts

Appendix

Table 5: The LLM-as-a-Judge prompt used for automatic evaluation.

Q: How much lower is the F1-score of the second-best model compared to GPT-4? A: 0.1066

SELECT F1-scores for all models

COMPUTE sort models by F1-score in descending order

COMPUTE difference = F1-score of GPT-4

-

F1-score of second-best model

RETURN difference

L_{plan}=4

I_{retr}=1

I_{calc}=2

C_{flow}=0

S_{tab}=1

S_{cell}=130

Q: Which zero-shot models outperform any of the fine-tuned models on any metric and language direction? A: InternVL2.5-38B, InternVL2.5-78B, and Qwen2.5-VL-72B (on chrF++ EN-ZH)

Appendix

Table 6: Complete SciTaRC instances across the complexity range (1 of 2). Each block shows the question, gold answer, expert-authored structured reasoning plan, and the plan-derived complexity metrics defined in Section 3.4 . Instances are ordered by plan horizon (

L_{plan}

Q: On which test sets are bigger models (either model size or longer token windows) always at least as good as smaller models of the same kind? A: HellaSwag, PiQA

SELECT all test sets

SELECT all model families

SELECT all token-window groups

LOOP for each test set

COMPUTE violation_exists = false

LOOP for each model family

Appendix

Table 7: Complete SciTaRC instances across the complexity range (2 of 2). Continued from Table 6 .

Standard Chain-of-Thought (CoT)
You are a helpful science assistant who answers questions about information in tables. Here is the relevant tabular data:
[Table Data] You may think through the question step by step. Your final response should be "Answer:" followed by the answer. Question: [Question]
Standard Program-of-Thought (PoT)
Write Python code to analyze the table and answer the question. Table:
[Table Data] Question: [Question] Write Python code that prints the final answer.

Appendix

Table 8: Baseline prompts for Language and Code execution without explicit planning.

Plan Generation Prompt
You are a helpful assistant that creates step-by-step plans for answering questions about tables. Operations:
- SELECT: Extract specific values or subsets from the table
- LOOP: Iterate over items (e.g., LOOP for each model)
- COMPUTE: Numerical operations (average, argmax, difference, etc.)
- IF: Conditional logic
- RETURN: State what to return for the final answer Rules:

Appendix

Table 9: The prompt used to elicit structured plans.

Plan-Guided Chain-of-Thought
You are a helpful science assistant who answers questions about information in tables. Here is the relevant tabular data:
[Table Data] Here is a step-by-step plan to follow:
[Plan] How to read the plan:
- SELECT: Look up and extract the specified values from the table
- LOOP for each X: Repeat the indented steps for every X
- COMPUTE: Perform the calculation described

Appendix

Table 10: Prompts for executing a provided reasoning plan (Self or Gold).

Figure 6: Extended Complexity Analysis. Performance breakdown across three dimensions of input scale: (Left) Structural Size (

S_{cell}

), (Middle) Context Load (

S_{tok}

), and (Right) Input Scope (

S_{tab}

). While increasing structural size and context load generally degrades performance for weaker models, multi-table reasoning shows divergent behaviors: top models often improve on tasks requiring cross-table linking. Sample sizes (

N

) for each bin are denoted on the x-axis.

Figure 7: Autonomous Gain (Direct

\to

Autonomous). Consistent with the Gain Curve, planning exhibits a dual effect. For the Code model (left), we observe net positive gains (

+7

) concentrated on harder questions. For Language models (right), we observe net regression (e.g.,

-37

for Qwen3-30B), primarily driven by the "Cost of Compliance" on easier questions where the overhead of planning disrupts simple retrieval tasks.

Figure 8: Plan Correction (Autonomous

\to

Oracle). This plot isolates the quality of the generated plan. While we observe net positive gains (

+16

+39

) when moving to oracle plans, the non-trivial volume of Regression (Red) cases indicates that external strategies can sometimes disrupt the model’s intrinsic reasoning flow. The dominance of shared failures (gray pixels) between autonomous and oracle planning confirms that the primary bottleneck is execution fidelity, as even perfect strategies often fail to overcome implementation errors.

Figure 9: Total Potential (Direct

\to

Oracle). This compares the baseline directly against the upper bound. Even with perfect Oracle guidance, the net gains are moderate (

-1

+23

). The vast majority of difficult questions remain Failed (Gray) , demonstrating that a significant portion of the dataset hits a hard capability ceiling that cannot be solved by planning assistance alone.

Model	Accuracy (%)			FS-Oracle Errors
Model	ZS-Direct	FS-Direct	FS-Oracle	Total	Regressions
Qwen3-30B	44.4	47.1	47.7	193	52
Gemma-3-27B	30.1	28.7	32.5	249	57
Llama-3.3-70B	34.4	29.0	29.3	261	71
Qwen3-Coder-30B (PoT)	19.5	25.7	35.2	239	18

Appendix

Table 11: Two-shot ablation (

N=369

). ZS-Direct : zero-shot, no guidance. FS-Direct : two demonstrations. FS-Oracle : two demonstrations plus the ground-truth structured plan. Regressions counts questions answered correctly under ZS-Direct but failed under FS-Oracle.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 66 papers

From Table to Cell: Attention for Better Reasoning with TABALIGN
May 14, 2026Tung Sum Thomas Kwok, Zeyong Zhang, Xinyu Wang +6Table QA Table Reasoning
University of California, Los Angeles · SimpleWay · New Jersey Institute of Technology +4
RSAT: Structured Attribution Makes Small Language Models Faithful Table Reasoners
Apr 30, 2026Jugal Gajjar, Kamalasankari SubramaniakuppusamyLLM Reasoning LLM Grounding
Department of Computer Science, The George Washington University, USA
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Apr 14, 2026Oliver Bentham, Vivek SrikumarLLM Evaluation Scientific Agent Benchmarks
Kahlert School of Computing University of Utah

May 14, 2026J/K move · Enter open · S save

From Table to Cell: Attention for Better Reasoning with TABALIGN

Tung Sum Thomas Kwok, Zeyong Zhang, Xinyu Wang, Chunhe Wang+5

University of California, Los Angeles · SimpleWay · New Jersey Institute of Technology+4

Table QA Table Reasoning Structured Reasoning Diffusion Language Models LLM Reasoning

Open paper AskPDF arXiv

Multi-step LLM reasoning over structured tables fails because planning and execution share no explicit cell-grounding contract. Existing methods constrain the planner to a left-to-right factorization at odds with table permutation invariance, and score intermediate states by generated content alone, overlooking cell grounding. We conduct a pilot study showing that diffusion language models (DLMs) produce more human-aligned and permutation-stable cell attention on tables than autoregressive models, with a 40.2% median reduction in attention-AUROC variability under row reordering. Motivated by this, we propose TABALIGN, a planned table reasoning framework that operationalizes the contract. TABALIGN pairs a masked DLM planner, whose bidirectional denoising emits plan steps as binary cell masks, with TABATTN, a lightweight verifier trained on 1,600 human-verified attention standards to score each step by its attention overlap with the plan-designated mask. Across eight benchmarks covering table question answering and fact verification, TABALIGN improves average accuracy by 15.76 percentage points over the strongest open-source baseline at comparable 8B-class scale, with a matched-backbone ablation attributing 2.87 percentage points of this gain to the DLM planner over an AR planner on a fixed reasoner. Cleaner DLM plans also accelerate downstream reasoning execution by 44.64%.

Abstract

Table QA

Explore similar work

From Table to Cell: Attention for Better Reasoning with TABALIGN

RSAT: Structured Attribution Makes Small Language Models Faithful Table Reasoners

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis