We introduce SciTaRC, an expert-authored benchmark for question answering over scientific tables that targets composite, multi-step reasoning. To enable fine-grained diagnostic analysis beyond end-task accuracy, SciTaRC pairs each question with a manually constructed reasoning plan and explicit complexity metrics. State-of-the-art models fail on at least 23% of these questions, while highly capable open-weight models like Llama-3.3-70B collapse on 65.5% of the benchmark. Error analysis shows that, in zero-shot settings, failures are driven primarily by question comprehension, where models misinterpret the scientific query and derive the wrong reasoning objective. To determine whether overcoming this gap is sufficient, we use the structured plans to decouple strategy formulation from execution. Surprisingly, providing oracle step-by-step plans yields only limited gains and fails to eliminate the performance gap. This reveals a substantial execution bottleneck: both natural language and code-based methods struggle to reliably carry out long-horizon computational chains over structured data. Ultimately, SciTaRC serves as a rigorous diagnostic testbed for studying both planning and execution in scientific table reasoning.
Figures & tables
Figure 1: A representative SciTaRC task. Instead of simple fact retrieval, the models must execute a multi-step procedure that combines targeted lookup, computation, aggregation, and conditional comparison to derive the final answer.
Input Complexity
Reasoning Complexity
Metric
Stab
Scell
Stok
Lplan
Iretr
Icalc
Cflow
Mean
1.3
287.0
2,527
9.8
3.4
4.0
3.0
Max
3
3,338
22,671
27
18
18
12
Table 1: SciTaRC Dataset Statistics. These metrics illustrate the composite reasoning focus over large scientific tables. High average values for plan horizon ( Lplan ) and table size ( Scell ) highlight the requirement for high-fidelity execution across long-horizon scientific contexts.
Figure 2: Model Agreement Matrix. Rows represent questions sorted by difficulty, defined as the total number of models that answered correctly (easiest at top). Columns represent models sorted by accuracy. Blue indicates a correct prediction.
Model Family
Model Name
Method
Size
LLM-Judge (%)
EM (%)
Proprietary Models
OpenAI
GPT-5
CoT
-
76.8
22.1
xAI
Grok-4.1-fast-reasoning
CoT
-
76.8
2.7
xAI
Grok-4
CoT
-
74.4
34.0
Moonshot
Kimi-K2-Thinking
CoT
-
73.3
24.8
Anthropic
Claude-Opus-4.5
CoT
-
72.0
20.2
Table 2: SciTaRC Main Leaderboard. Models are grouped by access level and sorted by primary accuracy. All evaluations are zero-shot, using CoT by default and PoT for code-optimized models. Our primary metric, LLM-Judge (%) , evaluates semantic and logical correctness; EM (%) measures strict string equivalence as a traditional TableQA baseline. Best results per category are bolded .
Figure 3: Impact of Reasoning Complexity. We measure accuracy against four metrics: Calculation Intensity ( Icalc ), Retrieval Demand ( Iretr ), Plan Horizon ( Lplan ), and Control Flow ( Cflow ). Across all dimensions, increased complexity correlates with lower performance.
Category
Ratio
Comprehension
73%
Calculation
17%
Localization
7%
Memory
3%
Table 3: Error distribution.
Model
Accuracy (%)
Diagnostics
Direct
Auto
Oracle
ΔAuto
ΔMax
ΔPlan
Qwen2.5-Coder-32B
21.3
23.2
27.5
+1.9
+6.2
4.3
DS-R1-Dist-32B
41.5
36.1
45.8
-5.4
+4.3
9.7
Qwen3-30B
44.7
34.8
45.3
-10.0
+0.6
10.5
Qwen3-80B
49.3
41.8
49.1
-7.5
-0.2
7.3
Table 4: Impact of Planning and Upper Bounds. Diagnostics : ΔAuto denotes the gain from autonomous planning (Auto − Direct). ΔMax denotes the absolute potential gain when the comprehension bottleneck is eliminated (Oracle − Direct). ΔPlan represents the performance gap caused by flawed strategy formulation (Oracle − Auto). Even with perfect Oracle guidance, models hit a hard ceiling at ∼ 49% accuracy.
Figure 4: Decomposition of Failure Modes. We classify failure modes by testing if the Oracle plan rescues autonomous ( Sauto ) failures. Plan Failure (Yellow) : Failed autonomously but succeeded with an Oracle plan (flawed strategy). Execution Failure (Red) : Failed under both autonomous and Oracle conditions (unrecoverable by planning). The dominant red bars confirm that execution capacity, not strategy, is the primary bottleneck.
Figure 5: Performance Gain Curves. We plot success probability across the difficulty spectrum (see Appendix F for question-level changes). Code Models (Left) show consistent gains from planning on hard tasks. Generalist Models (Right) often regress on easy tasks (Red Shading) and gain on original unsolved tasks (Green Shading)
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
System Prompt
You are evaluating a table QA system. Compare the predicted answer against the ground truth answer. SCORING RULES (follow exactly):
- 1.0: CORRECT - Prediction contains the same core factual information as ground truth
- 0.5: PARTIALLY CORRECT - Prediction has some correct information but missing key parts
- 0.0: INCORRECT - Prediction is factually wrong or completely unrelated CRITICAL REQUIREMENTS:
- ALL components in ground truth MUST be mentioned in prediction
- If ground truth has multiple parts, prediction must include ALL parts
Appendix
Table 5: The LLM-as-a-Judge prompt used for automatic evaluation.
Q: How much lower is the F1-score of the second-best model compared to GPT-4? A: 0.1066
SELECT F1-scores for all models
COMPUTE sort models by F1-score in descending order
COMPUTE difference = F1-score of GPT-4 − F1-score of second-best model
Q: Which zero-shot models outperform any of the fine-tuned models on any metric and language direction? A: InternVL2.5-38B, InternVL2.5-78B, and Qwen2.5-VL-72B (on chrF++ EN-ZH)
Appendix
Table 6: Complete SciTaRC instances across the complexity range (1 of 2). Each block shows the question, gold answer, expert-authored structured reasoning plan, and the plan-derived complexity metrics defined in Section 3.4 . Instances are ordered by plan horizon ( Lplan ).
Q: On which test sets are bigger models (either model size or longer token windows) always at least as good as smaller models of the same kind? A: HellaSwag, PiQA
SELECT all test sets
SELECT all model families
SELECT all token-window groups
LOOP for each test set
COMPUTE violation_exists = false
LOOP for each model family
Appendix
Table 7: Complete SciTaRC instances across the complexity range (2 of 2). Continued from Table 6 .
Standard Chain-of-Thought (CoT)
You are a helpful science assistant who answers questions about information in tables. Here is the relevant tabular data:
[Table Data] You may think through the question step by step. Your final response should be "Answer:" followed by the answer. Question: [Question]
Standard Program-of-Thought (PoT)
Write Python code to analyze the table and answer the question. Table:
[Table Data] Question: [Question] Write Python code that prints the final answer.
Appendix
Table 8: Baseline prompts for Language and Code execution without explicit planning.
Plan Generation Prompt
You are a helpful assistant that creates step-by-step plans for answering questions about tables. Operations:
- SELECT: Extract specific values or subsets from the table
- LOOP: Iterate over items (e.g., LOOP for each model)
- RETURN: State what to return for the final answer Rules:
Appendix
Table 9: The prompt used to elicit structured plans.
Plan-Guided Chain-of-Thought
You are a helpful science assistant who answers questions about information in tables. Here is the relevant tabular data:
[Table Data] Here is a step-by-step plan to follow:
[Plan] How to read the plan:
- SELECT: Look up and extract the specified values from the table
- LOOP for each X: Repeat the indented steps for every X
- COMPUTE: Perform the calculation described
Appendix
Table 10: Prompts for executing a provided reasoning plan (Self or Gold).
Figure 6: Extended Complexity Analysis. Performance breakdown across three dimensions of input scale: (Left) Structural Size ( Scell ), (Middle) Context Load ( Stok ), and (Right) Input Scope ( Stab ). While increasing structural size and context load generally degrades performance for weaker models, multi-table reasoning shows divergent behaviors: top models often improve on tasks requiring cross-table linking. Sample sizes ( N ) for each bin are denoted on the x-axis.
Figure 7: Autonomous Gain (Direct → Autonomous). Consistent with the Gain Curve, planning exhibits a dual effect. For the Code model (left), we observe net positive gains ( +7 ) concentrated on harder questions. For Language models (right), we observe net regression (e.g., −37 for Qwen3-30B), primarily driven by the "Cost of Compliance" on easier questions where the overhead of planning disrupts simple retrieval tasks.
Figure 8: Plan Correction (Autonomous → Oracle). This plot isolates the quality of the generated plan. While we observe net positive gains ( +16 to +39 ) when moving to oracle plans, the non-trivial volume of Regression (Red) cases indicates that external strategies can sometimes disrupt the model’s intrinsic reasoning flow. The dominance of shared failures (gray pixels) between autonomous and oracle planning confirms that the primary bottleneck is execution fidelity, as even perfect strategies often fail to overcome implementation errors.
Figure 9: Total Potential (Direct → Oracle). This compares the baseline directly against the upper bound. Even with perfect Oracle guidance, the net gains are moderate ( −1 to +23 ). The vast majority of difficult questions remain Failed (Gray) , demonstrating that a significant portion of the dataset hits a hard capability ceiling that cannot be solved by planning assistance alone.
Model
Accuracy (%)
FS-Oracle Errors
ZS-Direct
FS-Direct
FS-Oracle
Total
Regressions
Qwen3-30B
44.4
47.1
47.7
193
52
Gemma-3-27B
30.1
28.7
32.5
249
57
Llama-3.3-70B
34.4
29.0
29.3
261
71
Qwen3-Coder-30B (PoT)
19.5
25.7
35.2
239
18
Appendix
Table 11: Two-shot ablation ( N=369 ). ZS-Direct : zero-shot, no guidance. FS-Direct : two demonstrations. FS-Oracle : two demonstrations plus the ground-truth structured plan. Regressions counts questions answered correctly under ZS-Direct but failed under FS-Oracle.
Multi-step LLM reasoning over structured tables fails because planning and execution share no explicit cell-grounding contract. Existing methods constrain the planner to a left-to-right factorization at odds with table permutation invariance, and score intermediate states by generated content alone, overlooking cell grounding. We conduct a pilot study showing that diffusion language models (DLMs) produce more human-aligned and permutation-stable cell attention on tables than autoregressive models, with a 40.2% median reduction in attention-AUROC variability under row reordering. Motivated by this, we propose TABALIGN, a planned table reasoning framework that operationalizes the contract. TABALIGN pairs a masked DLM planner, whose bidirectional denoising emits plan steps as binary cell masks, with TABATTN, a lightweight verifier trained on 1,600 human-verified attention standards to score each step by its attention overlap with the plan-designated mask. Across eight benchmarks covering table question answering and fact verification, TABALIGN improves average accuracy by 15.76 percentage points over the strongest open-source baseline at comparable 8B-class scale, with a matched-backbone ablation attributing 2.87 percentage points of this gain to the DLM planner over an AR planner on a fixed reasoner. Cleaner DLM plans also accelerate downstream reasoning execution by 44.64%.
Tung Sum Thomas Kwok, Zeyong Zhang, Xinyu Wang +6
University of California, Los Angeles · SimpleWay · New Jersey Institute of Technology +4
When a language model answers a table question, users have no way to verify which cells informed which reasoning steps. We introduce RSAT, a method that trains small language models (SLMs, 1-8B) to produce step-by-step reasoning with cell-level citations grounded in table evidence. Phase 1 (SFT) teaches a structured JSON output format from verified reasoning traces. Phase 2 (GRPO) optimizes a composite reward centered on NLI-based faithfulness, alongside citation validity and parsimony. Across six models from two families-Qwen 2.5 (1.5B/3B/7B) and Llama 3 (1B/3B/8B)-RSAT improves faithfulness 3.7× over SFT alone (0.224→0.826), with near-perfect citation validity (0.992). Post-hoc attribution collapses below 13% format success, confirming that attribution must be integrated into reasoning, not retrofitted. Ablations show the faithfulness reward is essential: removing it drops faithfulness from 0.97 to 0.03.
Jugal Gajjar, Kamalasankari Subramaniakuppusamy
Department of Computer Science, The George Washington University, USA
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.