Distortions are a natural consequence of tables collected from real-world sources, where scraping, extraction, or manual intervention can introduce semantic and structural inconsistencies. Humans can often identify and mentally correct such errors autonomously before reasoning over the data. In this work, we investigate whether Large Language Models (LLMs) possess a similar capability by evaluating them on 506 distorted table question answering (TQA) samples derived from canonical tables. Our findings reveal systematic weaknesses in current LLM pre-training paradigms, where models exhibit a strong inductive bias toward assuming tables are well-formed, often overlooking distortions or discarding inconsistent entries instead of repairing them. Even frontier models like GPT-5.4 and Opus-4.6 achieve only 68% accuracy under distortion. While distortion-aware prompting partially mitigates these failures, errors persist. Through lightweight finetuning, we demonstrate that models can learn to autonomously detect and repair human-recoverable distorted tables.
Figures & tables
Figure 1: LLMs often overlook distortions in tables. Illustration of structural and semantic table distortions and their influence on how LLMs reason.
Model
Input
Can.
Dist. Unaware
Dist. Aware
Distorted
Rbst.
Distorted
Rbst.
Sem †
Str ‡
All
Sem †
Str ‡
All
Table-Finetuned Large Language Models
TableLLM-7B
Text
34.79
12.00
3.97
7.76
22.30
11.40
3.53
7.25
20.84
TableGPT2-7B
Text
46.96
14.41
7.63
10.84
23.08
19.07
10.73
14.69
31.28
Open-sourced Large Language Models
Table 1: Depicts accuracy degradation (%) when answering identical queries over canonical versus distorted tables. Distortions are categorized into semantic (Sem † ) and structural (Str ‡ ). Robustness ( Rbst%=Dist./Can. ) quantifies performance preservation under distortion. Models are evaluated across None , Text and Image modes, with and without code sandbox ( ). Variance across 5 iterations ≈0.6857 .
Model
Can.
Distorted
Rbst.
Sem †
Str ‡
All
Qwen2.5-1.5B-Instruct
Base
26.09
18.97
17.65
18.48
70.83
Finetuned
31.52
22.07
25.29
23.26
73.79
± 1.33
± 5.64
± 9.44
± 6.55
–
Gemma-3-4B-it
Table 2: Finetuning improves success under distortion.
Distortions
Qwen2.5-1.5B
Gemma-3-4B
Base
Finetuned
Base
Finetuned
Semantic Distortions
CharMis
17.24
15.17 ± 6.26
20.69
32.41 ± 9.32
StrChar
25.00
30.00 ± 11.18
50.00
95.00 ± 11.18
LabMis
0.00
20.00 ± 14.49
22.22
20.00 ± 9.30
DecSwap
31.25
33.75 ± 5.59
37.50
41.25 ± 5.59
Table 3: Comparing base and finetuned Qwen2.5-1.5B and Gemma-3-4B-it models across distortion types.
Figure 2: The model must realize that the first and the second row have been merged together and they are not a single row data.
Figure 3: The model should align the displacement of partial columns into different rows correctly. This should not be interpreted as new data with missing fields.
Figure 4: The model must correctly map the country names and numerical values and not simply use a greater than logic and compute directly on the string.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Distortion Type
Percentage (%)
Semantic
character_misinterpretation
19.96
stray_characters_insertion
7.51
label_mismatch
6.32
decimal_separator_swap
13.83
Structural
Appendix
Table 4: Distribution of distortion types across the 506 samples curated for this study. Values represent percentage of the total samples.
Model
Input
Semantic
Structural
CharMis
StrChar
LabMis
DecSwap
Vert
Horz
CMerge
CSplit
Foot
Table-Finetuned Large Language Models
TableLLM
Text
10.53
11.43
15.62
12.70
3.39
0.00
6.32
0.00
9.52
TableGPT2
Text
6.00
29.73
21.88
14.93
6.67
3.70
11.58
3.23
9.09
Open-source Large Language Models
Qwen2.5VL
Text
10.89
55.26
28.12
27.14
9.84
14.29
23.40
19.35
40.91
Appendix
Table 5: Breakdown of model robustness across individual distortion categories in a Distortion Unaware setting.
Model
Input
Semantic
Structural
CharMis
StrChar
LabMis
DecSwap
Vert
Horz
CMerge
CSplit
Foot
Table-Finetuned Large Language Models
TableLLM
Text
9.38
14.29
18.75
9.23
3.39
2.00
5.26
0.00
4.76
TableGPT2
Text
8.00
37.84
31.25
19.40
6.67
3.77
18.95
3.23
13.64
Open-source Large Language Models
Qwen2.5VL
Text
15.84
60.53
31.25
26.09
11.67
14.29
18.95
23.33
31.82
Appendix
Table 6: Breakdown of model robustness across individual distortion categories in a Distortion Aware setting.
Figure 5: Fine-tuning loss curve on Qwen2.5-1.5B-Instruct.
Figure 6: Fine-tuning loss curve on Gemma-3-4B-it.
Figure 7: Distortion-Unaware prompt given to models during evaluation.
Figure 8: Distortion-UnAware prompt given to models during evaluation without the code execution sandbox.
Figure 9: Distortion-Aware prompt given to models during evaluation.
Figure 10: Distortion-Aware prompt given to models during evaluation without the code execution sandbox.
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
Answering questions over imperfect tables requires handling errors that can affect the answer. We investigate two challenges for large language models (LLMs): whether error discovery depends on where errors appear in a table, and whether providing their locations is sufficient for accurate question answering (QA). Using human-reviewed instances from RADAR-T, we conduct controlled studies across three LLMs by varying row order and comparing original, error-marked, and repaired tables. First, reordering rows changes error discovery even when the table contents and gold answer remain unchanged. During direct inspection, LLMs are more likely to discover all rows containing relevant errors when these rows appear later in the table or are grouped more closely together. Second, providing verified error locations alone is insufficient for accurate QA: with code execution, accuracy on repaired tables exceeds that on error-marked tables by 39.0-59.1 percentage points across the three LLMs. As a practical application of these findings, we combine error discovery across shuffled table views with explicit guidance for verifying and handling the reported errors in a simple workflow, Geometry-Balanced Discovery and Intervention (GBDI). On RADAR-T, GBDI improves QA accuracy by 3.8-18.5 percentage points over a code-agent baseline across five LLMs (paired 95% confidence intervals exclude zero for four), at the cost of additional inference. These results highlight the importance of both reliable error discovery and effective error handling in QA over imperfect tables. Code is available at https://github.com/645-t/GBDI-ICLR-2027.
Baowen Zhang, Wei Fan, Ruman Wang +1
University of Wisconsin–Madison · University of Auckland, Auckland, New Zealand · Liaoning Provincial People’s Hospital +1
While large language models (LLMs) perform well on table tasks, they still make data referencing errors (DREs), i.e., incorrectly citing or omitting table values, despite understanding the table structure. Beyond final-answer accuracy, DREs directly compromise the correctness and reliability of intermediate reasoning steps. Yet prior studies have only offered limited, small-scale analyses. In this work, we present the first systematic evaluation of tabular data referencing errors across different models and tasks. Our results show that DREs occur across all tested models (1.7B to 20B parameters). Furthermore, we demonstrate that incorporating data referencing as a critic significantly improves answer accuracy up to 12.0%, through critic-based filtering and rejection sampling. Finally, we trained a lightweight 4B-parameter critic model that achieves an average F1 score of 78.2% in detecting both in-distribution and out-of-distribution DREs, and effectively assists inference for larger models.
Large Language Models have achieved remarkable success and are increasingly deployed in critical applications involving tabular data, such as Table Question Answering. However, their robustness to the structure of this input remains a critical, unaddressed question. This paper demonstrates that modern LLMs exhibit a significant vulnerability to the layout of tabular data. Specifically, we show that semantically-invariant permutations of rows and columns - rearrangements that do not alter the table's underlying information - are sometimes sufficient to cause incorrect or inconsistent model outputs. To systematically probe this vulnerability, we introduce Adversarial Table Permutation, a novel, gradient-based attack that efficiently identifies worst-case permutations designed to maximally disrupt model performance. Our extensive experiments demonstrate that ATP significantly degrades the performance of a wide range of LLMs. This reveals a pervasive vulnerability across different model sizes and architectures, including the most recent and popular models. Our findings expose a fundamental weakness in how current LLMs process structured data, underscoring the urgent need to develop permutation-robust models for reliable, real-world applications.
Xinshuai Dong, Haifeng Chen, Xuyuan Liu +5
CMU & NEC Labs · NEC Labs · Dartmouth College & NEC Labs +1