Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Despite the success of Large Language Models (LLMs) in table understanding, their internal mechanisms remain unclear. In this paper, we conduct an empirical study on 16 LLMs, covering general LLMs, specialist tabular LLMs, and Mixture-of-Experts (MoE) models, to explore how LLMs understand tabular data and perform downstream tasks. Our analysis focus on 4 dimensions including the attention dynamics, the effective layer depth, the expert activation, and the impacts of input designs. Key findings include: (1) LLMs follow a three-phase attention pattern -- early layers scan the table broadly, middle layers localize relevant cells, and late layers amplify their contributions; (2) tabular tasks require deeper layers than math reasoning to reach stable predictions; (3) MoE models activate table-specific experts in middle layers, with early and late layers sharing general-purpose experts; (4) Chain-of-Thought prompting increases table attention, further enhanced by table-tuning. We hope these findings and insights can facilitate interpretability and future research on table-related tasks.
Figures & tables
Dataset
Task
Structure
Avg Tokens
#Samples
WTQ
TQA
Flat
1012
500
HiTab
TQA
Hierarchical
964
500
AITQA
TQA
Hierarchical
651
500
TabFact
TFV
Flat
681
500
Table 1: Summary of 2,000 evaluation samples used in our empirical study.
Figure 1: The attention dynamics of Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct on tabular tasks. Each input consists of three segments: system prompt , table content , and user question . Upper : A case study at selected layers—left sub-panels show step-wise attention ratio trend throughout generation steps; right sub-panels show cell-level attention heatmaps. Lower : Aggregated results— Layer-wise Segment Attn Ratio shows the proportion of attention allocated to each segment per layer; Table Attn Entropy measures the degree of focus towards table cells, with lower entropy indicating a more concentrated attention distribution on specific cells); Attn Contribution measures the influence of each segment on the final output. Notably, the entropy minimum at Layer 10 and Layer 14 for Llama3.1-8B-Instruct align with the concentrated attention on the answer cell “China” in the upper heatmap.
Figure 2: Performance drop when masking table attention in different layer ranges across 5 models (averaged over 3 benchmarks × 2 formats). Llama-family models show minimal impact from Deep masking, while Qwen-family models degrade uniformly across all ranges.
Figure 3: Prediction stability analysis using LogitLens. Bars show KL divergence between each layer’s decoded distribution and the final output (lower = closer to final prediction). Lines show top-5 token overlap with the final output. Vertical dashed lines mark where predictions stabilize.
KL Depth
Overlap Depth
Model
Layers
Table
Math
Table
Math
Instruct LLMs
Qwen2.5-3B
36
32 (0.89) †
27 (0.75)
33 (0.92)
31 (0.86)
Qwen2.5-7B
28
25 (0.89) †
23 (0.82)
25 (0.89)
25 (0.89)
Qwen2.5-14B
48
44 (0.92) †
43 (0.90)
44 (0.92)
43 (0.90)
Llama-3.1-8B
32
26 (0.81)
29 (0.91)
25 (0.78)
29 (0.91)
Table 2: Stabilization depth comparison between table and math (GSM8K) tasks. Values are reported as effective layer / total layers (ratio). KL-based depth is higher for table tasks in 5 of 7 models ( † ), while Top- k overlap depth is broadly comparable.
Figure 4: Expert activation analysis in MoE models across three architectures (DeepSeek-V2, Qwen3-30B-A3B, OLMoE). Left : Activation heatmaps showing table-specific experts per layer (darker = higher activation probability). Middle : Entropy of expert activation distribution for table tasks (lower entropy indicates concentrated activation on fewer experts). Right : Number of overlapping experts between table and math (GSM8K) tasks per layer.
Figure 5: Comparison between Markdown and HTML table formats.
Figure 6: Comparison between direct answering and Chain-of-Thought (CoT) reasoning.
Markdown
HTML
Model
Vanilla
CoT
Vanilla
CoT
Instruct LLMs
Qwen2.5-7B
62.15
67.20
61.35
67.80
Llama3.1-8B
56.95
62.35
57.95
58.35
Table-Specific Fine-tuned LLMs
Table-R1-7B
69.75
82.30
70.30
81.60
Table 3: Average performance over 2,000 test samples under different input formats and inference strategies.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Examples of table serialization formats.
Model
WTQ
TabFact
HiTab
AITQA
Instruct LLMs
LLaMA3.2-1B-Instruct
6.00
5.40
6.40
9.80
LLaMA3.2-3B-Instruct
33.00
58.00
30.80
52.00
LLaMA3.1-8B-Instruct
50.20
70.80
49.00
57.80
Qwen2.5-3B-Instruct
52.40
78.00
49.80
55.60
Qwen3-4B-Instruct-2507
52.00
75.20
49.60
60.00
Appendix
Table 4: LLMs’ performance (accuracy %) on evaluation data of four benchmarks. Best results are bolded.
Markdown
HTML
Model
Direct
CoT
Direct
CoT
Instruct LLMs
Qwen2.5-7B
62.2 ± 2.1
67.2 ± 2.0 †
61.4 ± 2.1
67.8 ± 2.0 †
Llama-3.1-8B
57.0 ± 2.2
62.4 ± 2.1 †
58.0 ± 2.1
58.4 ± 2.1 †
Table-Specific Fine-tuned LLMs
Table-R1-7B
69.8 ± 2.0
82.3 ± 1.7 †
70.3 ± 1.9
81.6 ± 1.7 †
Appendix
Table 5: Performance (%) with 95% bootstrap confidence intervals under different input formats and inference strategies. † marks statistically significant differences between CoT and Direct (non-overlapping CIs).
Model
Total Layers
Eff. Layers
Eff. Ratio
Instruct LLMs
Llama-3.2-1B-Instruct
16
14
0.88
Llama-3.2-3B-Instruct
28
23
0.82
Meta-Llama-3.1-8B-Instruct
32
25
0.78
Qwen2.5-3B-Instruct
36
33
0.92
Qwen2.5-7B-Instruct
28
25
0.89
Appendix
Table 6: Layer effectiveness analysis on table-related tasks using LogitLens. Eff. Layers identifies the first layer where the predicted vocabulary distribution aligns with the final output (defined by the top-5 token overlap first exceeds 0.3). Eff. Ratio (Eff. Layers / Total Layers) reflects the stage at which the model stabilizes its prediction; a lower ratio indicates that the model reaches its final decision in its shallower layers.
Figure 8: The qualitative case example for Q1.
Figure 9: Top-32 retrieval heads’ overlap with attention norm heatmaps across LLMs. Background color indicates attention norm intensity (yellow = higher); red boxes mark the Top-32 retrieval heads identified by prior work. The substantial overlap, especially in middle-to-late layers, suggests that table understanding reuses pre-existing retrieval circuits.
Figure 10: Attention dynamics of Qwen3-30B-A3B-Instruct-2507.
Figure 11: Attention dynamics of TAMA.
Figure 12: Attention dynamics of Table-R1-Zero-8B.
Figure 13: Attention dynamics of Table-R1-Zero-7B.
Figure 14: Attention dynamics of TAMA-Qwen2.5.
Figure 15: Attention dynamics of Qwen3-4B-Instruct.
Figure 16: Attention dynamics of Qwen2.5-3B-Instruct.
Figure 17: Attention dynamics of Qwen2.5-14B-Instruct.
Figure 18: Attention dynamics of Qwen2.5-32B-Instruct.
Figure 19: Attention dynamics of TableGPT2-7B.
Figure 20: Attention dynamics of LLaMA3.2-3B-Instruct.
Figure 21: Attention dynamics of LLaMA3.2-1B-Instruct.
Figure 22: Attention dynamics of DeepSeek-V2-Lite-Chat.
Figure 23: Attention dynamics of OLMoE-1B-7B-Instruct.
Figure 24: Attention dynamics of Qwen’s 1B gated attention baseline.
Figure 25: Attention dynamics of Qwen’s 1B model with element-wise gated attention.
Figure 26: Attention dynamics of Qwen’s 1B model with Headwise Gated Attention.
Figure 27: Analysis results of LLMs’ effectiveness layer.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables
Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian +6
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China
Multi-step LLM reasoning over structured tables fails because planning and execution share no explicit cell-grounding contract. Existing methods constrain the planner to a left-to-right factorization at odds with table permutation invariance, and score intermediate states by generated content alone, overlooking cell grounding. We conduct a pilot study showing that diffusion language models (DLMs) produce more human-aligned and permutation-stable cell attention on tables than autoregressive models, with a 40.2% median reduction in attention-AUROC variability under row reordering. Motivated by this, we propose TABALIGN, a planned table reasoning framework that operationalizes the contract. TABALIGN pairs a masked DLM planner, whose bidirectional denoising emits plan steps as binary cell masks, with TABATTN, a lightweight verifier trained on 1,600 human-verified attention standards to score each step by its attention overlap with the plan-designated mask. Across eight benchmarks covering table question answering and fact verification, TABALIGN improves average accuracy by 15.76 percentage points over the strongest open-source baseline at comparable 8B-class scale, with a matched-backbone ablation attributing 2.87 percentage points of this gain to the DLM planner over an AR planner on a fixed reasoner. Cleaner DLM plans also accelerate downstream reasoning execution by 44.64%.
Tung Sum Thomas Kwok, Zeyong Zhang, Xinyu Wang +6
University of California, Los Angeles · SimpleWay · New Jersey Institute of Technology +4
Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.
Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)