TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables

Xiaoyu Chen; Lu Dai; Hanqing Wang; Zhuoyu Li; Wenbin Dai; Yanzong Zheng; Zhenggang Xia; Junyong Lin

cs.AIApr 4, 2026

TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables

Authors: Xiaoyu Chen, Lu Dai, Hanqing Wang, Zhuoyu Li, Wenbin Dai, Yanzong Zheng, Zhenggang Xia, Junyong Lin,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong · Shanghai Jiao Tong University

Abstract

Structured tables are essential for conveying high-density information in professional domains such as finance, healthcare, and scientific research. Despite the progress in Multimodal Large Language Models (MLLMs), reasoning performance remains limited for complex tables with hierarchical layouts. In this paper, we identify a critical Perception Bottleneck through quantitative analysis. We find that as task complexity scales, the number of involved discrete visual regions increases disproportionately. This processing density leads to an internal "Perceptual Overload," where MLLMs struggle to maintain accurate spatial attention during implicit generation. To address this bottleneck, we introduce TableVision, a large-scale, trajectory-aware benchmark designed for spatially grounded reasoning. TableVision stratifies tabular tasks into three cognitive levels (Perception, Reasoning, and Analysis) across 13 sub-categories. By utilizing a rendering-based deterministic grounding pipeline, the dataset explicitly couples multi-step logical deductions with pixel-perfect spatial ground truths, comprising 6,799 high-fidelity reasoning trajectories. Our empirical results, supported by diagnostic probing, demonstrate that explicit spatial constraints significantly recover the reasoning potential of MLLMs. Furthermore, our two-stage decoupled framework achieves a robust 12.3% overall accuracy improvement on the test set. TableVision provides a rigorous testbed and a fresh perspective on the synergy between perception and logic in document understanding.

Figures & tables

Figure 2: Overview of the TableVision benchmark and the proposed grounding-enhanced reasoning framework. Our TableVision dataset encompasses diverse multimodal reasoning tasks across complex tables with varying structural densities (e.g., hierarchical headers). To mitigate structural entanglement , we introduce a decoupled fine-tuning paradigm. Rather than predicting the final answer end-to-end, the model explicitly performs question-aware structural localization (Step 1) to identify relevant cell coordinates. These spatial anchors are then integrated to condition the multi-step logical calculation (Step 2 & 3), enabling highly reliable and grounded table reasoning.

Dataset	Year	Format	Header	Bbox	Task Type	CoT	Size
WTQ [ 39 ]	2015	Text	Flat	$\times$	Retr., Multi-hop	$\times$	220k QA
ToTTo [ 38 ]	2020	Text	Flat	$\times$	Table-to-Text	$\times$	120k Tbl
PubTabNet [ 61 ]	2020	Img	Flat	$\times$	Structure Recog.	$\times$	568k Tbl
HybridQA [ 10 ]	2020	Text	Flat	$\times$	Retr., Multi-hop	$\times$	70k QA
OTT-QA [ 7 ]	2021	Text	Flat	$\times$	Retr., Multi-hop	$\times$	45k QA
TAT-QA [ 62 ]	2021	Text	Flat	$\times$	Retr., Arith.	$\times$	16k QA

Table 1: Comprehensive comparison with existing tabular datasets. "Hier." indicates multi-level hierarchical headers; "Bbox" denotes spatial grounding; "CoT" refers to step-by-step logical rationale. Compared to existing benchmarks, TableVision provides reasoning steps and bounding boxes that unify structural perception with complex multi-step reasoning.

Cognitive Level	Category	Train	Test	Total	Avg Bbox	Avg Steps
L1: Perception	Retrieval	1,813	371	2,184	3.10	4.33
	Listing	109	28	137	3.72	5.44
	Structure	394	114	508	4.12	2.61
L2: Reasoning	Comparison	494	113	607	3.33	4.58
	Arithmetic	393	48	441	4.52	4.54
	Ranking	344	47	391	4.66	4.78

Table 2: Hierarchical distribution and complexity metrics of TableVision. The taxonomy stratifies queries into three cognitive levels. Notably, the visual perception burden (Avg Bbox) scales exponentially from simple Retrieval to complex Multi-hop analysis.

Figure 3: The TableVision Annotation Pipeline. The framework integrates deterministic rendering for coordinate mapping, LLM-based rationale generation (CoT), and a human-in-the-loop verification loop (Modify/Drop) to ensure high-fidelity spatial-logical alignment.

Figure 4: The SFT training pipeline of our framework. We fine-tune LoRA adapters on a frozen Qwen3-VL-8B-Instruct backbone. The process is decoupled into Explanatory Structural Localization and Grounding-Conditioned Reasoning.

L1: Perception (N=513)
Category	N	MiniCPM-V	HiPPO	InternVL3	Phi-3.5-V	LLaVA-OV	DS-VL2	GLM-4.1V	Ovis2.5	Glyph	Qwen3-VL
Params	-	8B	8B	8B	4.2B	7B	4.5B*	9B	9B	9B	8B
Retrieval	371	73.0	73.6	36.4	24.8	57.7	61.2	80.6	76.5	78.4	78.8
Listing	28	57.1	39.3	25.0	21.4	21.4	17.9	42.9	75.0	82.1	59.3
Structure	114	66.7	39.5	48.2	20.2	50.0	61.4	41.2	20.2	78.1	90.4
L2: Reasoning (N=425)

Table 3: Comprehensive Zero-Shot Evaluation on the TableVision Test Set. Accuracy is reported in percentage (%). We evaluate 10 VLMs with comparable active parameter scales (4B to 9B). Evaluated variants include: MiniCPM-V-2.6, HiPPO, InternVL3-8B, Phi-3.5-Vision-Instruct, LLaVA-OneVision-7B, DeepSeek-VL2 (27B MoE, 4.5B active), GLM-4.1V-9B, Ovis2.5-9B, Glyph(GLM-4.1V-9B-Base), and Qwen3-VL-8B-Instruct.

Panel of Figure 5 — Figure 5: Quantitative analysis of Stage-1 spatial grounding and its correlation with downstream S2 Pipeline accuracy.

Setting	Perception	Reasoning	Analysis	Overall
# Test Samples	513	425	363	1301
Zero-shot	80.3	61.1	27.3	59.3
End-to-End SFT	70.8 ( -9.5 )	61.6 ( +0.5 )	60.6 ( +33.3 )	65.0 ( +5.7 )
Decoupled SFT (Ours)	79.1 ( -1.2 )	73.1 ( +12.0 )	59.2 ( +31.9 )	71.6 ( +12.3 )
Oracle (GT Bbox)	85.8 ( +5.5 )	83.9 ( +22.8 )	67.5 ( +40.2 )	80.0 ( +20.7 )

Table 4: Ablation Study on the TableVision Test Set. Zero-shot : Qwen3-VL-8B baseline. End-to-End SFT : Single-stage supervised model predicting CoT and answer directly. Decoupled SFT (Ours) : Our pipeline predicting bounding boxes before CoT. Oracle : Setting using ground-truth boxes.

Table Reasoning

See all 9 papers

Jun 8, 2026

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.

Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2

1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)

Document Understanding Table QA LLM Evaluation Table Structure Recognition Multimodal Model Evaluation Multimodal Understanding

Ask PDF

May 14, 2026

From Table to Cell: Attention for Better Reasoning with TABALIGN

Multi-step LLM reasoning over structured tables fails because planning and execution share no explicit cell-grounding contract. Existing methods constrain the planner to a left-to-right factorization at odds with table permutation invariance, and score intermediate states by generated content alone, overlooking cell grounding. We conduct a pilot study showing that diffusion language models (DLMs) produce more human-aligned and permutation-stable cell attention on tables than autoregressive models, with a 40.2% median reduction in attention-AUROC variability under row reordering. Motivated by this, we propose TABALIGN, a planned table reasoning framework that operationalizes the contract. TABALIGN pairs a masked DLM planner, whose bidirectional denoising emits plan steps as binary cell masks, with TABATTN, a lightweight verifier trained on 1,600 human-verified attention standards to score each step by its attention overlap with the plan-designated mask. Across eight benchmarks covering table question answering and fact verification, TABALIGN improves average accuracy by 15.76 percentage points over the strongest open-source baseline at comparable 8B-class scale, with a matched-backbone ablation attributing 2.87 percentage points of this gain to the DLM planner over an AR planner on a fixed reasoner. Cleaner DLM plans also accelerate downstream reasoning execution by 44.64%.

Tung Sum Thomas Kwok, Zeyong Zhang, Xinyu Wang +6

University of California, Los Angeles · SimpleWay · New Jersey Institute of Technology +4

Table QA Table Reasoning Structured Reasoning Diffusion Language Models LLM Reasoning

Ask PDF

Sep 26, 2026

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Table images preserve structural information that are often lost in text serialization, and reasoning over them requires locating relevant rows, columns, and cells step by step. Current multimodal large language models (MLLMs) encode the whole image once before reasoning, so they cannot pick up row-, column-, and cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each reasoning step to that structure. We propose InterTab, an Interleaved structure-aware framework for CoT reasoning over Table images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware alignment (SSA) on InterTab-22K teaches the model to interleave reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.

Hanqian Li, Sirui Huang, Chen Ling +12

The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University +4

Table QA Multimodal CoT Reasoning Tool Use in VLMs Visual Grounding

Ask PDF

Abstract

Table Reasoning

Explore similar work

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

From Table to Cell: Attention for Better Reasoning with TABALIGN

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning