Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

Yitong Zhou; Mingyue Cheng; Qingyang Mao; Jiahao Wang; Feiyang Xu; Xin Li

cs.CVDec 30, 2024

Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

Authors: Yitong Zhou, Mingyue Cheng, Qingyang Mao, Jiahao Wang, Feiyang Xu, Xin Li

Organizations: State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Artificial Intelligence Research Institute, iFLYTEK Co., Ltd

Abstract

Pre-trained foundation models have recently made significant progress in table-related tasks such as table understanding and reasoning. However, recognizing the structure and content of unstructured tables using Vision Large Language Models (VLLMs) remains under-explored. To bridge this gap, we propose a benchmark based on a hierarchical design philosophy to evaluate the recognition capabilities of VLLMs in training-free scenarios. Through in-depth evaluations, we find that low-quality image input is a significant bottleneck in the recognition process. Drawing inspiration from this, we propose the Neighbor-Guided Toolchain Reasoner (NGTR) framework, which is characterized by integrating diverse lightweight tools for visual operations aimed at mitigating issues with low-quality images. Specifically, we transfer a tool selection experience from a similar neighbor to the input and design a reflection module to supervise the tool invocation process. Extensive experiments on public datasets demonstrate that our approach significantly enhances the recognition capabilities of the vanilla VLLMs. We believe that the benchmark and framework could provide an alternative solution to table recognition.

Figures & tables

Figure 1: Comparison of modeling paradigms: domain-specific lightweight models vs. universal pre-trained VLLMs.

Granularity	Recognition Task	Description
Table-level	Visual Table Size Detection	Get the number of rows and columns.
Row-level	Row Index-based Data Recognition	Get the content list of a specific row.
Column-level	Column Index-based Data Recognition	Get the content list of a specific column.
Cell-level	Merged Cell Detection	Get contents of all merged cells.
	Content-based Cell recognition	Get the location of specific cell content.
	Index-based Cell Recognition	Get the cell content of specific location.

Table 1: Descriptions of the proposed hierarchical recognition tasks.

Figure 2: Experimental results of VLLMs for the proposed hierarchical tasks. The tasks evaluated include the following: merged cell detection (MCD), content/index-based cell recognition (CCR, ICR), index-based row/column data recognition (IRDR, ICDR), and visual table size detection (VTSD).

Figure 3: Row-Column Sensitivity Analysis of VLLMs on Hierarchical Tasks with Gemini (Left) and GPT (Right).

Challenge	Scenario	Description
Visual Conditions	Blur	The image is out of focus, with details appearing smeared or indistinct.
	Underexposure	The image is too dark, which may cause the content to be unclear.
	Overexposure	The image is overly bright, losing detail in some regions.
Table Border Quality	Unclear Borders	The image’s table borders are faint, blending into the background.
	Missing Borders	The table is without expected borders or separators.
	Thickened Borders	The table borders are thickened.

Table 2: Specific description of the challenges and scenarios in low-quality image inputs.

Figure 4: Evaluation results of bottleneck scenarios: abbreviations UE (Underexposure), OE (Overexposure), MB (Missing Borders), UB (Unclear Borders), TB (Thickened Borders).

Figure 5: Illustration of a pipeline for table image preprocessing leveraging a toolkit of lightweight vision models.

Tool	Descriptions
Border Enhancement	The border enhancement tool improves the legibility of tables and their structures by thickening the border lines in the image. This process enhances the structural information features of tabular data.
Image Upscaling	Image upscaling optimize image resolution to improve visual quality. This technique is commonly employed to repair and enhance blurry images.
Noise Reduction	The noise reduction tool enhances image quality by adjusting brightness and contrast to mitigate noise interference and underexposure issues.
Binarization	This tool converts images to black and white, highlighting key features for easier extraction.
Detection and Cropping	This tool identifies table regions within an image and crops them into independent segments. It is particularly suitable for processing images of tables embedded in complex backgrounds.

Table 3: Specific descriptions of built tools in the toolkit.

Dataset	Metrics	Lightweight OCR Model			Prompt Tuning in VLLMs
		EDD	LGPMA	LORE	Phi	GPT-mini	Qwen	Llama	Gemini			GPT-4o
		EDD	LGPMA	LORE	Phi	GPT-mini	Qwen	Llama	direct	NGTR	$\Delta$	direct	NGTR	$\Delta$
SciTSR	TEDS	-	95.08	-	66.18	87.18	89.40	87.24	90.15	91.07	+0.92	90.70	92.58	+1.88
SciTSR	TEDS-Struct	-	96.24	97.22	71.56	92.03	93.06	92.31	93.73	95.09	+1.36	94.20	95.78	+1.58
PubTabNet	TEDS	89.67	94.63	-	49.92	58.68	52.53	79.04	81.00	84.80	+3.80	74.46	85.03	+10.57
PubTabNet	TEDS-Struct	-	96.70	96.94	57.65	73.00	63.90	87.64	85.28	89.30	+4.02	84.91	92.31	+7.40

Table 4: Performance comparison of methods on the SciTSR, PubTabNet, and WTW datasets. ”-” indicates the method’s lack of results (specific reasons are provided in the implementation details). Best scores in the lightweight OCR model category are highlighted in blue , while best scores in the prompt tuning in VLLMs category are highlighted in green .

Method	SciTSR		PubTabNet
Method	TEDS	TEDS-Struct	TEDS	TEDS-Struct
NGTR	92.56	95.43	85.03	92.31
w/o EXP	90.33	93.68	80.57	88.40
w/o REF	91.53	94.77	82.08	91.85

Table 5: Ablation study results on key components of the framework. ”EXP” denotes the tool invocation experience learning module, ”REF” represents the reflection-driven tool selection module.

Dataset	Image Upscaling	Border Enhancement	Binarization	Noise Reduction	Detection and Cropping
PubTabNet	93.71	76.42	4.40	15.64	64.39
SciTSR	77.97	90.79	3.70	20.64	54.23
WTW	79.81	86.86	6.65	15.46	82.93

Table 6: Tool invocation preference analysis. Values show the percentage of samples using each tool, with bold values marking the dataset where each tool is most frequently used.

Figure 6: The impact of the number of tools (shown on the left) and the number of multi-paths (shown on the right) on performance.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Table size distribution and its impact on TR performance in SciTSR (left) and PubTabNet (right) Datasets

Figure 8: A case study is conducted using a sample from PubTabNet.

Figure 9: A case study is conducted using a sample from SciTSR.

Dataset	Tables	Source	Markup Sequence Annotations	Textual Annotations	Year
SciTSR	15K	ArXiv	$\times$	$\checkmark$	2019
PubTabNet	568K	PubMed	$\checkmark$	$\checkmark$	2020
WTW	14.6K	Wild Scenarios	$\times$	$\times$	2021

Appendix

Table 7: Characteristics of selected table recognition datasets.

Figure 10: Simplified prompt template for VLLMs on table recognition tasks.

Figure 11: Chain-of-Thought prompt template for VLLMs on table recognition tasks.

Figure 12: Prompt template on visual table size detection task.

Figure 13: Prompt template on merged cell detection task.

Figure 14: Prompt template on content-based cell recognition task.

Figure 15: Prompt template on index-based cell recognition task.

Figure 16: Prompt template for row index recognition task.

Figure 17: Prompt template for column index recognition task.

Figure 18: Prompt template for generating multiple tool invocation plans in the NGTR framework

Figure 19: Prompt template for the reflection-driven tool utilization module in the NGTR framework

Table Structure Recognition

Table structure recognition recovers rows, columns, cells, spans, and hierarchy from table images and documents.

See all 32 papers

Cycle-of-Thought: Self-Evaluative Iterative Reasoning for Table Visual Question Answering in Document Images
Oct 23, 2026Somraj Gautam, Gaurav HaritVisual Question Answering Reasoning for VLMs
IIT Jodhpur, Rajasthan, India.
TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables
Apr 4, 2026Xiaoyu Chen, Lu Dai, Hanqing Wang +6Table Reasoning Spatial Reasoning Benchmarks
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1
TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
Jun 8, 2026Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2Document Understanding Table QA
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)

Oct 23, 2026J/K move · Enter open · S save

Cycle-of-Thought: Self-Evaluative Iterative Reasoning for Table Visual Question Answering in Document Images

Somraj Gautam, Gaurav Harit

IIT Jodhpur, Rajasthan, India.

Visual Question Answering Reasoning for VLMs Self-Correction in VLMs LLM Self-Refinement

Open paper AskPDF arXiv

Recent advances in Vision-Language Models (VLMs) have significantly improved Visual Question Answering (VQA) performance, yet their reasoning over structured content such as tables remains inconsistent, often plagued by spatial misalignment and logical hallucinations. Existing approaches for improving table VQA typically rely on architectural modifications or expensive supervised fine-tuning (SFT), which impose high barriers to entry due to the need for large-scale, expert-annotated datasets and significant computational resources. In this work, we propose Self-RefineVQA, a training-free, inference-time self-correction framework that improves table VQA performance without additional labeled data or model updates. Inspired by human revision behavior, our method enables a single VLM to iteratively act as a generator, critic, and refiner, forming a closed self-feedback loop that corrects common reasoning errors through prompt-guided re-evaluation. Experiments on tabular VQA benchmarks demonstrate that Self-RefineVQA yields consistent accuracy gains in monolingual and cross-lingual understanding, with a particularly significant boost for small models, enabling lightweight models (e.g., 2B-7B parameters) to rival the performance of significantly larger counterparts. Furthermore, we provide a comparative complexity analysis demonstrating that while inference cost increases linearly, our framework offers superior data efficiency and a lower cumulative cost by eliminating the prohibitive annotation bottleneck of human labeling required for fine-tuning. Our results highlight the potential of prompt-based self-improvement as a scalable, accessible paradigm for reliable table reasoning in VLMs.

Abstract

Table Structure Recognition

Explore similar work

Cycle-of-Thought: Self-Evaluative Iterative Reasoning for Table Visual Question Answering in Document Images

TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs