InstructTable: Improving Table Structure Recognition Through Instructions

Boming Chen; Zining Wang; Zhentao Guo; Jianqiang Liu; Chen Duan; Yu Gu; Kai zhou; Pengfei Yan

cs.CVApr 3, 2026

InstructTable: Improving Table Structure Recognition Through Instructions

Authors: Boming Chen, Zining Wang, Zhentao Guo, Jianqiang Liu, Chen Duan, Yu Gu, Kai zhou, Pengfei Yan

Organizations: Meituan · Beijing Institute of Technology

Abstract

Table structure recognition (TSR) holds widespread practical importance by parsing tabular images into structured representations, yet encounters significant challenges when processing complex layouts involving merged or empty cells. Traditional visual-centric models rely exclusively on visual information while lacking crucial semantic support, thereby impeding accurate structural recognition in complex scenarios. Vision-language models leverage contextual semantics to enhance comprehension; however, these approaches underemphasize the modeling of visual structural information. To address these limitations, this paper introduces InstructTable, an instruction-guided multi-stage training TSR framework. Meticulously designed table instruction pre-training directs attention toward fine-grained structural patterns, enhancing comprehension of complex tables. Complementary TSR fine-tuning preserves robust visual information modeling, maintaining high-precision table parsing across diverse scenarios. Furthermore, we introduce Table Mix Expand (TME), an innovative template-free method for synthesizing large-scale authentic tabular data. Leveraging TME, we construct the Balanced Complex Dense Synthetic Tables (BCDSTab) benchmark, comprising 900 complex table images synthesized through our method to serve as a rigorous benchmark. Extensive experiments on multiple public datasets (FinTabNet, PubTabNet, MUSTARD) and BCDSTab demonstrate that InstructTable achieves state-of-the-art performance in TSR tasks. Ablation studies further confirm the positive impact of the proposed tabular-data-specific instructions and synthetic data.

Figures & tables

Figure 1: Visualized comparison among traditional visual-centric TSR (VCTSR) models , vision-language models (VLM) , and InstructTable . By leveraging instruction pre-training and TSR fine-tuning to jointly model visual information and instruction dependencies, InstructTable enhance fine-grained structural comprehension of tables.

Figure 2: Implicit row problem visualization – identical images from divergent ground truths causing input-output misalignment, misleading model training. In the matrix, green regions denote normal cells while red dotted patterns indicate implicit rows.

Figure 3: TME synthesis pipeline. Authentic table data undergoes matrix processing, partitioning, splicing, content generating and validity checking to generate novel synthetic tables for scenario-agnostic data expansion. In the atomic cell matrix, “C” denotes independent cells, “L” denotes left-merged cells, “U” denotes up-merged cells, and “X” denotes bidirectionally merged (left and up) cells.

Figure 4: Main framework of InstructTable. The red dashed region indicates text encoding occurs only in training, where instruction embeddings are cached. During inference, cached vectors enable efficient processing without real-time text encoding.

Group	Templates
(1)	Recognize all cells.
	Recognize all cells, the table has $n$ rows.
	Recognize all cells, the table has $m$ columns.
	Recognize all cells, the table has $n$ rows and $m$ columns.
(2)	Cells in the $R$ rows.
	Cells in the $C$ columns.

Table 1: Instruction templates for TSR in InstructTable.

n

and

m

denote table rows and columns respectively;

R

and

C

be subsets of

\{1,2,\ldots,n\}

and

\{1,2,\ldots,m\}

, representing the randomly selected rows and columns;

x\in\{1,2,\ldots,n\}

and

y\in\{1,2,\ldots,m\}

indicate randomly chosen single row or column indices, ensuring high instruction diversity.

Figure 5: Attention heatmaps of the cross-attention layer under four instruction groups. Visualizations reveal how task-specific instructions dynamically modulate attention focus across table regions during parsing.

Methods	Param	FinTabNet		PubTabNet		MUSTARD	BCDSTab
Methods	Param	S-TEDS	TEDS	S-TEDS	TEDS	S-TEDS	S-TEDS	TEDS
General Vision-Language Models
GPT-4o ∗ [ 27 ]	-	89.02	81.30	87.05	74.68	89.13	62.70	47.17
GPT-4.1 ∗ [ 28 ]	-	90.75	84.64	87.61	77.14	91.37	74.88	56.75
Qwen2.5-VL-72B ∗ [ 2 ]	72B	87.75	79.86	85.95	76.36	89.00	78.88	68.36
Qwen3-VL-235B-A22B ∗ [ 35 ]	235B	92.26	88.36	91.75	85.37	92.94	81.41	72.31

Table 2: TSR results on various datasets, respectively. InstructTable results are derived from pre-training on mixed data and fine-tuning on target datasets. Bolds denote the best and underlines denote the secondary best. ∗ indicates results calculated by us from official checkpoints or online APIs.

\dagger

denotes no incorporation of TME-generated data during the entire training process.

Method	TME	FinTabNet		PubTabNet
Method	TME	S-TEDS	TEDS	S-TEDS	TEDS
Unitable	✘	96.91	95.10	95.00	93.23
Unitable	✓	97.75	95.85	96.09	94.83
InstructTable	✘	98.66	96.82	96.84	94.97
InstructTable	✓	98.78	97.65	97.51	96.63

Table 3: Comparative results of UniTable and InstructTable. “TME” refers to the incorporation of TME-generated data.

INS	TME	FinTabNet		PubTabNet
INS	TME	S-TEDS	TEDS	S-TEDS	TEDS
✘	✘	97.11	96.57	95.91	94.03
✓	✘	98.66	96.82	96.84	94.97
✘	✓	98.37	97.07	96.22	95.01
✓	✓	98.78	97.65	97.51	96.63
		( +1.67 )	( +1.08 )	( +1.60 )	( +2.60 )

Table 4: Ablation study of our proposed components on FinTabNet and PubTabNet, “INS” and “TME” refer to utilization of instruction pre-training and the incorporation of TME-generated data.

Instruction Group				PubTabNet
G 1	G 2	G 3	G 4	S-TEDS
✘	✓	✓	✓	93.93
✓	✘	✓	✓	96.21
✓	✓	✘	✓	96.04
✓	✓	✓	✘	96.05
✓	✓	✓	✓	96.34

Table 5: Ablation study of instruction groups on PubTbaNet without further fine-tuning. “G 1 ”,“G 2 ”,“G 3 ” and “G 4 ” refer to the corresponding instruction groups.

Figure 6: An overview of BCDSTab benchmark. (a) Representative image sample; (b) Dual-level bounding box visualization with blue indicating cell-level boundaries and red showing content-level text regions; (c) HTML ground truth representation for metric evaluation; (d) Atomic cell matrix color-coded by merge patterns: green for normal cells, red for left-merged cells, yellow for up-merged cells; (e)-(h) Multidimensional data statistics.

Figure 7: Visual comparison of different methods on BCDSTab, including: (a) input image, (b) MinerU2.5 results, (c) Gemini 2.5 Pro results, and (d) our InstructTable results. Red regions highlight erroneous positions in the predictions, with corresponding areas marked by green zones in our predictions and delineated by red dashed boxes in the input image for intuitive comparison.

Figure 8: Visual comparison of different methods on BCDSTab, including: (a) input image, (b) MinerU2.5 results, (c) Gemini 2.5 Pro results, and (d) our InstructTable results. Red regions highlight erroneous positions in the predictions, with corresponding areas marked by green zones in our predictions and delineated by red dashed boxes in the input image for intuitive comparison.

Explore similar work

StrucTab: A Structured Optimization Framework for Table Parsing
Jun 29, 2026Gengluo Li, Shangpin Peng, Chengquan Zhang +10Table Structure Recognition VLM Reasoning
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, China · LLM Department, Tencent, China +3
TableNet A Large-Scale Table Dataset with LLM-Powered Autonomous
Feb 27, 2026Ruilin Zhang, Kai YangTable Structure Recognition LLM-Assisted Annotation
School of Computer Science and Technology, Tongji University, 4800 Cao’an Road, Shanghai, 201804, China
Enhancing Table Structure Recognition via Bounding Box Guidance
Sep 8, 2026Lei Hu, Shuangping HuangTable Structure Recognition
South China University of Technology, Guangzhou, China

Jun 29, 2026J/K move · Enter open · S save

StrucTab: A Structured Optimization Framework for Table Parsing

Gengluo Li, Shangpin Peng, Chengquan Zhang, Binghong Wu+9

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, China · LLM Department, Tencent, China+3

Table Structure Recognition VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

Table parsing aims to convert table images into structured, machine-readable representations, a task requiring the joint perception of complex spatial layouts and textual content. While recent vision-language models (VLMs) enable end-to-end parsing, they typically rely on direct supervision of the final output, thereby bypassing the explicit intermediate reasoning that is crucial for understanding complex table structures. Furthermore, attempts to optimize these models using reinforcement learning (RL) are often hindered by unstable or ambiguous reward designs, limiting potential performance gains. To address these limitations, we propose StrucTab, a table parsing model learned through intermediate structural supervision and reward decomposition. At the modeling level, by decomposing the parsing process into human-inspired subtasks, such as row-column counting and merged-cell analysis, StrucTab progressively unifies them through a sequential reasoning strategy. At the optimization level, we introduce Uni-TabRL, a unified RL framework that leverages decomposed rewards (validity, structure, and content) to provide stable and informative optimization signals. Finally, at the evaluation level, we present TableVerse-5K, a large-scale, challenging benchmark encompassing diverse, real-world table scenarios. Extensive experiments demonstrate the state-of-the-art performance of StrucTab across all evaluated public benchmarks and significant improvements on TableVerse-5K, validating the effectiveness of explicit structural modeling and decomposed reward optimization. Code and benchmark are publicly available at https://github.com/VirtualLUOUCAS/StrucTab.

Abstract

Explore similar work

StrucTab: A Structured Optimization Framework for Table Parsing

TableNet A Large-Scale Table Dataset with LLM-Powered Autonomous

Enhancing Table Structure Recognition via Bounding Box Guidance