Table Structure Recognition (TSR) requires the logical reasoning ability of large language models (LLMs) to handle complex table layouts, but current datasets are limited in scale and quality, hindering effective use of this reasoning capacity. We thus present TableNet dataset, a new table structure recognition dataset collected and generated through multiple sources. Central to our approach is the first LLM-powered autonomous table generation and recognition multi-agent system that we developed. The generation part of our system integrates controllable visual, structural, and semantic parameters into the synthesis of table images. It facilitates the creation of a wide array of semantically coherent tables, adaptable to user-defined configurations along with annotations, thereby supporting large-scale and detailed dataset construction. This capability enables a comprehensive and nuanced table image annotation taxonomy, potentially advancing research in table-related domains. In contrast to traditional data collection methods, This approach facilitates the theoretically infinite, domain-agnostic, and style-flexible generation of table images, ensuring both efficiency and precision. The recognition part of our system is a diversity-based active learning paradigm that utilizes tables from multiple sources and selectively samples most informative data to finetune a model, achieving a competitive performance on TableNet test set while reducing training samples by a large margin compared with baselines, and a much higher performance on web-crawled real-world tables compared with models trained on predominant table datasets. To the best of our knowledge, this is the first work which employs active learning into the structure recognition of tables which is diverse in numbers of rows or columns, merged cells, cell contents, etc, which fits better for diversity-based active learning.
Figures & tables
Datasets
Diversity
Annotation
Domain
Collection methodology
Tables
TableNet (Ours)
True
S, C, H, V
Telecom (configurable)
See Figure 2
445K
PubTabNet
False
S, C, H
Medicine
rule-based collection
510K
FinTabNet
False
T, S, C, H
Finance
augmentation
113K
TableBank
True
T, H
General
rule-based collection
145K
SynthTabNet
True
T, S, C, H
General
augmentation
600K
ICDAR 2019
False
T, X
General
institutional contribution
3K
Table 1: Comparison of TSR datasets. For TableBank and SynthTabNet, we randomly sampled 1,000 images to assess color diversity and found that fewer than 10 were colored, indicating weak diversity. For TabRecSet and WTW, their diversity came from being captured by cameras in real world. Annotation legend: T: table bounding box (supports table detection); S: structural annotation (e.g., HTML tags, spanning); C: cell-level annotation (bounding box and content); H/X: final HTML/XML output or re-constructable from S and C; V: visual labels (e.g., simplicity, color, border style) provided in our dataset. Definition of diversity: A table image dataset is deemed as diverse if it supports at least two distinct and meaningful classification standards by which its images can be systematically categorized. These standards may include structural, visual attributes, allowing for multifaceted analysis and evaluation. For example, one classification standard could be based on structural complexity, such as whether a table contains spanning cells; another could be based on visual style, including attributes like border presence, background color, or watermark interference. A dataset that enables classification along both dimensions provides more comprehensive coverage for training and evaluating table structure recognition models.
Figure 1: Workflow of our multi-agent system.
Figure 2: Data collection pipeline. From top to bottom: agent generating, web crawling and open source augmenting. Annotation explanation: is simple denotes whether the table contains cells spanning. is colored denotes whether the table includes any background color, colored borders, or non-black font colors. is lined specifies whether the cell borders are fully present; tables with only horizontal or vertical lines are considered not lined.
Figure 3: Invalid tables
Figure 4: examples in TableNet
Figure 5: Generated data composition
Complexity
Dim.
Avg Iters
Spear.
Pear.
Ken.
Simple
Struct.
1.03
0.8203
0.8111
0.7928
Topic
1.07
0.8004
0.8085
0.6990
Sem.
1.03
0.9253
0.8914
0.8522
Complex
Struct.
1.11
0.8826
0.8751
0.8832
Topic
1.08
0.7352
0.7408
0.6457
Sem.
1.17
0.8753
0.8411
0.8019
Table 2: Correlation between human and filling checker ranks. Struct. and Sem. represent structure and semantics.
dataset
TEDS
TableNet(Ours)
0.7403
PubTabNet
0.5041
FinTabNet
0.4495
SynthTabNet
0.5242
TableBank
0.5401
Table 3: TSR performance on unseen real-world data of models trained on different existing datasets.
is simple
is colored
is lined
Models
All
Simple
Complex
Colored
Colorless
Lined
Lineless
Qwen
Qwen2-VL-2B(FT)
0.877
0.860
0.912
0.874
0.892
0.891
0.861
Qwen2-VL-72B
0.721
0.822
0.604
0.726
0.713
0.711
0.729
Qwen2-VL-72B(1-shot)
0.696
0.834
0.587
0.706
0.685
0.699
0.695
Qwen2-VL-72B(5-shot)
0.688
0.791
0.576
0.695
0.682
0.689
0.687
Table 4: TEDS of HTML on TableNet and subsets.
Simple
Complex
Methods
with
without
with
without
Qwen2.5
0.954
0.978
0.782
0.835
DeepSeek-R1
0.938
0.974
0.940
0.901
GPT-4o
0.980
0.978
0.931
0.928
Agent Tool
1 -0.004
1 -0.003
1 -0.012
1 -0.009
Table 5: Table Fidelity using Structure-only TEDS of HTML generated by LLM through directly prompting. Small texts after ones means TEDS structure infidelity introduced by LLM infilling, which is lowest.
Figure 6: Compared with existing CoSyn pipeline, which is an LLM end-to-end method. In the middle subfigure, H represents horizontally lined and V represents vertically lined.
Figure 7: TSR Experiment results
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Sector
Industry
Spear.
Pear.
Ken.
Energy
Equipment & Service
0.8021 ± 0.015
0.7954 ± 0.007
0.7311 ± 0.007
0.8204 ± 0.016
0.8080 ± 0.015
0.7551 ± 0.010
Consumable Fuel
0.8344 ± 0.021
0.8047 ± 0.010
0.7612 ± 0.005
0.8121 ± 0.002
0.7853 ± 0.013
0.7224 ± 0.018
Material
Chemicals
0.8006 ± 0.017
0.8105 ± 0.015
0.7290 ± 0.022
0.8788 ± 0.027
0.8446 ± 0.017
0.8054 ± 0.020
Appendix
Table 6: Average semantic correlation adding or subtracting half standard deviation of multiple Global Industry Classification Standard (GICS) industries’ tables. For each industry, the upper results line represents Chinese tables and the other line represents English tables.