How to Utilize Complementary Vision-Text Information for 2D Structure Understanding

Jiancheng Dong; Pengyue Jia; Derong Xu; Jiawei Cheng; Jingyu Peng; Chao Zhang; Bowen Liu; Xin Sun

cs.CVMar 17, 2026

How to Utilize Complementary Vision-Text Information for 2D Structure Understanding

Authors: Jiancheng Dong, Pengyue Jia, Derong Xu, Jiawei Cheng, Jingyu Peng, Chao Zhang, Bowen Liu, Xin Sun,

Organizations: City University of Hong Kong · Baidu Inc.

Abstract

LLMs typically linearize 2D tables into 1D sequences to fit their autoregressive architecture, which weakens row-column adjacency and other layout cues. In contrast, purely visual encoders can capture spatial cues, yet often struggle to preserve exact cell text. Our analysis reveals that these two modalities provide highly distinct information to LLMs and exhibit strong complementarity. However, direct concatenation and other fusion methods yield limited gains and frequently introduce cross-modal interference. To address this issue, we propose DiVA-Former, a lightweight architecture designed to effectively integrate vision and text information. DiVA-Former leverages visual tokens as dynamic queries to distill long textual sequences into digest vectors, thereby effectively exploiting complementary vision--text information. Evaluated across 13 table benchmarks, DiVA-Former improves upon the pure-text baseline by 23.9% and achieves consistent gains over existing baselines using visual inputs, textual inputs, or a combination of both.

Figures & tables

Panel of Figure 1 — Figure 1: Pilot study on TCR. “Any-Correct” denotes the proportion of examples for which either the vision or text modality yields the correct answer.

Figure 2: The overall architecture of DiVA-Former. The model leverages spatial visual tokens (

\bm{v}_{i}

) extracted from a table image as dynamic queries. These queries attend to the linearized table text tokens (

\bm{c}_{j}

) via Cross-Attention to generate compact digest vectors (

\bm{d}_{i}

). A gating mechanism maintains an approximate identity mapping from the initial visual tokens to the output digest, preserving high-quality 2D spatial priors.

Method	Modality	Question Answering					Structure Understanding				Fact Verification		Text Generation		Avg
Method	Modality	WTQ	FeTaQA	TAT	HiTab	TabMWP	TSR	TCE	RCE	TCR	TabFact	InfoTabS	RotoWire	WikiBio	Avg
Direct	Vision	55.1	17.4	75.8	63.5	74.1	14.8	36.1	14.2	39.5	72.0	77.6	13.9	13.1	43.6
	Text	54.6	19.2	73.9	68.4	75.9	14.2	24.2	13.8	43.0	72.4	72.8	15.2	13.3	43.2
	V+T	55.2	18.4	75.7	68.0	73.8	15.1	35.7	14.3	48.7	72.8	74.8	14.0	12.9	44.6
Adapter	Vision	52.9	47.8	72.2	60.1	82.0	73.6	78.6	53.7	46.8	70.8	64.4	16.7	29.2	57.6
	Text	52.8	46.8	74.8	70.0	78.4	47.0	54.3	24.1	46.1	68.0	72.0	13.9	20.2	51.4

Table 1: Main results on 13 table benchmarks. The best results are highlighted in bold , and the second-best results are underlined . Our model delivers the strongest performance on 12 of the 13 benchmarks and surpasses all baselines in overall average score.

Figure 3: Prediction alignment barcode for the subset of examples where both unimodal settings fail. Each column represents a single example, and the color intensity indicates the number of correctly localized target coordinates. We categorize the comparison between DiVA-Former and the direct concat baseline into Win, Tie, and Loss groups. DiVA-Former successfully recovers substantially more of these difficult examples than direct concat, demonstrating that its performance gains stem from leveraging complementary cross-modal information to solve cases that unimodal approaches cannot.

Dataset	Direct concat	DiVA-Former
TCR	15.97%	55.23% $(\uparrow$ 39.26%)
HiTab	4.35%	26.09% $(\uparrow$ 21.74%)
TabMWP	3.70%	92.59% $(\uparrow$ 88.89%)
WTQ	5.26%	21.05% $(\uparrow$ 15.79%)
TabFact	5.56%	42.59% $(\uparrow$ 37.03%)
InfoTabS	5.66%	41.51% $(\uparrow$ 35.85%)

Table 2: Recovery rate on examples where both text-only and vision-only inputs fail.

Gate	Query	Context	Question Answering					Structure Understanding				Fact Verification		Text Generation		Avg
Gate	Query	Context	WTQ	FeTaQA	TAT	HiTab	TabMWP	TSR	TCE	RCE	TCR	TabFact	InfoTabS	RotoWire	WikiBio	Avg
$\checkmark$	Text	Text	22.8	44.3	72.8	56.1	91.2	71.0	75.4	38.9	34.5	62.0	70.8	17.0	35.2	53.2
$\checkmark$	Vision	Vision	48.6	47.9	72.8	58.8	89.2	76.7	78.5	61.3	64.2	71.2	72.4	18.3	32.3	60.9
$\checkmark$	Text	Vision	60.0	45.8	74.5	65.3	91.0	75.0	85.2	73.9	63.1	76.3	69.1	17.5	34.3	63.9
	Vision	Text	15.7	38.6	29.8	6.6	20.8	69.5	72.0	25.4	61.2	48.4	50.4	17.8	20.3	36.7
$\checkmark$	Vision	Text	60.9	50.4	78.0	70.5	92.0	79.1	85.3	76.2	71.6	79.1	75.6	18.7	35.5	67.1

Table 3: Ablation on modality roles and identity-preserving gates. “Query” indicates the initialization source of the resampler queries, and “Context” denotes the token sequence used as Keys/Values in cross-attention. The full DiVA-Former setting (Text context + Vision queries) with identity-preserving gates achieves the best performance.

Figure 4: Hyperparameter analysis of DiVA-Former. Left: training loss curves under different initial gate values

g_{0}

. Middle: final average score across the 13 benchmarks under different initial gate values

g_{0}

. Right: final average score under different numbers of DiVA-Former layers.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Vision	Text	V + T
2wikimqa	74.12	78.26	77.43
dureader	33.54	40.49	41.51
gov_report	25.47	26.26	25.79
hotpotqa	62.86	74.33	74.30
multi_news	21.62	23.06	22.69
multifieldqa_en	54.66	59.55	58.42

Appendix

Table 4: Comparison between visual compression and direct text input on long-context benchmarks. Despite strong performance from the visual compression framework, direct text input remains better on most datasets, indicating that current soft-embedding methods still incur non-trivial information loss.

Method	Modality	Question Answering					Structure Understanding				Fact Verification		Text Generation		Overall
Method	Modality	WTQ	FeTaQA	TAT	HiTab	TabMWP	TSR	TCE	RCE	TCR	TabFact	InfoTabS	RotoWire	WikiBio	Overall
Direct	Vision	55.1	17.4	75.8	63.5	74.7	14.8	36.1	13.8	39.5	72.0	77.6	13.9	13.1	43.6
	Text	54.6	19.2	73.9	68.4	75.3	14.2	24.2	14.8	43.0	72.4	72.8	15.2	13.3	43.2
	V+T	55.2	18.4	75.7	68.0	73.8	15.1	35.7	14.3	48.7	72.8	74.8	14.0	12.9	44.6
	T+V	54.8	18.3	70.8	68.5	76.2	14.0	35.4	14.5	45.9	70.8	75.2	13.5	13.6	44.0
Adapter	Vision	52.9	47.8	72.2	60.1	82.0	73.6	78.6	53.7	46.8	70.8	64.4	16.7	29.2	57.6

Appendix

Table 5: Main results with different modality order. Rows with modality T+V are highlighted in red.

Figure 5: Full-set per-example prediction alignment analysis on TCR. The left region corresponds to examples where DiVA-Former outperforms direct concatenation, the middle region indicates ties, and the right region contains examples where DiVA-Former underperforms. Notably, many examples in the left region are cases that unimodal models and direct concatenation fail to solve at all, while even on some examples in the right region DiVA-Former still recovers a substantial number of coordinates.

Document Understanding

Document understanding enables machines to interpret text, layout, tables, and images to extract information and answer questions about documents.

See all 93 papers

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
Jun 8, 2026Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2Document Understanding Table QA
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)
InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning
Sep 26, 2026Hanqian Li, Sirui Huang, Chen Ling +12Table QA Multimodal CoT Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University +4
Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner
Dec 30, 2024Yitong Zhou, Mingyue Cheng, Qingyang Mao +3Table Structure Recognition VLM Evaluation
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Artificial Intelligence Research Institute, iFLYTEK Co., Ltd

Jun 8, 2026J/K move · Enter open · S save

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee, Roy Ka-Wei Lee+1

1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)

Document Understanding Table QA LLM Evaluation Table Structure Recognition Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.

Abstract

Document Understanding

Explore similar work

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner