InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Hanqian Li; Sirui Huang; Chen Ling; Jungang Li; Yu Huang; Kening Zheng; Yonghua Hei; Xiangrong He

cs.CVSep 26, 2026

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Authors: Hanqian Li, Sirui Huang, Chen Ling, Jungang Li, Yu Huang, Kening Zheng, Yonghua Hei, Xiangrong He,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University · Zhejiang University ·

Abstract

Table images preserve structural information that are often lost in text serialization, and reasoning over them requires locating relevant rows, columns, and cells step by step. Current multimodal large language models (MLLMs) encode the whole image once before reasoning, so they cannot pick up row-, column-, and cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each reasoning step to that structure. We propose InterTab, an Interleaved structure-aware framework for CoT reasoning over Table images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware alignment (SSA) on InterTab-22K teaches the model to interleave reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.

Figures & tables

Figure 1: Comparison of two CoT-based reasoning methods on a table image.

Figure 2: Construction pipeline of InterTab-22K . Step 1: GLM-5.3-Flash is instructed to generate multi-turn trajectories with get_sub_table , where wrong visual evidence are corrected by locating the referenced cell in the table metadata. Step 2: LLM-as-judge for filtering trajectories based on visual-evidence structure and answer correctness. Step 3: deduplicate trajectories by sample IDs, do length-filtered, and balanced across the eight source datasets for SFT.

Figure 3: Overview of the InterTab framework. First, we perform supervised alignment of each reasoning step with interleaved, structure-aware visual tokens using InterTab-22K . Second, we optimize the model using ALO with tool and task rewards, enabling an active policy for interleaved reasoning and accurate visual localization.

General-purpose MLLMs
Model	HiTab	InfoTabs	TabFact	TabMWP	TATQA	WikiTQ	FeTaQA	HybridQA	VTQA	Average
GLM-4.1V-9B ( Team et al., 2026 )	71.43	72.93	82.03	86.58	38.37	72.03	38.81	54.01	51.75	63.10
InternVL3.5-8B ( Wang et al., 2025b )	77.12	71.67	87.10	95.53	51.26	69.07	42.17	70.94	49.67	68.28
Kimi-VL-A3B-Instruct ( Kimi Team et al., 2025 )	62.46	66.76	70.43	54.72	16.04	56.99	38.92	48.73	38.11	50.35
LLaVA-1.6 ( Liu et al., 2024 )	16.84	37.81	49.78	39.34	9.03	16.64	13.33	9.93	21.24	23.77
Qwen2.5-VL-7B-Instruct ( Bai et al., 2025b )	73.86	71.65	76.00	89.84	40.78	60.14	39.20	39.87	46.64	59.78

Table 1: Performance comparison on nine table understanding benchmarks. Accuracy (%) is reported for all benchmarks. Best results are in bold and second-best results are underlined.

Reasoning	HiTab	InfoTabs	TabFact	TabMWP	TATQA	WikiTQ	FeTaQA	HybridQA	VTQA	Average
Text-only CoT	71.45	70.44	87.01	94.91	51.25	72.78	45.19	72.60	54.40	68.89
Interleaved (Ours)	77.45	73.59	87.95	96.35	57.25	75.03	50.10	81.00	59.80	73.17
$\Delta$	+6.00	+3.15	+0.94	+1.44	+6.00	+2.25	+4.91	+8.39	+5.41	+4.28

Table 2: Interleaved reasoning versus a matched text-only baseline. Accuracy (%) is reported.

Figure 4: Training-strategy comparison. Top: accuracy of SSA-only, ALO-only, and SSA+ALO. Bottom: accuracy gain of SSA+ALO over each baseline, in percentage points.

Configuration	HiTab	InfoTabs	TabFact	TabMWP	TATQA	WikiTQ	FeTaQA	HybridQA	VTQA	Average
Ours	77.45	73.59	87.95	96.35	57.25	75.03	50.10	81.00	59.80	73.17
w/o call penalty	76.39	70.70	86.59	94.80	54.87	74.13	43.03	80.01	62.75	71.47
$\Delta$	$-1.06$	$-2.89$	$-1.36$	$-1.55$	$-2.38$	$-0.90$	$-7.07$	$-0.99$	$+2.95$	$-1.69$
w/o localization	76.14	68.72	86.02	97.46	51.99	73.93	40.95	80.59	56.86	70.30
$\Delta$	$-1.31$	$-4.87$	$-1.93$	$+1.11$	$-5.26$	$-1.10$	$-9.15$	$-0.41$	$-2.94$	$-2.87$

Table 3: Reward-component ablation. Accuracy (%) is reported.

\Delta

is the change relative to Ours.

Figure 5: (a) Average accuracy across unpenalized allowances. (b) Per-benchmark change from $N_{0}=7$ . (c) Accuracy on held-out Visual Table QA.

Figure 6: Case study of InterTab. Our method accurately localizes the most relevant region.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

General
Statistic	Mean	Min	Max
Total source samples	21,721
Samples containing tool calls	21,721 (100%)
Unique tools	1 ( get_sub_table )
Effective samples per SSA epoch	53,048 (with upsampling)
Conversation & Reasoning

Appendix

Table 4: Statistics of the SSA source dataset before upsampling. Text tokens are counted using the Qwen3-VL tokenizer without chat-template framing or image-patch tokens; literal <image> placeholders are retained. Observations contain only image placeholders.

Dataset	SSA source	SSA/epoch	ALO train	ALO val.
FeTaQA	1,311	6,631	400	10
HiTab	1,216	6,631	400	10
HybridQA	6,513	6,631	400	10
InfoTabs	410	6,631	400	10
TabFact	6,631	6,631	400	10
TabMWP	4,380	6,631	400	10

Appendix

Table 5: Dataset composition before and after SSA upsampling. ALO samples are drawn without replacement from the SSA source pool. ALO counts precede prompt-length filtering.

Hyperparameter	SSA	ALO
Framework	LLaMA-Factory	verl
Distributed strategy	DeepSpeed ZeRO-3	FSDP2
Initialization	Qwen3-VL-8B-Instruct	Instruct / SSA checkpoint
Vision tower	Frozen	Frozen
Multimodal projector	Frozen	Frozen
Learning rate	$10^{-5}$	$10^{-6}$

Appendix

Table 6: Training configurations. The SSA run was configured for three epochs, while the checkpoint used to initialize ALO was saved after epoch one (step 829). ALO values refer to the completed llmjudge_v1 main experiments, not sensitivity runs.

Figure 7: Inference prompt. The model may call get_sub_table and must place the final answer in <answer> tags.

Figure 8: Rollout prompt. Each step explains the needed evidence, then requests one complementary crop.

Figure 9: LLM-as-a-judge prompt. The judge checks semantic equivalence to the reference answer.

Figure 10: Effect of removing individual training sources. Left: score loss relative to the full mixture; positive values favor the full mixture. Outlined cells identify the corresponding source benchmark. Right: self-task and cross-task mean losses. VTQA is used only for evaluation.

Figure 11: ALO dynamics from the Base and SSA checkpoints. Curves show training reward, policy entropy, policy-gradient loss, and mean response length with EMA coefficient 0.9. The comparison covers the same 398 updates for each initialization.

Metric	Base $\to$ ALO	SSA $\to$ ALO
Mean reward	0.674	0.832
Rollout correctness (%)	79.89	87.61
Localization IoU	0.492	0.697
Format score	0.497	0.981
Policy entropy	0.039	0.069
Policy-gradient loss standard deviation	0.0346	0.0228

Appendix

Table 7: Late-training diagnostics from unsmoothed logs, averaged over steps 301–398 (98 updates). Policy-gradient loss variability is the sample standard deviation across updates. Higher is better for reward, correctness, and IoU; entropy and call count are descriptive diagnostics.

Figure 12: Decomposition of rollout reward during ALO. Positive areas show weighted correctness, tool-quality, and format contributions; the negative area combines missing-tool and excess-call penalties. The black curve is the observed total reward. All temporal components use EMA coefficient 0.9.

Figure 13: Rollout-level components during ALO. Lines show per-step means and shaded bands show the 10th–90th percentile range across rollouts, with EMA coefficient 0.9 applied to both. Bands represent rollout dispersion, not confidence intervals or variation across training seeds.

Figure 14: Rollout outcomes grouped by tool-call count across all recorded updates. Top: mean score, correctness, and tool quality; score error bars show

1.96

times the standard error within each bucket. Bottom: the fraction of rollouts in each bucket. The final bucket pools counts of ten or more.

Figure 15: Spearman correlations among rollout metrics, computed separately for Base and SSA across all recorded updates. The total score includes correctness, tool quality, and format by construction; these associations are descriptive and do not establish causality.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 65 papers

MERIT: Multimodal Evidence Reasoning over Image Tables
Oct 13, 2026Haoyang Jiang, Zhuotong Fan, Mingyang Yi +3Table QA Automated Fact-Checking
Renmin University of China · Independent Researcher · Beike Inc.
From Table to Cell: Attention for Better Reasoning with TABALIGN
May 14, 2026Tung Sum Thomas Kwok, Zeyong Zhang, Xinyu Wang +6Table QA Table Reasoning
University of California, Los Angeles · SimpleWay · New Jersey Institute of Technology +4
TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
Jun 8, 2026Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2Document Understanding Table QA
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)

Oct 13, 2026J/K move · Enter open · S save

MERIT: Multimodal Evidence Reasoning over Image Tables

Haoyang Jiang, Zhuotong Fan, Mingyang Yi, Ju Fan+2

Renmin University of China · Independent Researcher · Beike Inc.

Table QA Automated Fact-Checking Visual Question Answering Tabular Representation Learning

Open paper AskPDF arXiv

Table reasoning with language models typically assumes access to clean textual tables. In practice, tables are often serialized into text, which can corrupt cell contents, distort structural relationships, and discard visual cues, leaving the model to reason over incomplete evidence. In contrast, table images retain this information, but using it requires both accurate perception and reasoning grounded in the relevant cells. To address this, we propose MERIT (Multimodal Evidence Reasoning over Image Tables), a three-stage framework for learning to reason over table images, with serialized tables serving as optional auxiliary inputs (if provided). Our analysis identifies visual evidence acquisition as a bottleneck, motivating table-specific encoder adaptation through reconstruction, localization, and color-conditioned supervision. Building on the adapted representations, supervised fine-tuning teaches explicit evidence selection and reasoning using verified trajectories. Finally, reinforcement learning refines the policy with separate rewards for evidence correctness, calculation, final answers, and output format. By providing feedback on intermediate steps, the framework targets retrieval and computational errors that final-answer supervision alone does not distinguish. We evaluate the approach on table question answering and fact verification, analyze the contributions of visual inputs and training components, and assess generalization to additional datasets with different table structures and visual presentations.

Abstract

Table QA

Explore similar work

MERIT: Multimodal Evidence Reasoning over Image Tables

From Table to Cell: Attention for Better Reasoning with TABALIGN

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs