MERIT: Multimodal Evidence Reasoning over Image Tables

Haoyang Jiang; Zhuotong Fan; Mingyang Yi; Ju Fan; Lanqing Hu; Xiao Zan

cs.CVOct 13, 2026

MERIT: Multimodal Evidence Reasoning over Image Tables

Authors: Haoyang Jiang, Zhuotong Fan, Mingyang Yi, Ju Fan, Lanqing Hu, Xiao Zan

Organizations: Renmin University of China · Independent Researcher · Beike Inc.

Abstract

Table reasoning with language models typically assumes access to clean textual tables. In practice, tables are often serialized into text, which can corrupt cell contents, distort structural relationships, and discard visual cues, leaving the model to reason over incomplete evidence. In contrast, table images retain this information, but using it requires both accurate perception and reasoning grounded in the relevant cells. To address this, we propose MERIT (Multimodal Evidence Reasoning over Image Tables), a three-stage framework for learning to reason over table images, with serialized tables serving as optional auxiliary inputs (if provided). Our analysis identifies visual evidence acquisition as a bottleneck, motivating table-specific encoder adaptation through reconstruction, localization, and color-conditioned supervision. Building on the adapted representations, supervised fine-tuning teaches explicit evidence selection and reasoning using verified trajectories. Finally, reinforcement learning refines the policy with separate rewards for evidence correctness, calculation, final answers, and output format. By providing feedback on intermediate steps, the framework targets retrieval and computational errors that final-answer supervision alone does not distinguish. We evaluate the approach on table question answering and fact verification, analyze the contributions of visual inputs and training components, and assess generalization to additional datasets with different table structures and visual presentations.

Figures & tables

Figure 1: Common failure modes of OCR and table serialization. Converting table images into text can introduce content errors, structure errors, context loss, and visual cue loss, whereas the original table image preserves this evidence.

Figure 2: Overview of Merit . Table-centric visual encoder adaptation first specializes the visual encoder for table images. Multimodal SFT then teaches explicit evidence selection and structured reasoning, and table-verifiable RL further improves evidence and answer accuracy with verifiable rewards.

Panel of Figure 3 — Figure 3: Diagnosing the visual bottleneck in table question answering. (a) NaViT and MinerU 2.5 are connected to the same Qwen3 decoder. The accuracy gap under normal conditions largely disappears when gold facts are provided, indicating that the visual encoder is the primary bottleneck. (b) An example in which the table image corrects an evidence-selection error made from text-only input.

Method	LLM	TSD		TCE	TCL	MCD	RCE		TR			Avg.
		Row	Col.	Acc.	Acc.	F1	Row	Col.	HTML	Markdown	LaTeX
		Acc.	Acc.	Acc.	Acc.	F1	F1	F1	TEDS	TEDS	TEDS
Table-LLaVA 7B	Vicuna-1.5 7B	41.50	59.00	27.68	41.67	38.85	38.45	48.67	63.88	66.58	66.52	44.61
Table-LLaVA 13B	Vicuna-1.5 13B	44.40	75.00	30.88	45.49	43.60	37.74	55.02	66.24	66.75	66.69	48.77
Base	Qwen3-VL 8B-Instruct	37.80	72.40	37.46	38.63	30.99	41.19	68.02	59.26	88.19	46.25	46.89
Ours	Qwen3-VL 8B-Instruct	45.20	73.20	45.56	48.75	35.91	51.40	75.35	62.12	89.96	52.76	53.51

Table 1: Evaluation results on six held-in table-structure understanding benchmarks from MMTab ( Zheng et al., 2024 ) . Metric definitions follow Zheng et al. (2024) (see Section B.2 ). For benchmarks with multiple submetrics, we first average their submetrics; Avg. then denotes the macro-average over the six benchmarks. Higher values indicate better performance.

Figure 4: Illustration of a grounded reasoning trajectory. The model identifies supporting cells using logical coordinates, performs the required calculation, and produces the final answer in a standardized format. The example is illustrative, and intermediate steps are abbreviated for clarity.

Proprietary models
Method	Table Question Answering			Fact Verification		Avg.
Method	TABMWP	WTQ	TAT-QA	TabFact	InfoTabs	Avg.
GPT-5.5	89.97	88.96	85.62	93.65	76.59	86.96
Claude Sonnet 5	89.97	82.94	88.29	81.94	74.58	83.55
Gemini 3.5 Flash	84.62	87.63	88.63	92.64	78.93	86.49
General-purpose models

Table 2: Effectiveness on held-out splits of the five datasets used for reasoning post-training. Scores are accuracy (%); Avg. is the unweighted mean across the five datasets. The three rows for our model report results with different table inputs, indicated in parentheses.

\dagger

denotes results taken from published reports. Answers are extracted and evaluated against the ground truth, with ambiguous cases reviewed by GPT-5.5.

General-purpose models
Method	Serialized Table Available		No Serialized Table			Avg.
Method	HiTab	FinQA	FinMMR	TableVista	MMMU-T	Avg.
InternVL2.5-8B	31.03	24.96	25.20	35.30	22.35	27.77
Qwen3-VL-8B	67.43	48.55	33.02	69.54	42.77	52.27
Qwen3-8B	35.28	33.04	33.18	54.00	32.75	37.65
Table-specialized models

Table 3: Generalization ability on five additional datasets without further fine-tuning. Scores are accuracy (%); Avg. requires all five scores.

w/o Encoder adaptation
Variant	Table QA			Fact Verification		Avg.
Variant	TABMWP	WTQ	TAT-QA	TabFact	InfoTabs	Avg.	$\Delta$ Avg.
Base	85.97	61.04	82.51	77.71	73.04	76.05	–
+ SFT	97.38	71.52	86.53	88.49	78.39	84.46	+8.41
+ GRPO	93.68	60.52	81.74	77.93	73.13	77.40	+1.35
+ SFT + GRPO	97.70	71.52	86.14	89.38	77.78	84.50	+8.45

Table 4: Ablation of encoder adaptation and post-training. Scores are accuracy (%).

\Delta

Avg. is relative to the base.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Data source	Training examples
Table reconstruction	MMTab	65,068
Cell localization	MMTab	34,349
Color recognition	MMTab (recolored)	34,349
Table QA (auxiliary)	MMTab	50,000
Total	–	183,766

Appendix

Table 5: Statistics of the visual-adaptation data used in Stage 1.

Task	Instruction template
Table reconstruction	Reconstruct the structure and content of the table from the given table image.
Cell localization	Locate the logical cell corresponding to the specified target in the given table image.
Color recognition	Identify the visual color attribute of the specified cell in the given table image.
Table QA (auxiliary)	The original MMTab question for the given table image.

Appendix

Table 6: Instruction templates for the visual-adaptation tasks.

Dataset	Generated	Verified
TABMWP	2,000	1,540
WTQ	2,000	1,587
TAT-QA	2,000	1,660
TabFact	2,000	1,382
InfoTabs	2,000	1,856

Appendix

Table 7: Numbers of generated and verified SFT trajectories per dataset.

Dataset	Task	Prompts
TABMWP	Table QA	2,000
WTQ	Table QA	2,000
TAT-QA	Table QA	2,000
TabFact	Fact verification	2,000
InfoTabs	Fact verification	2,000

Appendix

Table 8: Composition of the data used for GRPO training.

Hyperparameter	Visual adaptation	SFT	GRPO
Learning rate	5e-6	1e-5	2e-6
Batch size	24	128	64 prompts
Gradient accumulation steps	3	16	–
Training epochs / steps	3 / 6,782	2 / 126	5 / 780
Maximum sequence length	4,608	3,328	18,432 + 1,024
Optimizer	AdamW	AdamW	AdamW

Appendix

Table 9: Training hyperparameters for table-aware visual adaptation, supervised fine-tuning, and GRPO.

Panel of Figure 5 — Figure 5: Error analysis. (a) Error distribution of general-purpose VLMs and Merit . (b) A representative perception failure.

Encoder Objective	TSD		TCE	TCL	MCD	RCE		TR			Avg.
Encoder Objective	Row	Col.	Acc.	Acc.	F1	Row	Col.	HTML	Markdown	LaTeX	Avg.
Base	37.80	72.40	37.46	38.63	30.99	41.19	68.02	59.26	88.19	46.25	46.89
Recon only	34.70	70.60	40.06	44.79	21.44	45.41	71.06	69.74	91.33	66.23	48.82
Locate only	26.60	46.30	43.07	57.57	24.51	46.56	70.09	58.08	88.49	52.40	47.71
Color only	35.50	52.60	30.34	38.39	15.47	34.67	61.36	57.70	89.33	48.75	40.25
Recon + Locate	38.80	71.20	46.67	53.79	28.71	48.84	76.89	63.28	89.75	54.96	52.73

Appendix

Table 10: Encoder-objective ablation before SFT and GRPO. Tasks and metrics are described in Appendix B.2 . Avg. first averages the submetrics within each benchmark and then computes the macro-average over the six held-in benchmarks. Scores are accuracy, F1, or TEDS, depending on the benchmark.

Reward Variant	Table Question Answering			Fact Verification		Avg.
Reward Variant	TABMWP	WTQ	TAT-QA	TabFact	InfoTabs	Avg.
Answer only	97.94	71.29	87.18	89.73	78.93	85.01
Answer + Evidence	98.13	72.31	87.05	89.80	78.26	85.11
Answer + Calculation	98.04	72.03	88.08	89.48	79.31	85.39
Full	98.13	72.86	87.56	90.08	78.63	85.45

Appendix

Table 11: Effect of different reward variants with the final checkpoint fixed. Questions, table inputs, and decoding settings are unchanged. The three QA datasets are TABMWP, WTQ, and TAT-QA, while the two fact verification datasets are TabFact and InfoTabs. Avg. averages all five datasets. Scores are accuracy (%).

Figure 6: Image–text comparison case 1, illustrating temporal row alignment.

Figure 7: Image–text comparison case 2, illustrating within-row association.

Figure 8: Grounded reasoning case 1, showing the table, question, cited cells, reasoning process, and final answer.

Figure 9: Grounded reasoning case 2, showing the table, question, cited cells, reasoning process, and final answer.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 65 papers

Sep 26, 2026

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Table images preserve structural information that are often lost in text serialization, and reasoning over them requires locating relevant rows, columns, and cells step by step. Current multimodal large language models (MLLMs) encode the whole image once before reasoning, so they cannot pick up row-, column-, and cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each reasoning step to that structure. We propose InterTab, an Interleaved structure-aware framework for CoT reasoning over Table images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware alignment (SSA) on InterTab-22K teaches the model to interleave reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.

Hanqian Li, Sirui Huang, Chen Ling +12

The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University +4

Table QA Multimodal CoT Reasoning Tool Use in VLMs Visual Grounding

Ask PDF

Apr 22, 2026

V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization

We introduce V-tableR1, a process-supervised reinforcement learning framework that elicits rigorous, verifiable reasoning from multimodal large language models (MLLMs). Current MLLMs trained solely on final outcomes often treat visual reasoning as a black box, relying on superficial pattern matching rather than performing rigorous multi-step inference. While Reinforcement Learning with Verifiable Rewards could enforce transparent reasoning trajectories, extending it to visual domains remains severely hindered by the ambiguity of grounding abstract logic into continuous pixel space. We solve this by leveraging the deterministic grid structure of tables as an ideal visual testbed. V-tableR1 employs a specialized critic VLM to provide dense, step-level feedback on the explicit visual chain-of-thought generated by a policy VLM. To optimize this system, we propose Process-Guided Direct Alignment Policy Optimization (PGPO), a novel RL algorithm integrating process rewards, decoupled policy constraints, and length-aware dynamic sampling. Extensive evaluations demonstrate that V-tableR1 explicitly penalizes visual hallucinations and shortcut guessing. By fundamentally shifting multimodal inference from black-box pattern matching to verifiable logical derivation, V-tableR1 4B establishes state-of-the-art accuracy among open-source models on complex tabular benchmarks, outperforming models up to 18x its size and improving over its SFT baseline

Yubo Jiang, Yitong An, Xin Yang +7

School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China

Multimodal Reasoning Reinforcement Learning with Verifiable Rewards Multimodal CoT Reasoning Process Supervision Visually Grounded Reasoning RL for VLMs+1

Ask PDF

May 1, 2026

WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild

Using multimodal foundation models to analyze table images is a high-value yet challenging application in consumer and enterprise scenarios. Despite its importance, current evaluations rely largely on structured-text tables or clean rendered images, leaving the visual complexity of in-the-wild table images underexplored. Such images feature varied layouts and diverse domains that demand sophisticated structural perception and numerical reasoning. To bridge this gap, we introduce WildTableBench, the first question-answering benchmark for naturally occurring table images from real-world settings. WildTableBench comprises 402 high-information-density table images collected from online forums and websites across diverse domains, together with 928 manually annotated and verified questions spanning 17 subtypes across five categories. We evaluate 21 frontier proprietary and open-source multimodal foundation models on this benchmark. Only one model exceeds 50% accuracy, while all remaining models range from 4.1% to 49.9%. We further conduct diagnostic analyses to characterize model failures and reveal persistent weaknesses in structural perception and reasoning. These results and analyses provide useful insights into current model capabilities and establish WildTableBench as a valuable diagnostic benchmark for table image understanding. Dataset: https://huggingface.co/datasets/jzhuang/WildTableBench Code: https://github.com/hjzhe/WildTableBench Leaderboard: https://hjzhe.github.io/WildTableBench

Junzhe Huang, Xiaoxiao Sun, Yan Yang +6

The University of Queensland · Stanford University · The Australian National University +3

Visual Question Answering Table QA Table Structure Recognition Multimodal Reasoning Multimodal Foundation Models Multimodal Model Evaluation

Ask PDF

Abstract

Table QA

Explore similar work

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization

WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild