ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution

Yahia Alqurnawi; Preetom Biswas; Anmol Rao; Tejas Anvekar; Chitta Baral; Vivek Gupta

cs.CLFeb 17, 2026

ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution

Authors: Yahia Alqurnawi, Preetom Biswas, Anmol Rao, Tejas Anvekar, Chitta Baral, Vivek Gupta

Organizations: School of Computing and Augmented Intelligence Arizona State University Tempe, AZ 85281, USA

Abstract

Multimodal Large Language Models (mLLMs) are often used to answer questions in structured data such as tables in Markdown, JSON, and images. While these models can often give correct answers, users also need to know where those answers come from. In this work, we study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer. We evaluate several mLLMs across different table formats and prompting strategies. Our results show a clear gap between question answering and evidence attribution. Although question answering accuracy remains moderate, attribution accuracy is much lower, near random for JSON inputs, across all models. We also find that models are more reliable at citing rows than columns, and struggle more with textual formats than images. Finally, we observe notable differences across model families. Overall, our findings show that current mLLMs are unreliable at providing fine-grained, trustworthy attribution for structured data, which limits their usage in applications requiring transparency and traceability.

Figures & tables

Figure 1: A brief overview of the general workflow of our proposed framework: ViTaB-A benchmarking.

Strategy	Model	Markdown		JSON		Images		Average
Strategy	Model	QA	Attr.	QA	Attr.	QA	Attr.	QA	Attr.
Zero Shot	Qwen3-VL	60.00	35.50	61.50	01.00	62.00	45.40	61.16	27.30
	Gemma3	43.00	13.40	40.00	00.80	28.00	16.60	37.00	10.27
	Molmo2	55.50	19.50	59.50	01.00	48.50	33.60	54.50	18.03
	InternVL3.5	63.50	42.50	64.00	01.50	60.50	53.10	62.66	32.37
Few Shot	Qwen3-VL	62.00	21.50	61.50	01.50	60.50	34.90	61.33	19.30

Table 1: Model Accuracy in QA vs Attribution in % Across Prompting Strategies Across Open-source Models (for 4B Parameter); Note : green depicts overall best model, and red depicts worst.

		Markdown		JSON		Images
Strategy	Model	Row	Column	Row	Column	Row	Column
Zero Shot	Qwen3-VL	73.75	48.83	10.13	36.92	78.01	58.55
	Gemma3	70.86	23.17	5.83	21.08	47.82	30.98
	Molmo2	68.75	27.58	9.16	23.33	58.51	56.13
	InternVL3.5	78.25	36	4.04	42.33	69.24	59.79
Few Shot	Qwen3-VL	77	29.83	10.53	11.75	79.05	46.65

Table 2: Row vs Column Accuracy in % Across Modalities and Prompting Strategies, Across Open-source Models (for 4B Parameter); Note : green depicts overall best model, and red depicts worst.

		Markdown		JSON		Images
Strategy	Model	Internal	Verbal	Internal	Verbal	Internal	Verbal
Zero Shot	Qwen3-VL	0.56	0.69	0.42	0.62	0.60	0.62
	Gemma3	0.45	0.40	0.27	0.32	0.41	0.39
	Molmo2	0.73	0.56	0.82	0.68	0.83	0.38
	InternVL3.5	0.64	0.65	0.74	0.83	0.77	0.67
Few Shot	Qwen3-VL	0.52	0.73	0.59	0.80	0.55	0.55

Table 3: Confidence-Accuracy correlation for Internal and Verbal; Across Multiple Modalities.

Figure 2: Radar charts comparing model families across attribution accuracy, QA accuracy, and confidence gap (

1-|\text{verbal}-\text{internal}|

) under different prompting strategies.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

		Markdown		JSON		Images		Confidence
Strategy	Model	Internal	Verbal	Internal	Verbal	Internal	Verbal	Gap
Zero Shot	Qwen3-VL	81.36	60.8	71.55	48.94	85.72	80.94	15.98
	Gemma3	83.74	86.30	75.23	82.11	83.50	85.98	3.98
	Molmo2	53.53	64.8	38.9	43.2	45.84	96.22	21.98
	InternVL3.5	65.50	61.10	47.75	32.70	65.78	73.24	4.00
Few Shot	Qwen3-VL	77.4	52.6	60.2	25.58	83.58	77.02	21.99

Appendix

Table 4: Internal and Verbal Confidence Scores in % Across Modalities and Prompting Strategies. Difference is calculated as

|

avg(verbal) - avg(internal)

|

for each row.

		Markdown		JSON		Images
Strategy	Model	SS	CR	SS	CR	SS	CR
Zero Shot	Qwen3-VL	184.96	83.50	186.58	79.50	187.22	87.50
	Gemma3	165.61	80.32	183.73	80.72	186.49	83.70
	Molmo2	184.11	83.50	187.80	79.00	187.53	90.10
	InternVL3.5	183.09	80.50	185.54	81.50	177.33	79.00
Few Shot	Qwen3-VL	183.67	82.50	188.79	81.00	188.16	87.30

Appendix

Table 5: Model uncertainty quantification results for attribution across representations and prompting strategies. We report average prediction set size (SS, number of cells) and coverage rate (CR, %).

		Markdown		JSON		Images
Strategy	Model	F1	Accuracy	F1	Accuracy	F1	Accuracy
Zero Shot	Gemma3-4b	0.159	13.40	0.113	0.80	0.887	16.60
	Gemma3-12b	0.306	27.20	0.141	0.60	0.396	37.10
	Gemma3-27b	0.372	31.00	0.279	1.51	0.464	44.10
Few Shot	Gemma3-4b	0.713	6.20	0.006	0.60	0.160	14.20
	Gemma3-12b	0.247	22.80	0.004	0.40	0.396	36.90

Appendix

Table 6: Attribution Metrics for Gemma Model Family

		Markdown		JSON		Images
Strategy	Model	F1	Accuracy	F1	Accuracy	F1	Accuracy
Zero Shot	Qwen3-VL-2b	0.10	8.50	0.005	0.50	0.15	14.10
	Qwen3-VL-4b	0.379	35.50	0.015	1.00	0.48	45.40
	Qwen3-VL-8b	0.261	23.50	0.004	0.00	0.40	37.50
	Qwen3-VL-32b	0.515	49.50	0.021	1.00	0.72	69.70
Few Shot	Qwen3-VL-2b	0.068	6.50	0.010	1.00	0.09	8.20

Appendix

Table 7: Attribution Metrics for Qwen3-VL Model Family

		Markdown		JSON		Images
Strategy	Model	F1	Accuracy	F1	Accuracy	F1	Accuracy
Zero Shot	Molmo2-4b	0.218	19.5	0.013	1.00	0.355	33.6
Zero Shot	Molmo2-8b	0.222	20.00	0.015	0.05	0.369	34.80
Few Shot	Molmo2-4b	0.196	18.00	0.01	0.50	0.215	20.20
Few Shot	Molmo2-8b	0.096	8.50	0.00	0.00	0.243	22.80
Chain of Thought	Molmo2-4b	0.162	15.50	0.019	1.00	0.243	22.90

Appendix

Table 8: Attribution Metrics for Molmo2 Model Family

		Markdown		JSON		Images
Strategy	Model	F1	Accuracy	F1	Accuracy	F1	Accuracy
Zero Shot	InternVL3.5-4b	0.273	42.50	0.013	1.50	0.441	53.10
	InternVL3.5-8b	0.193	17.50	0.016	1.00	0.254	22.50
	InternVL3.5-14b	0.44	46.50	0.026	1.50	0.557	56.30
	InternVL3.5-38b	0.595	56.50	0.023	1.50	0.616	59.40
Few Shot	InternVL3.5-4b	0.125	36.50	0.008	1.00	0.136	52.60

Appendix

Table 9: Attribution Metrics for InternVL3.5 Model Family

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
Jun 8, 2026Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2Document Understanding Table QA
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)
RSAT: Structured Attribution Makes Small Language Models Faithful Table Reasoners
Apr 30, 2026Jugal Gajjar, Kamalasankari SubramaniakuppusamyLLM Reasoning LLM Grounding
Department of Computer Science, The George Washington University, USA
When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors
Jun 30, 2026Yuqing Yang, Qi Zhu, Zhen Han +5LLM Evaluation LLM Grounding
University of Southern California · AWS AI Labs

Jun 8, 2026J/K move · Enter open · S save

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee, Roy Ka-Wei Lee+1

1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)

Document Understanding Table QA LLM Evaluation Table Structure Recognition Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.

Abstract

Multimodal Large Language Models

Explore similar work

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

RSAT: Structured Attribution Makes Small Language Models Faithful Table Reasoners

When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors