Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

Kun-Yang Yu; Zhi Zhou; Shi-Yu Tian; Xiao-Wen Yang; Zi-Yi Jia; Ming Yang; Zi-Jian Cheng; Lan-Zhe Guo

cs.CLMar 25, 2026

Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

Authors: Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian, Xiao-Wen Yang, Zi-Yi Jia, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo,

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables

Figures & tables

Figure 3: TWT framework. It realizes robust and good performance via code-base neuro-symbolic reasoning in TVMU tasks.

Models	Multimodal Table QA				Multimodal Table Prediction				AVG_rank
Models	TabMWP	WikiTQ	FinQA	TAT-QA	Adoption	SkinCA	Pawpularity	Paintings
	ACC $\uparrow$	ACC $\uparrow$	ACC $\uparrow$	ACC $\uparrow$	ACC $\uparrow$	ACC $\uparrow$	nRMSE $\downarrow$	nRMSE $\downarrow$	-
API-based MLLMs
GPT-5.1	92.50	40.82	47.68	56.38	26.12	65.51	0.366	0.098	3.63
Qwen3VL-Plus	98.70	76.10	64.41	72.39	18.67	51.55	0.539	0.178	4.25
General Open-Source MLLMs

Table 1: Comparison of different models on multimodal table question answering and multimodal table prediction benchmarks. The best results are shown in bold , and the second-best results are underlined.

Method	Table QA		Table Prediction
Method	WikiTQ	TAT-QA	SkinCA	Paintings
	ACC $\uparrow$	ACC $\uparrow$	ACC $\uparrow$	nRMSE $\downarrow$
Qwen3VL-8B	50.64	62.53	40.38	0.172
+TO-SFT	78.38	71.31	71.08	0.099
+AL-GRPO	78.95	72.82	73.65	0.093

Table 2: Experimental Results of Qwen3-VL-8B under TO-SFT and AL-GRPO on Table QA and Table Prediction Benchmarks. Best is shown in bold

Method	Adoption	Pawpularity
Method	ACC $\uparrow$	nRMSE $\downarrow$
TWT w features	32.90	0.199
TWT w/o features	31.40	0.200

Table 3: Experiment results of TWT under situation w and w/o table features in multimodal table prediction tasks

Figure 4: Comparison of code execution accuracy of TWT w and w/o AL-GRPO

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Multimodal Understanding

Multimodal understanding studies how AI systems interpret and reason across text, images, video, audio, and other signals.

See all 475 papers

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning
Sep 26, 2026Hanqian Li, Sirui Huang, Chen Ling +12Table QA Multimodal CoT Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University +4
A Closer Look into LLMs for Table Understanding
Mar 16, 2026Jia Wang, Chuanyu Qin, Mingyu Zheng +3LLM Interpretability Transformer Interpretability
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
CFMS: A Coarse-to-Fine Multimodal Synthesis Framework for Enhanced Tabular Reasoning
Apr 13, 2026Qixian Huang, Hongqiang Lin, Tong Fu +5Table QA Multimodal Reasoning

Sep 26, 2026J/K move · Enter open · S save

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Hanqian Li, Sirui Huang, Chen Ling, Jungang Li+11

The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University+4

Table QA Multimodal CoT Reasoning Tool Use in VLMs Visual Grounding

Open paper AskPDF arXiv

Table images preserve structural information that are often lost in text serialization, and reasoning over them requires locating relevant rows, columns, and cells step by step. Current multimodal large language models (MLLMs) encode the whole image once before reasoning, so they cannot pick up row-, column-, and cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each reasoning step to that structure. We propose InterTab, an Interleaved structure-aware framework for CoT reasoning over Table images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware alignment (SSA) on InterTab-22K teaches the model to interleave reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.

Abstract

Multimodal Understanding

Explore similar work

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

A Closer Look into LLMs for Table Understanding

CFMS: A Coarse-to-Fine Multimodal Synthesis Framework for Enhanced Tabular Reasoning