TABQAWORLD: Optimizing Multimodal Reasoning for Multi-Turn Table Question Answering
Authors: Tung Sum Thomas Kwok, Xinyu Wang, Xiaofeng Lin, Peng Lu, Chunhe Wang, Changlun Li, Hanwei Wu, Nan Tang, Elisa Kreiss, Guang Cheng
Organizations: University of California, Los Angeles · SimpleWay · McGill University · Université de Montréal · The Hong Kong University of Science and Technology (Guangzhou)
Multimodal reasoning has emerged as a powerful framework for enhancing reasoning capabilities of reasoning models. While multi-turn table reasoning methods have improved reasoning accuracy through tool use and reward modeling, they rely on fixed text serialization for table state readouts. This introduces representation errors in table encoding that significantly accumulate over multiple turns. Such accumulation is alleviated by tabular grounding methods in the expense of inference compute and cost, rendering real world deployment impractical. To address this, we introduce TABQAWORLD, a table reasoning framework that jointly optimizes tabular action through representation and estimation. For representation, TABQAWORLD employs an action-conditioned multimodal selection policy, which dynamically switches between visual and textual representations to maximize table state readout reliability. For estimation, TABQAWORLD optimizes stepwise reasoning trajectory through table metadata including dimension, data types and key values, safely planning trajectory and compressing low-complexity actions to reduce conversation turns and latency. Designed as a training-free framework, empirical evaluations show that TABQAWORLD achieves state-of-the-art performance with 4.87% accuracy improvements over baselines, with 5.42% accuracy gain and 33.35% inference latency reduction over static settings, establishing a new standard for reliable and efficient table reasoning.
Figures & tables
Figure 1: Motivation and overview of TabQAWorld . Fixed text serialization introduces state tracking noise (representation bottleneck), which propagates across multi-step reasoning and causes trajectory drift (estimation bottleneck). TabQAWorld addresses such failure process by jointly optimizing what to see and what to expect .
WTQ
MMQA
MMTU
Accuracy ± SD under different perceived modality
json
0.722 (0.032)
0.594 (0.035)
0.303 (0.033)
latex
0.707 (0.032)
0.574 (0.035)
0.333 (0.034)
Markdown
0.712 (0.032)
0.584 (0.035)
0.364 (0.034)
Image
0.818 (0.027)
0.614 (0.035)
0.374 (0.034)
Multimodal
0.788 (0.029)
0.635 (0.034)
0.364 (0.034)
Table 1: Reasoning performance and costs.
Figure 2: An illustrative example of how image-based parsing facilitates more human-preferred column attention than text-serialized tables. The value below each table indicates the mean-squared error (MSE) against human-preferred binary attention.
Figure 3: Hallucinations in full table estimation from frontier GPT-5.4 motivate lower-dimensional state estimation.
Figure 4: TabQAWorld dynamically selects the optimal data modality based on task purposes, and optimizes reasoning trajectory based on low dimensional metadata to minimize token usage and latency while maintaining a rigorous feedback loop to ensure convergence on an accurate final answer.
Figure 5: Illustration of metadata-guided execution. A mismatch in key_output (1964 vs. 1968) triggers replanning.
TQA
TFV
Summary
Method
WTQ
TABMWP
TAT-QA
HiTab
FeTaQA
TabFact
InfoTabs
WTQ + TabFact
Average
(Acc.)
(Acc.)
(Acc.)
(Acc.)
(BLEU)
(Acc.)
(Acc.)
(Acc.)
(Acc.)
Table-as-Text Baselines
Llama-2-7B
16.39
22.82
13.73
10.72
10.93
9.20
38.92
12.78
18.63
Llama3-Instruct-8B
21.24
42.01
13.08
6.97
12.66
73.89
54.00
47.57
35.20
TableLlama-7B
24.97
10.10
19.04
46.57
38.38
79.37
46.57
52.17
37.77
Table 2: Benchmark results for TabQAWorld and baseline models.
Figure 6: Ablation studies of TabQAWorld . (a): TabQAWorld brings model-agnostic joint improvement in accuracy and latency. (b): Visual-based modalities consistently outperform across question types. (c, d): TabQAWorld agent has internalized respective policy in modality selection and action compression preference. (e): Inference latency after trajectory optimization drops while retaining comparable compute.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task Type
Dataset
Train
Validation
Test
Table Question Answering (TQA)
TABMWP
500
200
1,000
WTQ
500
200
1,000
HiTab
500
200
1,000
TAT-QA
—
—
772
FeTaQA
500
200
1,000
Table Fact Verification (TFV)
TabFact
500
200
1,000
Appendix
Table 3: Dataset statistics by task type.
Parameter
Value / Default
Description
Image Rendering Hyperparameters
Max rows displayed
30
Truncate overly wide tables
Max cols displayed
10
Truncate overly long tables
Font size
8pt
Standardize patch size for each word
Row scale
1.2 ×
Standardize patch size for each row
DPI
200
Standardize resolution
Appendix
Table 4: Summary of image rendering, model generation, and experiment hyperparameters.
Figure 7: Overall performance of all models in the three datasets.
Accuracy
Modality
json
latex
Markdown
Image
Multimodal
Adaptive
json
latex
Markdown
Image
Image
Multimodal
Adaptive
Optimization
✗
✗
✗
✗
✗
✗
✓
✓
✓
✓(Metadata)
✓(Table)
✓
✓
WTQ
Qwen2.5-VL-7B
0.4343
0.4293
0.5101
0.5450
0.5859
0.6364
0.4293
0.4343
0.4646
0.5606
0.4814
0.6061
0.7020
Qwen3-VL-8B
0.7222
0.7071
0.7121
0.8182
0.7929
0.8333
0.7071
0.6869
0.7323
0.7980
0.7525
0.8131
0.8889
GPT-5-nano
0.8434
0.8535
0.8586
0.8636
0.8535
0.9040
0.8535
0.8131
0.8636
0.8586
0.8131
0.8687
0.9293
Appendix
Table 5: Detailed accuracy of Figure 6 (a) and Figure 7
Standard Deviation
Modality
json
latex
Markdown
Image
Multimodal
Adaptive
json
latex
Markdown
Image
Image
Multimodal
Adaptive
Optimization
✗
✗
✗
✗
✗
✗
✓
✓
✓
✓(Metadata)
✓(Table)
✓
✓
WTQ
Qwen2.5-VL-7B
0.0352
0.0352
0.0355
0.0354
0.0350
0.0342
0.0352
0.0352
0.0354
0.0353
0.0355
0.0347
0.0325
Qwen3-VL-8B
0.0318
0.0323
0.0322
0.0274
0.0288
0.0265
0.0323
0.0330
0.0315
0.0285
0.0307
0.0277
0.0223
GPT-5-nano
0.0258
0.0251
0.0248
0.0244
0.0251
0.0209
0.0251
0.0277
0.0244
0.0248
0.0277
0.0240
0.0182
Appendix
Table 6: Detailed standard deviation for Figure 6 (a) and Figure 7
Inference Latency (in seconds)
Modality
json
latex
Markdown
Image
Multimodal
Adaptive
json
latex
Markdown
Image
Image
Multimodal
Adaptive
Optimization
✗
✗
✗
✗
✗
✗
✓
✓
✓
✓(Metadata)
✓(Table)
✓
✓
WTQ
Qwen2.5-VL-7B
46.47
20.05
27.62
33.35
12.73
10.28
10.50
8.58
4.59
7.95
12.68
3.50
6.75
Qwen3-VL-8B
75.96
56.56
78.85
36.22
52.48
34.44
69.67
35.59
59.70
54.28
40.89
35.00
30.83
GPT-5-nano
31.86
37.71
28.48
26.07
22.73
29.80
18.99
18.17
14.31
13.27
27.34
15.61
26.53
Appendix
Table 7: Detailed time latency of Figure 6 (a) and Figure 7
ROUGE / BLEU
Modality
json
latex
Markdown
Image
Multimodal
Adaptive
json
latex
Markdown
Image
Image
Multimodal
Adaptive
Optimization
✗
✗
✗
✗
✗
✗
✓
✓
✓
✓(Metadata)
✓(Table)
✓
✓
WTQ
Qwen2.5-VL-7B
0.608 / 0.029
0.618 / 0.039
0.580 / 0.035
0.599 / 0.037
0.640 / 0.024
0.582 / 0.026
0.546 / 0.009
0.496 / 0.024
0.512 / 0.016
0.573 / 0.024
0.467 / 0.000
0.565 / 0.029
0.642 / 0.038
Qwen3-VL-8B
0.709 / 0.038
0.713 / 0.030
0.715 / 0.028
0.769 / 0.034
0.768 / 0.034
0.810 / 0.040
0.717 / 0.036
0.722 / 0.031
0.753 / 0.034
0.774 / 0.026
0.780 / 0.031
0.775 / 0.035
0.852 / 0.035
GPT-5-nano
0.745 / 0.045
0.742 / 0.033
0.758 / 0.037
0.764 / 0.031
0.746 / 0.025
0.785 / 0.040
0.772 / 0.035
0.732 / 0.040
0.738 / 0.037
0.734 / 0.038
0.701 / 0.031
0.744 / 0.040
0.793 / 0.039
Appendix
Table 8: ROUGE ( Lin, 2004 ) and BLEU ( Papineni et al., 2002 ) score of Main Result Figure 6 (a)
Figure 8: We annotate human-preferred attention standard, and visual-included representations not only lead to higher accuracy in different question types, but also lower MSE to human-preferred attention onto the table. This suggests that the relationship between more precise grounding over relevant table regions and quality of table reasoning.
Question
Ground Truth
Required Actions
How many plants are in Algeria?
6
sum_over_column
What is the average capacity of Algeria plants?
4.933
average_over_column
What is the name and capacity of the plant in Brunei?
Lumut 1 with 7.2 mmtpa
lookup_row
What is the average capacity of Malaysia plants?
6.267
lookup_row , average_over_column
What is the latest plant built in Qatar?
2010
lookup_row , max_over_column
What is the average capacity of all plants?
8.508
average_over_column
Appendix
Table 9: Designed questions and required reasoning actions.
Table 10: Canonical serialization templates for the same 3-column row.
Overall Metrics (Correct Q / Wrong Q)
Bin (Tokens)
JSON
LaTeX
Markdown
Accuracy (%)
72.2
70.7
71.2
Average Structural Token
231
147
121
Total Token
469
416
417
Format overhead ratio (%)
51.8 (51/55)
37.0 (38/34)
46.9 (48/45)
Header-data attention ratio
1.29 (1.3/1.2)
1.95 (2.0/1.7)
2.10 (2.3/1.6)
Appendix
Table 11: LLM Accuracy (%) by Token-Size Bin: Aggregated and Per-Dataset Breakdown
Figure 9: Heatmap of question-type performance and attention statistics across JSON , LaTeX and Markdown table serializations. Results show that while JSON devotes more attention to tables to improve lookup tasks, its heavier format overhead leads to underperformance in ordered computation tasks across multiple rows.
Overall Accuracy (%) (WTQ/MMQA/MMTU)
Bin (Tokens)
JSON
LaTeX
Markdown
Image
Multimodal
Adaptive
41–263
59.5% (80/60/21)
60.2% (82/59/21)
58.3% (81/59/20)
62.8% (85/63/30)
63.1% (84/63/30)
70.1% (91/68/41)
264–375
74.5% (83/65/75)
75.0% (85/67/79)
76.9% (85/70/75)
77.7% (86/63/82)
80.0% (85/72/82)
83.5% (88/74/86)
376–510
73.5% (88/67/74)
74.8% (84/66/74)
76.1% (89/68/86)
74.3% (88/69/81)
74.8% (88/67/86)
78.5% (92/73/83)
513–816
64.9% (69/62/30)
66.8% (70/65/33)
69.7% (73/69/34)
69.0% (73/64/34)
70.7% (74/65/38)
75.5% (79/69/43)
817–1695
67.3% (74/67/27)
66.0% (73/66/31)
68.9% (75/66/32)
71.4% (80/68/34)
72.9% (78/69/38)
77.8% (82/77/42)
Appendix
Table 12: LLM Accuracy (%) by Token-Size Bin: Aggregated and Per-Dataset Breakdown
Reasoning over tabular data is a crucial capability for tasks like question answering and fact verification, as it requires models to comprehend both free-form questions and semi-structured tables. However, while methods like Chain-of-Thought (CoT) introduce reasoning chains, purely symbolic methodes are inherently limited by their blindness to holistic visual patterns. To address this, we propose the Coarse-to-Fine Multimodal Synthesis framework (CFMS), a novel two-stage paradigm that hierarchically decouples high-level visual perception from granular symbolic reasoning. In the Coarse Stage, CFMS leverages the Multimodal Large Language Models (MLLMs) to perform a one-time synthesis of a multi-perspective knowledge tuple. This tuple subsequently serves as a dynamic reasoning map to guide the fine stage, where a symbolic engine executes a targeted and efficient sequence of iterative operations over the table. Extensive experiments on the WikiTQ and TabFact benchmarks demonstrate that CFMS achieves competitive accuracy. The framework exhibits particular robustness when handling large tables and when instantiated with smaller backbone models, validating its effectiveness and generalizability.
Table images preserve structural information that are often lost in text serialization, and reasoning over them requires locating relevant rows, columns, and cells step by step. Current multimodal large language models (MLLMs) encode the whole image once before reasoning, so they cannot pick up row-, column-, and cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each reasoning step to that structure. We propose InterTab, an Interleaved structure-aware framework for CoT reasoning over Table images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware alignment (SSA) on InterTab-22K teaches the model to interleave reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.
Hanqian Li, Sirui Huang, Chen Ling +12
The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · Hong Kong Polytechnic University +4
We introduce V-tableR1, a process-supervised reinforcement learning framework that elicits rigorous, verifiable reasoning from multimodal large language models (MLLMs). Current MLLMs trained solely on final outcomes often treat visual reasoning as a black box, relying on superficial pattern matching rather than performing rigorous multi-step inference. While Reinforcement Learning with Verifiable Rewards could enforce transparent reasoning trajectories, extending it to visual domains remains severely hindered by the ambiguity of grounding abstract logic into continuous pixel space. We solve this by leveraging the deterministic grid structure of tables as an ideal visual testbed. V-tableR1 employs a specialized critic VLM to provide dense, step-level feedback on the explicit visual chain-of-thought generated by a policy VLM. To optimize this system, we propose Process-Guided Direct Alignment Policy Optimization (PGPO), a novel RL algorithm integrating process rewards, decoupled policy constraints, and length-aware dynamic sampling. Extensive evaluations demonstrate that V-tableR1 explicitly penalizes visual hallucinations and shortcut guessing. By fundamentally shifting multimodal inference from black-box pattern matching to verifiable logical derivation, V-tableR1 4B establishes state-of-the-art accuracy among open-source models on complex tabular benchmarks, outperforming models up to 18x its size and improving over its SFT baseline
Yubo Jiang, Yitong An, Xin Yang +7
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China