Cycle-of-Thought: Self-Evaluative Iterative Reasoning for Table Visual Question Answering in Document Images

Somraj Gautam; Gaurav Harit

cs.CVOct 23, 2026

Cycle-of-Thought: Self-Evaluative Iterative Reasoning for Table Visual Question Answering in Document Images

Authors: Somraj Gautam, Gaurav Harit

Organizations: IIT Jodhpur, Rajasthan, India.

Abstract

Recent advances in Vision-Language Models (VLMs) have significantly improved Visual Question Answering (VQA) performance, yet their reasoning over structured content such as tables remains inconsistent, often plagued by spatial misalignment and logical hallucinations. Existing approaches for improving table VQA typically rely on architectural modifications or expensive supervised fine-tuning (SFT), which impose high barriers to entry due to the need for large-scale, expert-annotated datasets and significant computational resources. In this work, we propose Self-RefineVQA, a training-free, inference-time self-correction framework that improves table VQA performance without additional labeled data or model updates. Inspired by human revision behavior, our method enables a single VLM to iteratively act as a generator, critic, and refiner, forming a closed self-feedback loop that corrects common reasoning errors through prompt-guided re-evaluation. Experiments on tabular VQA benchmarks demonstrate that Self-RefineVQA yields consistent accuracy gains in monolingual and cross-lingual understanding, with a particularly significant boost for small models, enabling lightweight models (e.g., 2B-7B parameters) to rival the performance of significantly larger counterparts. Furthermore, we provide a comparative complexity analysis demonstrating that while inference cost increases linearly, our framework offers superior data efficiency and a lower cumulative cost by eliminating the prohibitive annotation bottleneck of human labeling required for fine-tuning. Our results highlight the potential of prompt-based self-improvement as a scalable, accessible paradigm for reliable table reasoning in VLMs.

Figures & tables

Panel of Figure 1 — Figure 1: The Challenge of Spatial Drift in Table VQA. Examples from (a) TableVQA-Bench, (b) DocVQA, and (c) WTQ. These samples illustrate how minor shifts in attention (spatial drift) lead to reasoning failures, highlighting the need for iterative re-evaluation provided by Self-RefineVQA.

Figure 2: Minor shifts in visual attention cause the model to attend to semantically adjacent but incorrect cells. The proposed Generator–Critic–Refiner cycle detects row-column misalignment and iteratively re-grounds attention to the correct table intersection.

Figure 3: The Cycle-of-Thought Framework : The structured Critic prompt changes the model’s conditioning context, prompting re-examination of spatial evidence rather than reinforcement of the initial output. The structured prompting demonstrably reduces error persistence, consistent with a visual re-grounding effect.

Dataset	Domain/Source	# Images	# QA Pairs
WikiTableQuestions (Image)	Wikipedia tables	2062	2062
DocVQA - Table	Real documents	1500	1500
TableVQA-Bench	Financial reports, scientific articles	1758	1758
IndotabVQA	Real Documents	1043	4172

Table 1: Dataset statistics used for evaluating Self-RefineVQA.

Model	Zero-shot	Self-consistency	Self-refine (Ours)	Improvement
WikiTableQuestions
TAPAS	24.10	24.10	26.96	+ 2.86
Qwen2-VL [2B]	53.54	58.58	61.06	+ 7.52
Qwen2.5-VL [3B]	58.97	56.45	71.97	+ 13.00
Qwen3-VL [2B]	65.86	44.40	69.59	+ 3.73
Qwen2-VL [7B]	63.29	65.91	74.59	+ 11.30

Table 2: Performance of Models on three different dataset.

Panel of Figure 4 — Figure 4: Iteration-wise accuracy progression across WikiTableQuestions, DocVQA-Table, and TableVQA-Bench. Most gains occur in the first 1-2 iterations, after which performance saturates, supporting our choice of Tmax = 3.

Lang.	Model	Zero-shot	Self-Cons.	Self-refine	$\Delta$
Arabic	Qwen2-VL	26.75	–	30.49	+3.74
Arabic	Qwen2.5-VL	26.94	–	33.08	+6.14
English	Qwen2-VL	39.41	–	44.49	+5.08
English	Qwen2.5-VL	40.08	–	45.45	+5.37
Bahasa	Qwen2-VL	69.51	–	75.26	+5.75
Bahasa	Qwen2.5-VL	68.84	–	76.80	+7.96

Table 3: Performance of Qwen-VL models on IndotabVQA across different languages. Self-refine denotes our proposed approach.

Dataset	Model	Stage 1 Only (Zero-shot)	Stage (1+2) (No Refine)	Stage (1+3) (No Critique)	Full Pipeline (Ours)
WikiTableQuestions	Qwen2.5-VL [7B]	63.48	63.48	68.57	72.55
	Qwen3-VL [8B]	73.62	73.62	74.35	75.90
DocVQA-Table	Qwen2.5-VL [7B]	89.48	89.48	90.27	93.31
	Qwen3-VL [8B]	93.63	93.63	94.37	95.02
TableVQA-Bench	Qwen2.5-VL [7B]	72.40	72.40	75.53	79.87
	Qwen3-VL [8B]	82.07	82.07	84.07	86.73

Table 4: Ablation study.

Metric	Supervised Fine-Tuning (SFT)	Self-RefineVQA (Ours)
Data	High (Requires 1k+ labeled pairs)	Zero (Training-free)
Compute	High (Backprop, Optimizer states)	Zero (Inference only)
Inference Cost	Low ( $1\times T_{\text{gen}}$ )	Moderate ( $\approx 4\times T_{\text{gen}}$ )
Memory	High (Weights + Gradients + Optimizer)	Low (Weights only)
Adaptability	Low (Requires retraining for new schemas)	High (Prompt adjustment)
Access	Open Weights (Required)	Black Box API (Compatible)

Table 5: Operational Efficiency Comparison

Figure 5: Failure taxonomy before and after Self-RefineVQA : We illustrate representative failure modes of zero-shot table VQA models (top), including spatial drift, numerical reasoning errors, and entity mismatches. Self-RefineVQA corrects these errors via iterative self-critique and refinement (bottom), re-grounding predictions in verified row–column intersections without additional training.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

Dec 30, 2024

Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

Pre-trained foundation models have recently made significant progress in table-related tasks such as table understanding and reasoning. However, recognizing the structure and content of unstructured tables using Vision Large Language Models (VLLMs) remains under-explored. To bridge this gap, we propose a benchmark based on a hierarchical design philosophy to evaluate the recognition capabilities of VLLMs in training-free scenarios. Through in-depth evaluations, we find that low-quality image input is a significant bottleneck in the recognition process. Drawing inspiration from this, we propose the Neighbor-Guided Toolchain Reasoner (NGTR) framework, which is characterized by integrating diverse lightweight tools for visual operations aimed at mitigating issues with low-quality images. Specifically, we transfer a tool selection experience from a similar neighbor to the input and design a reflection module to supervise the tool invocation process. Extensive experiments on public datasets demonstrate that our approach significantly enhances the recognition capabilities of the vanilla VLLMs. We believe that the benchmark and framework could provide an alternative solution to table recognition.

Yitong Zhou, Mingyue Cheng, Qingyang Mao +3

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Artificial Intelligence Research Institute, iFLYTEK Co., Ltd

Table Structure Recognition VLM Evaluation Tool Use in VLMs

Ask PDF

Apr 22, 2026

V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization

We introduce V-tableR1, a process-supervised reinforcement learning framework that elicits rigorous, verifiable reasoning from multimodal large language models (MLLMs). Current MLLMs trained solely on final outcomes often treat visual reasoning as a black box, relying on superficial pattern matching rather than performing rigorous multi-step inference. While Reinforcement Learning with Verifiable Rewards could enforce transparent reasoning trajectories, extending it to visual domains remains severely hindered by the ambiguity of grounding abstract logic into continuous pixel space. We solve this by leveraging the deterministic grid structure of tables as an ideal visual testbed. V-tableR1 employs a specialized critic VLM to provide dense, step-level feedback on the explicit visual chain-of-thought generated by a policy VLM. To optimize this system, we propose Process-Guided Direct Alignment Policy Optimization (PGPO), a novel RL algorithm integrating process rewards, decoupled policy constraints, and length-aware dynamic sampling. Extensive evaluations demonstrate that V-tableR1 explicitly penalizes visual hallucinations and shortcut guessing. By fundamentally shifting multimodal inference from black-box pattern matching to verifiable logical derivation, V-tableR1 4B establishes state-of-the-art accuracy among open-source models on complex tabular benchmarks, outperforming models up to 18x its size and improving over its SFT baseline

Yubo Jiang, Yitong An, Xin Yang +7

School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China

Multimodal Reasoning Reinforcement Learning with Verifiable Rewards Multimodal CoT Reasoning Process Supervision Visually Grounded Reasoning RL for VLMs+1

Ask PDF

Oct 13, 2026

MERIT: Multimodal Evidence Reasoning over Image Tables

Table reasoning with language models typically assumes access to clean textual tables. In practice, tables are often serialized into text, which can corrupt cell contents, distort structural relationships, and discard visual cues, leaving the model to reason over incomplete evidence. In contrast, table images retain this information, but using it requires both accurate perception and reasoning grounded in the relevant cells. To address this, we propose MERIT (Multimodal Evidence Reasoning over Image Tables), a three-stage framework for learning to reason over table images, with serialized tables serving as optional auxiliary inputs (if provided). Our analysis identifies visual evidence acquisition as a bottleneck, motivating table-specific encoder adaptation through reconstruction, localization, and color-conditioned supervision. Building on the adapted representations, supervised fine-tuning teaches explicit evidence selection and reasoning using verified trajectories. Finally, reinforcement learning refines the policy with separate rewards for evidence correctness, calculation, final answers, and output format. By providing feedback on intermediate steps, the framework targets retrieval and computational errors that final-answer supervision alone does not distinguish. We evaluate the approach on table question answering and fact verification, analyze the contributions of visual inputs and training components, and assess generalization to additional datasets with different table structures and visual presentations.

Haoyang Jiang, Zhuotong Fan, Mingyang Yi +3

Renmin University of China · Independent Researcher · Beike Inc.

Table QA Automated Fact-Checking Visual Question Answering Tabular Representation Learning

Ask PDF

Abstract

Visual Question Answering

Explore similar work

Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization

MERIT: Multimodal Evidence Reasoning over Image Tables