RECAP: Looking Once Is Not Enough for Vision-Language Reasoning

Zhaolu Kang; Tailong Luo; Chenxin Li; Yingjie He; Zhenyu Yu; Fengyu Zhou; Jiachen Qian; Lei Wei

cs.CVOct 22, 2026

RECAP: Looking Once Is Not Enough for Vision-Language Reasoning

Authors: Zhaolu Kang, Tailong Luo, Chenxin Li, Yingjie He, Zhenyu Yu, Fengyu Zhou, Jiachen Qian, Lei Wei,

Organizations: Peking University · Chengdu Minto Tech

Abstract

Long chain-of-thought reasoning improves vision-language models (VLMs), but it also exposes a temporal grounding failure: as generation unfolds, models may drift from image evidence while reinforcement learning still provides only a final-answer reward. We study this gap through visual revisit, a temporally localized reactivation of image evidence during reasoning. Across three VLM backbones, revisit peaks are predictive of correctness and causally linked to performance: masking them degrades accuracy more than masking non-peaks, while injecting revisit-like peaks into failed rollouts partially restores correct answers. Control analyses show that this signal is not explained by raw attention strength, uncertainty, logit margin, or hidden-state norm. We propose RECAP, a GRPO-compatible training method that turns detached revisit traces into a credit-assignment signal. RECAP uses revisit-conditioned advantage estimation, dynamic discounting, and gated value prediction to propagate final rewards through visually important reasoning steps, without extra rewards, additional rollouts, or inference-time changes. Across Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-OV-7B, RECAP consistently improves over GRPO and VPPO, yielding $+6.1$ -- $+6.5$ HallusionBench gains over the SFT base while preserving reasoning and text-only performance. Results on larger LoRA-tuned models, Chinese zero-shot benchmarks, visual perturbations, and human evaluation further demonstrate robust improvements in image-grounded reasoning.

Figures & tables

Panel of Figure 1 — Figure 1: Visual revisits are causal and localized. Correct rollouts show recurring CMA peaks, while incorrect rollouts lose visual grounding after the initial window. Masking revisit peaks hurts accuracy more than masking random non-peaks, and peak injection partially recovers failed rollouts. A small set of heads explains most of the causal effect.

Figure 2: The RECAP framework. A detached revisit signal $\tilde{r}_{t}$ is extracted from cross-modal attention and injected into three parts of the surrogate estimator: the trace-augmented input, dynamic discount, and revisit-gated value predictor. The rollout policy, clipped ratio, and KL anchor remain unchanged.

Method	Hallucination			Reasoning				Grounding			Text-only		$\Delta$
	Hall. $\uparrow$	POPE $\uparrow$	MMHal $\uparrow$	MathV. $\uparrow$	MMMU $\uparrow$	MathVr. $\uparrow$	MM-Vet $\uparrow$	GQA $\uparrow$	BLINK $\uparrow$	MMVP $\uparrow$	MMLU $\uparrow$	GSM8K $\uparrow$
Qwen2.5-VL-7B
SFT base	60.1	86.5	3.12	59.4	41.3	33.2	51.4	63.8	48.2	54.0	72.4	78.9	–
TVC (no RL)	61.4	86.9	3.18	59.2	41.0	33.5	51.8	64.1	48.5	54.3	72.3	78.7	$+1.3$
GRPO	62.8	87.1	3.27	$\underline{61.2}$	$\underline{42.5}$	$\underline{34.7}$	$\underline{52.8}$	64.3	$\underline{49.1}$	54.8	$\underline{72.1}$	$\underline{78.5}$	$+2.7$
VPPO	$\underline{64.2}$	$\underline{87.6}$	$\underline{3.35}$	60.8	41.9	34.1	52.1	$\underline{64.7}$	48.9	$\underline{55.1}$	71.8	77.9	$\underline{+4.1}$

Table 1: Main results across three VLM backbones. Values are mean benchmark scores or accuracies over

8

seeds; MMHalBench is reported on its native score scale. Per-seed logs, standard deviations, standard errors, and 95% confidence intervals are reported in the supplementary reproducibility logs. Bold marks the best value within each backbone block; underlines mark the strongest non-RECAP RL baseline per column. A dagger marks a significant RECAP-over-GRPO improvement after Holm–Bonferroni correction.

\Delta

denotes the HallusionBench gain over the per-backbone SFT base.

Configuration	Hallusion $\uparrow$	MathVista $\uparrow$	MMMU $\uparrow$	$\mathrm{Var}(A_{t})$ rel. $\downarrow$
(a) VPPO orthogonality
GRPO	62.8 [62.0, 63.5]	61.2	42.5	$1.81\times$
VPPO	64.2 [63.3, 65.0]	60.8	41.9	$1.35\times$
RECAP	$\underline{66.6}$ [65.7, 67.4]	$\underline{63.6}$	42.7	$1.00\times$
VPPO + RECAP (composition)	68.0 [67.1, 68.8]	63.9	$\underline{42.6}$	$0.98\times$
(b) Component ablations

Table 2: Ablations on Qwen2.5-VL-7B. Bold marks the best value within each panel and underlines mark the strongest non-best variant. The table compares VPPO composition, component removal, and dynamic-discount reweighting variants across accuracy and surrogate-advantage variance.

Configuration	Qwen2.5-VL 7B $\uparrow$	InternVL3 8B $\uparrow$	LLaVA-OV 7B $\uparrow$	Extra
(a) Signal identity
Token entropy	63.2	61.6	60.1	$\rho{=}0.29$
Logit margin	63.0	61.4	59.8	$\rho{=}0.24$
Hidden-state norm	$\underline{63.5}$	$\underline{61.9}$	$\underline{60.4}$	$\rho{=}0.36$
Revisit $\tilde{r}_{t}$ (ours)	66.6	64.7	63.4	$\rho{=}1.00$
(b) Same signal without RL

Table 3: Signal identity and no-RL controls. Bold marks the best value within each panel; underlines mark the strongest signal-replacement baseline in panel (a) and the strongest no-RL baseline in panel (b). Replacing

\tilde{r}_{t}

with surface token signals weakens RECAP, and using the revisit signal without RL does not recover the full gain.

Panel of Figure 3 — Figure 3: Causal and mechanistic evidence. Top revisit positions are more decision-relevant than random positions, and revisit peaks align with annotated decision steps. The revisit signal further concentrates in a small set of heads, and zeroing those heads reproduces most of the accuracy drop from masking cross-modal attention.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Epochs	Peak LR	Batch (tokens)	Max seq. len.	GPU-days
Qwen2.5-VL-7B	3	$2{\times}10^{-5}$	$2{\times}10^{6}$	4096	12
InternVL3-8B	3	$1{\times}10^{-5}$	$2{\times}10^{6}$	4096	14
LLaVA-OneVision-7B	3	$2{\times}10^{-5}$	$2{\times}10^{6}$	4096	12
Qwen2.5-VL-72B (LoRA)	2	$1{\times}10^{-4}$	$1{\times}10^{6}$	4096	60

Appendix

Table 4: Stage 1 SFT hyperparameters. Per-backbone supervised fine-tuning settings used before Stage 2 RL.

Backbone	$\gamma_{\mathrm{base}}$	$\gamma_{\max}$	$\beta$	$\eta$	history $K$
Qwen2.5-VL-7B	0.97	0.99	0.6	6	8
InternVL3-8B	0.96	0.99	0.5	6	8
LLaVA-OneVision-7B	0.97	0.99	0.7	6	8
Qwen2.5-VL-72B	0.97	0.99	0.5	6	8

Appendix

Table 5: RECAP Stage 2 RL hyperparameters. Dynamic-discount, revisit-normalization, and history-window settings used for each backbone.

Method	Hall.	POPE	MathV.	MMMU	MathVr.	MMLU	GSM8K
SFT base (72B)	67.4	89.1	68.2	52.9	44.8	81.6	88.4
LoRA-GRPO	69.8	89.6	69.6	53.4	45.7	81.4	88.1
LoRA-VPPO	70.2	90.0	69.0	52.9	45.1	80.9	87.5
LoRA-RECAP	72.9	90.7	71.5	53.7	46.8	81.5	88.2
$\Delta$ vs. LoRA-GRPO	$+3.1$	$+1.1$	$+1.9$	$+0.3$	$+1.1$	$+0.1$	$+0.1$

Appendix

Table 6: Qwen2.5-VL-72B LoRA results under the same rank-64 LoRA setting and RL budget. Bold indicates significant improvement over LoRA-GRPO.

The default configuration attains the best per-axis accuracy on all four axes ( $66.6$ pt);
$\eta$	Hall.	$\mathcal{I}$	Hall.	$W$	Hall.	head-pool	Hall.
3	65.9	lower third	64.1	8	65.6	mean (def.)	66.6
6 (def.)	66.6	middle third (def.)	66.6	16 (def.)	66.6	max	65.8
9	66.3	upper third	64.8	32	66.2	top-4	66.5
12	65.4	all layers	63.9	64	65.8	top-8	66.4
this is the same checkpoint entered in four marginals of the grid, not four independently-optimized runs.

Appendix

Table 7: HallusionBench accuracy under marginal sweeps of the revisit-signal pipeline, with the max–min spread over the full hyperparameter grid summarized below. The default setting is robust within

\pm 0.6

pt among nearby reasonable configurations, while the full grid has a 3.1 pt max–min spread. The main source of degradation is all-layer pooling, where top-layer label-projection noise dilutes the revisit estimate.

Panel of Figure 4 — Figure 4: $\mathrm{RAP}$ rises early under RECAP training. RECAP reaches a higher $\mathrm{RAP}$ plateau than VPPO, GRPO, and the SFT reference, and the plateau appears before final accuracy convergence.

Panel of Figure 5 — Figure 5: $\mathrm{RAP}$ correlates with HallusionBench accuracy on checkpoints from every method, not just RECAP. Pearson $\rho\geq 0.38$ on all four methods with $95\%$ bootstrap CIs strictly above zero ( $\geq 48$ checkpoints per method pooled over three backbones), establishing $\mathrm{RAP}$ as a method-agnostic diagnostic. The correlation is highest on RECAP (as expected, since RECAP explicitly optimizes revisit-aware return propagation), but substantial non-RECAP correlations confirm $\mathrm{RAP}$ is not a restatement of the training objective.

Method	Qwen2.5-VL-7B	InternVL3-8B	LLaVA-OneVision-7B	Pooled
GRPO	0.44	0.39	0.43	0.42
VPPO	0.49	0.44	0.48	0.47
TVC (no RL)	0.36	0.41	0.37	0.38
RECAP	0.63	0.58	0.62	0.61

Appendix

Table 8: Cross-method RAP–accuracy correlations. Pearson correlations between

\mathrm{RAP}

and HallusionBench accuracy across checkpoints, reported per backbone and pooled across backbones.

Benchmark	Spearman $\rho(\mathrm{RAP},\text{accuracy})$
HallusionBench	0.61
MathVista	0.34
POPE	0.55
MMMU	0.21
MathVerse	0.30

Appendix

Table 9: Per-benchmark RAP correlations. Spearman correlations between

\mathrm{RAP}

and accuracy on Qwen2.5-VL-7B checkpoints pooled across methods.

Panel of Figure 6 — Figure 6: RECAP training consolidates revisit responsibility onto fewer heads. The top- $10$ head share of the total revisit signal rises from $34\%$ (SFT base) to $54\%$ after RECAP training, while GRPO on identical data produces only a $+4$ pt shift ( $\pm 3$ bootstrap CI). Concentration is therefore specific to revisit-aware optimization rather than to RL post-training in general. Similar pattern on the other two backbones ( section G.3 ).

	Top- $10$ revisit heads		Random $10$ heads
Backbone	AUC	F1	AUC	F1
Qwen2.5-VL-7B	$0.68$	$0.63$	$0.54$	$0.50$
InternVL3-8B	$0.66$	$0.61$	$0.53$	$0.49$
LLaVA-OV-7B	$0.67$	$0.62$	$0.55$	$0.51$
Pooled mean	0.67	0.62	$0.54$	$0.50$

Appendix

Table 10: Linear probing on the top-revisit heads. Probes trained on the top-

10

revisit heads predict near-future answer correctness better than probes trained on matched random heads, supporting the causal localization of revisit behavior.

Backbone	Top-10 heads $(\ell,h)$
Qwen2.5-VL-7B	(14,7), (15,3), (13,11), (16,5), (14,19), (15,22), (13,4), (17,8), (14,12), (16,21)
InternVL3-8B	(12,5), (13,2), (14,9), (12,11), (13,14), (15,3), (11,7), (14,1), (13,6), (15,12)
LLaVA-OneVision-7B	(13,9), (14,3), (12,15), (15,7), (13,21), (14,18), (12,4), (16,11), (13,2), (15,25)

Appendix

Table 11: Top revisit heads. Top-10 head coordinates for each backbone, sorted by revisit contribution.

$k$	SFT base	GRPO	RECAP
5	$21\%$	$25\%$	$36\%$
10	$34\%$	$38\%$	$54\%$
20	$51\%$	$53\%$	$72\%$
50	$77\%$	$78\%$	$89\%$

Appendix

Table 12: Pre/post-training head concentration data. Top-

k

share of the revisit signal before RL, after GRPO, and after RECAP.

Backbone	Median normalized layer	Median normalized head
Qwen2.5-VL-7B	0.46	0.27
InternVL3-8B	0.50	0.31
LLaVA-OneVision-7B	0.43	0.24

Appendix

Table 13: Cross-backbone head-position alignment. Median normalized layer and head coordinates for the top revisit heads on each backbone.

	Method score			Agreement
Axis	GRPO	VPPO	RECAP	statistic
Image-groundedness win rate (%)	14.2	23.7	62.1	$\kappa=0.55$
Coherence win rate (%)	29.4	30.1	40.5	$\kappa=0.48$
Trustworthiness mean ( $1$ – $5$ )	3.18	3.34	3.87	$\alpha=0.51$
All axes (McNemar $p$ vs. RECAP)	$<10^{-3}$	$<10^{-2}$	–	–
Inter-rater agreement with LLM judges	–	–	$72\%$	–

Appendix

Table 14: Human study. Ten trained annotators rank

100

anonymized samples on image-groundedness, coherence, and trustworthiness. RECAP achieves the highest image-groundedness win rate, with significant gains over both baselines and

72\%

agreement with the independent LLM-as-judge ranking.

Method	GPT-4V	Claude	Qwen-Max	judge-mean
GRPO	$3.21$	$3.28$	$3.14$	$3.21$
VPPO	$3.48$	$3.51$	$3.43$	$3.47$
TVC (no RL)	$3.29$	$3.35$	$3.22$	$3.29$
RECAP	4.12	4.08	4.05	4.08
Inter-judge Krippendorff’s $\alpha$	$0.63$			–
Agreement with human ( table 14 )	$72\%$			–

Appendix

Table 15: LLM-as-judge rankings. Three independent judges score anonymized CoTs for image-grounded reasoning. All judges rank RECAP highest, and their ranking agrees with the human study on most items.

Backbone	Method	CMMMU	MMMU	MathVista-CN	mean $\Delta$
Qwen2.5-VL-7B	GRPO	46.3	39.4	55.1	–
	VPPO	46.8	40.0	55.6	$+0.5$
	RECAP	49.1	42.2	57.3	$\mathbf{+2.6}$
InternVL3-8B	GRPO	43.8	37.2	52.7	–
	VPPO	44.1	37.6	53.0	$+0.3$
	RECAP	46.2	39.5	54.8	$\mathbf{+2.3}$

Appendix

Table 16: Cross-lingual generalization. RECAP is trained on English data only and evaluated zero-shot on three Chinese benchmarks. It yields consistent gains across backbones, suggesting transfer beyond the English training distribution.

Panel of Figure 7 — Figure 7: RECAP is consistently more robust to image-side perturbations on Qwen2.5-VL-7B. Across four perturbation families, RECAP has smaller relative accuracy drops than VPPO and maintains a clear advantage over GRPO under stronger perturbations.

Method	Gauss $\sigma{=}0.2$	JPEG $Q{=}10$	Patch $p{=}0.3$	Distractor
GRPO	53.4	53.9	52.1	58.9
VPPO	54.0	54.6	53.2	60.0
RECAP	58.5	57.0	59.0	63.2

Appendix

Table 17: Accuracy under the strongest image perturbations. HallusionBench accuracy on Qwen2.5-VL-7B for the strongest setting in each perturbation family.

	Qwen2.5-VL-7B		InternVL3-8B		LLaVA-OV-7B
Method	mean	std	mean	std	mean	std
SFT + TVC (no RL)	60.9	$3.54$	59.5	$3.31$	58.1	$3.70$
GRPO	62.2	$2.81$	60.7	$2.64$	58.8	$2.92$
VPPO	63.4	$2.58$	61.8	$2.47$	59.9	$2.73$
RECAP	66.0	0.84	64.2	0.91	62.9	0.97

Appendix

Table 18: Robustness to text-side question reformulation. RECAP remains stable across five meaning-preserving reformulations of each HallusionBench question, with lower accuracy variance than GRPO, VPPO, and TVC.

	Estimator context	Discount	Value	Reward	Advantage
GRPO	–	static	single	external	raw $A_{t}$
VPPO	–	static	single	external	$\tilde{r}_{t}\cdot A_{t}$ (reweight)
TVC	–	static	single	external	raw $A_{t}$
$\lambda$ -GRPO	–	static	single	external	$w_{t}\cdot A_{t}$ (learned)
TreeAdv	–	static	single	external	tree- $A_{t}$
RECAP	post-hoc	$\boldsymbol{f(\tilde{r}_{t})}$	gated	external	no local reweight

Appendix

Table 19: Where each method places its per-token visual signal, broken down by the five structural slots in the GRPO update. Only RECAP uses revisit at all three of post-hoc estimator context, discount, and value; every baseline touches exactly one slot (typically the advantage) and leaves the others at their GRPO defaults. This is the structural statement behind the expressiveness gap of Proposition 1 : because dynamic discount acts multiplicatively across future tokens, no single-slot advantage reweighting (VPPO,

\lambda

-GRPO) can realize its effect, regardless of how

w_{t}

is chosen. RECAP still forms the surrogate advantage

A_{t}^{\mathrm{VR}}

in Eq. ( 11 ); “no local reweight” means that it does not multiply an otherwise fixed advantage by a same-step visual weight.

Reasoning for VLMs

See all 55 papers

Improving Visual Reasoning with Iterative Evidence Refinement
Mar 14, 2026Zeru Shi, Kai Mei, Yihao Quan +2VLM Reasoning Visually Grounded Reasoning
Department of Computer Science Rutgers University
ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination
Aug 5, 2026Lei Peng, Shuai Lv, Wei HuVLM Reasoning Visually Grounded Reasoning
University of Science and Technology of China School of Artificial Intelligence and Data Science Hefei, Anhui, China · University of Science and Technology of China State Key Laboratory of Precision and Intelligent Chemistry Hefei, Anhui, China
ReGuLaR: Relation-Grounded Latent Reasoning for Large Vision-Language Models
May 28, 2026Zihu Wang, Karthik Somayaji N. S, Peng LiLatent Visual Reasoning Large Vision-Language Models
University of California, Santa Barbara

Mar 14, 2026J/K move · Enter open · S save

Improving Visual Reasoning with Iterative Evidence Refinement

Zeru Shi, Kai Mei, Yihao Quan, Dimitris N. Metaxas+1

Department of Computer Science Rutgers University

VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Vision language models (VLMs) are increasingly capable of reasoning over images, but robust visual reasoning often requires re-grounding intermediate steps in the underlying visual evidence. Recent approaches typically rely on external image operations such as zooming or cropping to re-access fine-grained details during inference, which requires additional image re-encoding and can disrupt the reasoning trajectory. We argue that VLMs already provide strong internal signals for identifying and reusing visual evidence, and that these signals can be directly leveraged to support image-grounded reasoning. Motivated by this insight, we propose an end-to-end self-revisit framework, SIEVE, that trains models to re-engage image evidence through internal representations. SIEVE automatically extracts embeddings of salient image regions and injects them into the reasoning chain when additional grounding is needed, enabling later steps to condition on relevant visual cues without external tool calls or re-encoding. We use reinforcement learning to teach the model when to trigger visual revisiting and which region embeddings to retrieve and insert during the reasoning process. Experiments on multiple visual reasoning benchmarks, together with perception, reasoning, and hallucination evaluations, show that SIEVE yields consistent gains, improving performance by 8 percent on average across several benchmarks.

Abstract

Reasoning for VLMs

Explore similar work

Improving Visual Reasoning with Iterative Evidence Refinement

ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination

ReGuLaR: Relation-Grounded Latent Reasoning for Large Vision-Language Models