VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Models

Soumya Suvra Ghosal; Youngeun Kim; Zhuowei Li; Ritwick Chaudhry; Linghan Xu; Hongjing Zhang; Jakub Zablocki; Yifan Xing

cs.CVFeb 27, 2026

VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Models

Authors: Soumya Suvra Ghosal, Youngeun Kim, Zhuowei Li, Ritwick Chaudhry, Linghan Xu, Hongjing Zhang, Jakub Zablocki, Yifan Xing,

Organizations: University of Maryland, College Park · Amazon · Physion Labs

Abstract

Advances in large reasoning models have shown strong performance on complex reasoning tasks by scaling test-time compute through extended reasoning. However, recent studies observe that in vision-dependent tasks, extended textual reasoning at inference time can degrade performance as models progressively lose attention to visual tokens and increasingly rely on textual priors alone. To address this, prior works use reinforcement learning (RL)-based fine-tuning to route visual tokens or employ refocusing mechanisms during reasoning. While effective, these methods are computationally expensive, requiring large-scale data generation and policy optimization. To leverage the benefits of test-time compute without additional RL fine-tuning, we propose VisRef, a visually grounded test-time scaling framework. Our key idea is to actively guide the reasoning process by re-injecting a coreset of visual tokens that are semantically relevant to the reasoning context while remaining diverse and globally representative of the image, enabling more grounded multi-modal reasoning. Experiments on three visual reasoning benchmarks with state-of-the-art multi-modal large reasoning models demonstrate that, under fixed test-time compute budgets, VisRef consistently outperforms existing test-time scaling approaches by up to 6.4%.

Figures & tables

Figure 1: Illustration comparing our approach with prior test-time scaling methods. (a) Textual self-reflection-based test-time scaling [ 34 , 2 ] extends reasoning by encouraging the model to think longer, but progressively loses grounding in the visual input as the reasoning chain grows. (b) Training-free Visual Refocusing (ours) dynamically selects and re-injects reasoning-relevant visual cues during inference, effectively restoring visual grounding without retraining and yielding substantial accuracy gains on the MathVista and MM-Star benchmarks with InternVL3.5-8B.

Figure 2: Overview of VisRef . Given an image-text input, VisRef enables multi-modal large reasoning models (MLRMs) to maintain visual grounding throughout the reasoning process without retraining. At each reasoning step, the model projects visual tokens into the textual reasoning subspace and selects a subset of reasoning-relevant tokens via a determinantal point process (DPP)-based criterion. The selected tokens are then reinjected to guide subsequent reasoning. This iterative process continues until the entropy of the model’s answer distribution falls below a confidence threshold $\delta_{\mathrm{entropy}}$ , forming an adaptive stopping criterion. The bottom-left plot shows the visual-to-text attention ratio across reasoning steps. As shown, our method maintains a higher level of visual attention ( green curve) by reinjecting a carefully selected coreset of visual tokens, whereas text-based self-reflection ( red curve) exhibits a more rapid decline in visual attention, consistent with prior observations [ 51 , 9 ] . Example shown uses InternVL-3.5-8B on MathVista.

Model	Method	MathVision	MathVista	MM-Star
InternVL3.5-8B	ST (Baseline)	39.2	68.1	57.2
	TSR [ 34 ]	40.1	73.9	58.3
	VisRef (Ours)	44.6 ( +4.5 )	79.3 ( +5.4 )	63.1 ( +4.8 )
Qwen3-VL-8B	ST (Baseline)	53.8	74.1	66.5
	TSR [ 34 ]	54.3	74.2	65.9
	VisRef (Ours)	56.6 ( +2.3 )	77.1 ( +2.9 )	69.1 ( +3.2 )

Table 1: Evaluation on visual reasoning benchmarks. We evaluate VisRef across three visual reasoning benchmarks. To ensure a fair comparison, all methods adopt the adaptive stopping criterion described in Section 4.1.2 . For brevity, we denote Standard Thinking as ST , and Textual Self-Reflection [ 34 ] as TSR . All results are reported in accuracy (%), and the numbers in parentheses indicate the performance gain over the ST baseline.

Figure 3: Test-time scaling of VisRef . We evaluate the test-time scaling behavior of VisRef by generating multiple parallel visual-integrated reasoning chains under a fixed token budget. Results are shown across three benchmarks (MathVision, MathVista, and MM-Star) and three MLRMs: InternVL-3.5-8B (first row), Qwen-3-VL-8B (second row), and SAIL-VL2 (third row). The star marker (✩) denotes standard thinking—the baseline with no additional test-time compute. Parallel thinking [ 19 , 46 ] generates multiple parallel chains-of-thought without visual refocusing. Across all models and benchmarks, VisRef consistently achieves superior accuracy for any given computational budget.

Figure 4: Ablation study of hyper-parameters. (a) We visualize the ablation results of the entropy threshold

\delta_{\mathrm{entropy}}

used as the stopping criterion. Although the accuracy does not vary significantly across different thresholds, our evaluation shows that

\delta_{\mathrm{entropy}}=0.25

achieves the best balance between accuracy and inference efficiency. (b) We show the ablation of the token budget

m

(fraction of visual tokens selected) on MathVista using InternVL-3.5-8B. Accuracy improves from 76.1% to 79.2% as

m

increases from 20% to 30% but plateaus for

m\geq 30\%

Figure 5: Attention Visualization. Attention maps show how VisRef progressively refocuses on relevant visual regions during multi-step reasoning. Initially, the attention maps are noisy. With visual reinjection, VisRef reinforces grounding on task-critical objects, leading to more accurate visual reasoning.

Method	MathVista [ 32 ]	MathVison [ 43 ]	MM-Star [ 8 ]
ST	68.1	39.2	57.2
Look-Back [ 51 ]	80.8	44.2	63.7
VisRef	79.3	44.6	63.1
Look-Back [ 51 ] + VisRef	83.1	48.2	66.0

Table 2: Comparison with training-based method ( i.e . , Look-Back [ 51 ] ) across benchmarks using InternVL-3.5-8B.

Relevance	Diversity	MathVista [ 32 ]	MathVison [ 43 ]	MM-Star [ 8 ]
✓	✗	75.6	43.3	61.0
✗	✓	77.4	42.9	62.8
✓	✓	79.3	44.6	63.1

Table 3: Importance of relevance and diversity. Ablation study to understand the individual importance of the relevance term and diversity term in our scoring function (Equation 10 ). We evaluate on InternVL-3.5-8B across all three benchmarks. Accuracy values are reported in percentage (%). Best results are highlighted in bold .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	TallyQA	RealWorldQA
InternVL3.5-8B	ST (Baseline)	79.4	44.6
	TSR [ 34 ]	79.6	44.9
	VisRef (Ours)	84.5 ( +5.1 )	47.2 ( +2.6 )
Qwen3-VL-8B	ST (Baseline)	74.3	55.4
	TSR [ 34 ]	75.1	56.9
	VisRef (Ours)	78.9 ( +4.6 )	59.1 ( +3.7 )

Appendix

Table 4: Evaluation on additional visual reasoning benchmarks. We evaluate VisRef across three visual reasoning benchmarks: TallyQA, and RealQA. To ensure a fair comparison, all methods adopt the adaptive stopping criterion described in Section 4.1.2. For brevity, we denote Standard Thinking as ST , and Textual Self-Reflection [ 34 ] as TSR . All results are reported in accuracy (%), and the numbers in parentheses indicate the performance gain over the ST baseline.

Figure 6: Test-time scaling of VisRef . We evaluate the test-time scaling behavior of VisRef by generating multiple parallel visual-integrated reasoning chains under a fixed token budget. Results are shown across two benchmarks (TallyQA, and RealWorldQA) and three MLRMs: InternVL-3.5-8B (first row), Qwen-3-VL-8B (second row), and SAIL-VL2 (third row). The star marker (✩) denotes standard thinking—the baseline with no additional test-time compute. Parallel thinking [ 19 , 46 ] generates multiple parallel chains-of-thought without visual refocusing. Across all models and benchmarks, VisRef consistently achieves superior accuracy for any given computational budget.

Figure 7: Attention Visualization. Attention maps show how VisRef progressively refocuses on relevant visual regions during multi-step reasoning. Initially, the attention maps are noisy. With visual reinjection, VisRef reinforces grounding on task-critical objects, leading to more accurate visual reasoning.

Method	Time
ST	7.1s
TSR	7.7s
Look-Back [1]	7.6s
VisRef	8.2s

Appendix

Table 5: Latency per prompt on MathVista.

Model	ST	TSR	VisRef
InternVL-1B	46.1	48.5	52.0
InternVL-2B	52.9	53.7	58.1
InternVL-8B	68.1	73.9	79.3

Appendix

Table 6: Accuracy (%) across model scales on MathVista.

Selection	MVista	MVision	MM-Star
Random	67.3	40.8	57.3
Relevance-only	75.6	43.3	61.0
DPP (Ours)	79.3	44.6	63.1

Appendix

Table 7: Token selection strategies (InternVL-8B).

$\lambda$	0.0 (Div)	0.25	0.5 (Ours)	0.75	1.0 (Rel)
MVista	71.2	76.8	79.3	77.4	75.6
MVision	41.5	43.1	44.6	44.2	43.3

Appendix

Table 8: Effect of

\lambda

weighting (InternVL-8B).

Test-Time Scaling for VLMs

Test-time scaling for vision-language models studies how extra inference-time computation can improve visual understanding and reasoning.

See all 23 papers

Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning
Sep 28, 2026Hao-Xuan Ma, Yihao Liu, Yutao Sun +8Test-Time Scaling for VLMs Test-Time Scaling
Qwen Business Unit of Alibaba · School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University +5
Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach
Aug 4, 2026Tianbao Jiang, Weicong Ni, Gerard de Melo +1Large Vision-Language Models VLM Reasoning
East China Normal University Shanghai, China · Hasso Plattner Institute Potsdam, Germany · University of Potsdam Potsdam, Germany +1
On Test-Time Scaling for Vision-Language Models
Jun 27, 2026Fawaz Sammani, Tzoulio Chamiti, Nikos DeligiannisVLM Evaluation Test-Time Scaling for VLMs
ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium

Sep 28, 2026J/K move · Enter open · S save

Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning

Hao-Xuan Ma, Yihao Liu, Yutao Sun, Yanting Miao+7

Qwen Business Unit of Alibaba · School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University+5

Test-Time Scaling for VLMs Test-Time Scaling VLM Reasoning

Open paper AskPDF arXiv

Latent test-time scaling improves reasoning by refining hidden states during inference, but existing methods typically apply a single scalar reward to all editable latent tokens. For multimodal large language models, this global update ignores that generated tokens play different roles: some are sensitive to visual evidence, while others correspond to uncertain reasoning decisions. We present Token-Disentangled Latent Test-Time Scaling, an inference-time framework that makes latent refinement token-role-aware. Starting from an initial generated trajectory, we optimize a short hidden-state prefix while routing perception-side visual feedback to image-sensitive tokens and reasoning feedback to high-entropy tokens. Tokens selected by neither route are constrained by an anchor regularizer. Across both perception and reasoning benchmarks on Qwen2.5-VL-7B and InternVL3.5-8B, our method lifts macro accuracy over CoT by +2.57 and +1.51 respectively, and outperforms strong output-space test-time scaling baselines under matched decoded-candidate budgets. Code is available at https://github.com/Qwen-Applications/TD-LTTS.

Abstract

Test-Time Scaling for VLMs

Explore similar work

Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning

Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

On Test-Time Scaling for Vision-Language Models