LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Yuxiang Shen; Hailong Huang; Zhenkun Gao; Xueheng Li; Man Zhou; Chengjun Xie; Haoxuan Che; Xuanhua He

cs.CVFeb 26, 2026

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Authors: Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li, Man Zhou, Chengjun Xie, Haoxuan Che, Xuanhua He, +1 more

Organizations: the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China · the University of Science and Technology of China, Hefei 230026, China · the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China · The Hong Kong University of Science and Technology, Hong Kong, China

Abstract

Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately $4.0\times$ inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.

Figures & tables

Fig. 1: Illustration of LookWise: enabling MLLMs to know when and where to look for adaptive visual reasoning.

Fig. 2: Probability density distributions of initial average confidence on the AOKVQA and POPE datasets.

Fig. 3: Limitations of indiscriminate cropping. Left: Cropping a region containing the “SLOW DOWN” sign misleads the model to answer “slow down,” even though the question asks what action is prohibited and the correct answer is “speeding.” Right: Cropping the drink area adds no new visual information, but the model’s prediction remains unchanged while additional computation is introduced.

Fig. 4: Overview of the LookWise framework. The model evaluates token confidence to decide when to look more carefully. If uncertain, it extracts key textual targets and uses them as queries in cross-attention over visual tokens to determine where to crop. The selected local regions are then provided together with the global image to produce the final answer.

Fig. 5: Comparison of localization precision. The baseline model often suffers from single-target bias (A) and attention drift (C). In contrast, LookWise explicitly decouples multiple queried targets to ensure complete coverage (B) and accurately identifies the intended subject (D). The attention maps show that semantic-guided localization suppresses noisy activations and maintains precise spatial focus.

Model	Method	Training-free	AOKVQA	POPE	V ∗ Bench	HR-Bench 4K	HR-Bench 8K
LLaVA-v1.5-7B	Baseline ∗ [ 24 ]	✓	71.00	86.98	48.68	36.13	32.13
	DC 2 [ 20 ]	✓	-	-	57.60	-	39.50
	VisCrop [ 37 ]	✓	-	-	62.30	46.25	35.75
	MLLMs-Know ∗ [ 22 ]	✓	72.31	87.25	56.02	44.38	37.25
	ZoomEye [ 21 ]	✓	70.56	88.94	83.25	49.88	48.63
	Ours (LookWise)	✓	72.90	87.37	62.80	47.38	42.00

TABLE I: Comparison with state-of-the-art methods. Highest results are marked in darker colors ( Blue for LLaVA, Red for Qwen). The gray rows show the absolute improvement (

\Delta

) of Ours over the Baseline . ∗ Results are reproduced by us using official implementations.

Fig. 6: Comparison of total GPU inference time (in minutes) across different methods (measured on NVIDIA H200 GPUs).

Decision Mechanism	AOKVQA	POPE	V*-Bench	HR 4K	HR 8K
LLM-Prompting	16m45s	1h33m	1h29m	1h47m	3h06m
Ours (Confidence)	12m33s	1h08m	1h22m	1h36m	3h01m

TABLE II: Total inference time comparison between LLM-Prompting and our Token Confidence decision mechanisms (measured on NVIDIA H200 GPUs).

Model	AOKVQA	V*-Bench
Qwen2.5VL-3B
w/o confidence-based decision	71.96%	85.86%
w/ confidence-based decision	73.10%	86.38%
$\Delta$	+1.14%	+0.52%

TABLE III: Quantitative evaluation of confidence-based decision module.

Benchmark	Acc	Time	Crop%
AOKVQA
w/o confidence-based decision	71.96	26m24s	100%
w/ confidence-based decision	73.10	12m33s	32.40%
$\Delta$	+1.14	-13m51s	-67.60%
V*-Bench
w/o confidence-based decision	85.86	1h04m	100%

TABLE IV: Performance and efficiency comparison (measured on NVIDIA H200 GPUs).

Model & Strategy	AOKVQA	POPE	V*-Bench	HR 4K	HR 8K
LLaVA-v1.5-7B
w/o Semantic-guided Localization	72.31%	87.25%	56.02%	44.38%	37.25%
w/ Semantic-guided Localization	72.90%	87.37%	62.80%	47.38%	42.00%
$\Delta$	0.59%	0.12%	6.78%	3.00%	4.75%
Qwen2.5VL-3B
w/o Semantic-guided Localization	71.62%	89.12%	75.90%	66.36%	64.88%

TABLE V: Ablation on semantic-guided localization module.

Model	Baseline	Full Local Inspection	Confidence-based Decision ( $\lambda$ )			Fixed Thresh.
Model	(Original)	(All Inspect)	1.0	1.5	2.0	( $\tau=0.92$ )
Threshold ( $\tau$ )	-	-	0.830	0.734	0.533	0.92
Qwen2.5-VL	71.44	71.96	72.58	73.10	72.49	72.05

TABLE VI: Sensitivity Analysis of the confidence-based decision module.

Dataset	Base	random		continuous
Dataset	Base	0.80	0.90	0.94	0.96	0.98	1.00
HR-4K	67.50	72.75	72.50	72.75	72.50	72.50	72.50
HR-8K	58.88	68.25	69.88	70.00	70.00	70.00	70.00
V*-Bench	75.90	81.15	84.81	86.38	86.38	85.86	85.86
AOKVQA	71.44	72.31	72.40	72.23	72.23	72.14	71.97

TABLE VII: Sensitivity analysis across multiple benchmarks under varying confidence thresholds

\tau

Benchmark	Baseline	$\tau=0.96$	Extraction Reduced
AOKVQA	71.44	72.23	10.39%
V*-Bench	75.90	86.38	14.13%
HR-Bench 8K	58.88	70.00	8.00%
HR-Bench 4K	67.50	72.50	10.00%
Average $\Delta$	–	+9.35%	10.63%

TABLE VIII: Evaluation under a fixed threshold (

\tau=0.96

)

Fig. 7: Qualitative examples of adaptive visual reasoning. Left: For small object perception, LookWise precisely targets the “Bag” to identify its purple color despite a complex background. Right: For fine-grained attribute recognition, the model isolates the “Blue Sail” to reveal the subtle spider logo.

Fig. 8: Spatial Reasoning with Multiple Targets. The model attends to both the "Baby stroller" and the "Person in orange," extracting a region that preserves their relative spatial position.

Explore similar work

Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding
Jun 15, 2026Yifan Wang, Peiming Li, Shiyu Li +5Efficient VLM Inference Large Vision-Language Models
1Tencent BAC · School of Electronic and Computer Engineering, Peking University · School of Mathematics and Statistics, University of Glasgow +1
LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
Jun 15, 2026Zhou Tao, Fang Zhang, Zewen Ding +5Multimodal Large Language Models Vision-Language Grounding
University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence
Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
Oct 1, 2025Sanghwan Kim, Rui Xiao, Stephan Alaniz +2Multimodal Large Language Models Long-Video Understanding
1Technical University of Munich · 2Munich Center for Machine Learning · 20263Helmholtz Munich +2

Jun 15, 2026J/K move · Enter open · S save

Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding

Yifan Wang, Peiming Li, Shiyu Li, Zhiyuan Hu+4

1Tencent BAC · School of Electronic and Computer Engineering, Peking University · School of Mathematics and Statistics, University of Glasgow+1

Efficient VLM Inference Large Vision-Language Models Vision-Language Grounding Visually Grounded Reasoning

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) excel in cross-modal reasoning, they often struggle to perceive fine-grained details in complex high-resolution images. Recent training-free methods address this through image scaling and localized cropping. However, applying these manipulations indiscriminately introduces computational redundancy for simple queries and can degrade accuracy by truncating essential global context or introducing irrelevant background noise. To this end, we propose LazyMCoT, a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty. The framework features an Adaptive Routing mechanism that evaluates predictive uncertainty using first-token statistics from a single forward pass. This efficiently bypasses confident cases while ensuring the recall of difficult samples via conformal calibration. For these challenging cases, a Collaborative Grounding module integrates the inherent cross-modal attention of the model with an external visual expert through a two-stage refinement process. This refinement process generates a precise localized display to recover small or occluded targets. Extensive experiments across diverse benchmarks demonstrate that LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency. Our code is availble at https://github.com/TencentBAC/LazyMCoT.

Abstract

Explore similar work

Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding

LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models

Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs