From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognition
Authors: Yutong Yang, Zhuqi Liu, Mouxing Yang, Xi Peng
Organizations: College of Computer Science, Sichuan University, Chengdu 610065, China · School of Artificial Intelligence, Sichuan University, Chengdu 610065, China, and also with Tianfu Jincheng Laboratory, Chengdu 610093, China
Multimodal Large Language Models (MLLMs) generally fall short of expectations in Fine-Grained Visual Recognition (FGVR), although they exhibit promising FGVR foundations, i.e., visual perception and broad category knowledge. Most existing studies seek to remedy this deficiency by adopting domain-specific post-training, at the risk of compromising generalization to unseen domains. Different from these efforts, in this paper, we revisit the FGVR mechanism underlying MLLMs from both the empirical and theoretical perspectives and uncover an inference-time phenomenon termed FGVR Laziness. In brief, FGVR Laziness refers to the insufficient association between visual cues and category concepts under certain candidate category lists and recognition guidance, leaving the FGVR foundations of MLLMs largely under-utilized. To mitigate FGVR Laziness, we propose Laziness-resIstant Visual rEcognition (LIVE), a training-free framework which jointly refines candidate lists and recognition guidance to elicit the latent FGVR capability of MLLMs. Specifically, LIVE first performs query-adaptive candidate inference to derive a compact yet reliable candidate list. Meanwhile, LIVE constructs category-specific FGVR skills by mining the intra-category characteristics and inter-category differences. Finally, LIVE selectively invokes detailed skill guidance according to recognition uncertainty and category confusion, shifting MLLM-based FGVR from a cursory glance to a focus on associating discriminative visual cues with category concepts. Extensive experiments on 13 benchmarks against 24 state-of-the-art methods spanning fine-grained category recognition, action recognition, and general category recognition, verify the effectiveness and generalizability of LIVE across different MLLM backbones. The code will be released upon acceptance.
Figures & tables
Fig. 1: Our motivations and observations. (a) FGVR Potential . MLLMs can describe fine-grained visual cues and category-specific characteristics, demonstrating promising perceptual and conceptual capabilities required for FGVR. (b) FGVR Laziness . FGVR Laziness refers to the inference-time phenomenon where MLLMs might fail to consistently inspect the query image with respect to all plausible candidate categories. To investigate this phenomenon, we conduct the following two kinds of controlled experiments. i) Position variation: we move the ground-truth category forward in the candidate list while keeping the other candidates unchanged (①). ii) Candidate-competition variation: we keep the ground-truth category and the original mispredicted category unchanged, while either replacing the remaining candidates with irrelevant categories (②) or removing them altogether (③). Furthermore, we introduce explicit inter-category comparison guidance to distinguish the ground-truth category from the original mispredicted category (④). (c) Statistics . Quantitative results show that moving the ground-truth category forward, reducing candidate competition, and introducing comparison guidance generally increase category-to-image attention and improve recognition. In this figure, Recovery Rate denotes the proportion of initially incorrect predictions that are corrected under each setting. The attention score measures category-to-image attention averaged over ground-truth category tokens. Qwen2.5-VL-7B serves as the default MLLM, while GPT-5.4 is employed to evaluate the category-specific descriptions generated by the MLLM.
Fig. 2: Overview of the proposed LIVE. Based on the empirical observations and theoretical analyses, LIVE aims to mitigate FGVR Laziness by reducing candidate competition and providing adaptive recognition guidance. Specifically, for a given query image, Plausible Candidate Inference (PCI) first combines query-specific visual relevance with a pre-constructed inter-category affinity graph to infer a compact candidate list with reliable ground-truth coverage. Meanwhile, Category Skill Construction (CSC) leverages MLLM self-reflection to construct reusable Category Skills, consisting of lightweight headers and detailed visual and comparative guidance. Based on the inferred candidates and Category Skills, Skill-inspired Confusion Resolution (SCR) performs initial recognition with skill headers and estimates prediction uncertainty from the output-token distribution. For uncertain predictions, SCR further identifies confused categories through first-token prefix matching and adaptively invokes the corresponding detailed guidance for skill-enhanced recognition. In the figure, for illustration, we present the LIVE pipeline in the image-based FGVR setting using examples from the Flower dataset.
Method
Source
Flower
CUB
Pet
Aircraft
Cars
Dogs
DTD
Food
SUN
Avg.
VLM-based Methods
CLIP RN50
ICML 2021
61.18
46.08
83.59
15.60
53.88
54.56
38.72
78.62
58.71
54.55
CLIP ViT-B/16
ICML 2021
67.63
54.98
88.22
23.82
63.87
62.55
43.19
88.22
62.92
61.71
SigLIP
ICCV 2023
83.60
68.03
91.93
37.14
90.58
74.98
61.64
91.68
68.21
74.20
DCLIP
ICLR 2023
70.66
57.75
86.92
25.08
63.21
63.08
45.59
88.50
66.89
63.08
CuPL
ICCV 2023
73.31
56.77
91.20
27.93
66.12
66.78
54.26
88.85
68.99
66.02
TABLE I: Comparisons with representative methods on nine fine-grained image recognition benchmarks. Top-1 accuracy (%) is reported, with the best and second-best results marked in bold and underlined , respectively. “ +PCI ” denotes augmenting the corresponding post-trained MLLM with the CLIP-based PCI module. Results of the compared methods are taken from their original papers or reproduced using publicly available code.
Method
Source
UCF101
HMDB51
Avg.
General VLMs
CLIP RN50
ICML 2021
62.72
37.77
50.25
CLIP ViT-B/16
ICML 2021
67.88
47.25
57.57
SigLIP
ICCV 2023
74.86
46.14
60.50
SigLIP2
arXiv 2025
73.24
45.29
59.27
General MLLMs
TABLE II: Comparisons with representative methods on human action recognition. Top-1 accuracy (%) is reported on UCF101 and HMDB51 datasets, with the best results marked in bold .
Method
Source
ImageNet-1K
ImageNet-S
Avg.
General VLMs
CLIP RN50
ICML 2021
55.50
32.35
43.93
CLIP ViT-B/16
ICML 2021
63.84
44.98
54.41
SigLIP
ICCV 2023
71.58
64.80
68.19
SigLIP2
arXiv 2025
73.08
65.10
69.09
MLLM-based Methods
TABLE III: Comparisons with representative methods on general and OOD image recognition. Top-1 accuracy (%) is reported on ImageNet-1K and ImageNet-Sketch, abbreviated as ImageNet-S in the table, with the best results marked in bold . “ +PCI ” denotes augmenting the corresponding post-trained MLLM with the SigLIP-based PCI module.
Fig. 3: Ablation study of LIVE c on Flower and CUB datasets. Results are reported as Top-1 accuracy gains over the corresponding MLLM baseline. Here, Top-10 and Top-20 use CLIP-ranked candidates for recognition, while PCI uses the inferred plausible candidates. PCI+CSC further introduces skill headers, and PCI+CSC+SCR denotes the complete LIVEc .
Fig. 4: Scaling analysis of LIVE c across different Qwen2.5-VL model sizes on the Flower and CUB datasets.
Fig. 5: Quantitative analysis of PCI with different VLM backbones and reference-shot settings. The solid curves report the ground-truth coverage, while the annotated values denote the corresponding average size of the inferred candidate list Cx . The dashed lines report fixed Top- k baselines, with k chosen according to the lower and upper bounds of the candidate-list sizes produced by PCI.
Fig. 6: Analytic study on Category Skill components and invocation strategy. “Des.”, “Vis.”, and “Comp.” denote category description, visual checklist, and comparative information, respectively. “All” introduces all components simultaneously. “Ours” invokes detailed skill bodies on demand through SCR.
Fig. 7: Entropy distributions of correct and incorrect predictions. Higher entropy indicates greater prediction uncertainty.
Metric
Flower
CUB
Avg.
Entropy-based Uncertainty Identification
Low-entropy Acc. (%)
95.46
87.64
91.55
High-entropy Acc. (%)
58.59
37.40
48.00
High-entropy Ratio (%)
32.44
55.97
44.21
Detailed Skill Invocation for High-entropy Samples
Avg. Size of All Candidates
9.94
10.61
10.28
TABLE IV: Analytic study on SCR. The upper part evaluates entropy-based uncertainty identification, while the lower part compares detailed skill invocation for all candidates and confused categories on high-entropy samples.
Fig. 8: Parameter analysis of PCI on Flower and CUB. The three rows examine the ranking-reliability threshold γ , support threshold ρ , and seed number B , respectively. The blue and green curves report ground-truth coverage and average candidate size, while the dotted lines indicate the default parameter settings.
Fig. 9: Parameter analysis of the entropy threshold η in SCR on Flower and CUB. The left and right panels report the Top-1 accuracy gain over Stage-I recognition and the Stage-II trigger rate, respectively. The dotted lines indicate the default setting η=0.1 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 10: Empirical analysis of the Candidate Ordering Effect. For each dataset, we randomly sample 1,500 test images and independently shuffle the complete candidate list for each image. The blue curve shows the average category-to-image attention of the ground-truth category at each position, while the green curve shows the corresponding recognition accuracy.
Fig. 11: Visualization of FGVR Laziness on CUB-200. We show representative samples misclassified by Qwen2.5-VL-7B under full-category recognition but correctly recognized after retaining only the ground-truth category and its original incorrect prediction. Glance (red) and Focus (green) correspond to recognition under the complete and reduced candidate lists, respectively, with attention maps showing category-to-image attention for the ground-truth category. Compared with Glance, Focus generally exhibits stronger and more concentrated attention on the target bird.
Dataset
Classes
Test Samples
Modality
Oxford Flowers-102 [ 38 ]
102
6,149
Image
CUB-200-2011 [ 34 ]
200
5,794
Image
Oxford-IIIT Pet [ 39 ]
37
3,669
Image
FGVC-Aircraft [ 35 ]
100
3,333
Image
Stanford Cars [ 40 ]
196
8,041
Image
Stanford Dogs [ 41 ]
120
8,580
Image
Appendix
TABLE V: Statistics of the 13 benchmarks used in our experiments.
Fig. 12: Prompt for constructing category-specific visual knowledge in CSC.
Fig. 13: Prompt for constructing confusion-specific comparative knowledge in CSC.
Fig. 14: Prompt for Stage-I recognition in SCR, where [CATEGORY SKILL HEADERS] contains the category name and description of each candidate.
Fig. 15: Prompt for Stage-II recognition in SCR. [CATEGORY SKILLS] contains the description of each candidate, while the visual checklist and available comparative knowledge are additionally provided for the identified confused categories.
Fig. 16: A representative Category Skill constructed by CSC using Qwen2.5-VL-7B for wallflower . The skill combines category-specific visual knowledge with comparative knowledge for distinguishing wallflower from its confused category californian poppy .
Fig. 17: A representative Category Skill constructed by CSC using Qwen3-VL-8B for purple coneflower . The skill combines category-specific visual knowledge with comparative knowledge for distinguishing purple coneflower from its confused category blanket flower .
Fig. 18: Score-gap distributions for correct and incorrect VLM predictions over the nine FGVR datasets. The score gap is computed between the top-1 and top-2 predictions for CLIP ViT-B/16 (left) and SigLIP (right).
Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or detecting key moments in long videos. Existing methods typically rely on complex, task-specific fine-tuning, which reduces generalizability and increases system complexity. In this work, we propose an effective, training-free framework that uses an MLLM's intrinsic uncertainty as proactive guidance. Our core insight is that a model's uncertainty decreases when provided with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most informative data. We apply this simple principle to three challenging visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned systems. Our results demonstrate that leveraging intrinsic uncertainty is a powerful strategy for improving fine-grained multimodal performance.
Sanghwan Kim, Rui Xiao, Stephan Alaniz +2
1Technical University of Munich · 2Munich Center for Machine Learning · 20263Helmholtz Munich +2
Recent advances in Large Vision-Language Models (LVLMs) have enabled training-free Fine-Grained Visual Recognition (FGVR). However, effectively exploiting LVLMs for FGVR remains challenging due to the inherent visual ambiguity of subordinate-level categories. Existing methods predominantly adopt either retrieval-oriented or reasoning-oriented paradigms to tackle this challenge, but both are constrained by two fundamental limitations:(1) They apply the same inference pipeline to all samples without accounting for uneven recognition difficulty, thereby leading to suboptimal accuracy and efficiency; (2) The lack of mechanisms to consolidate and reuse error-specific experience causes repeated failures on similar challenging cases. To address these limitations, we propose SARE, a Sample-wise Adaptive textbfREasoning framework for training-free FGVR. Specifically, SARE adopts a cascaded design that combines fast candidate retrieval with fine-grained reasoning, invoking the latter only when necessary. In the reasoning process, SARE incorporates a self-reflective experience mechanism that leverages past failures to provide transferable discriminative guidance during inference, without any parameter updates. Extensive experiments across 14 datasets substantiate that SARE achieves state-of-the-art performance while substantially reducing computational overhead.
Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately 4.0× inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.
Yuxiang Shen, Hailong Huang, Zhenkun Gao +6
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3