SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognition

Jingxiao Yang; DaLin He; Miao Pan; Kaixiang Yao; Ge Su; Wenqi Zhang; Yifeng Hu; Tangwei Li

cs.CVMar 18, 2026

SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognition

Authors: Jingxiao Yang, DaLin He, Miao Pan, Kaixiang Yao, Ge Su, Wenqi Zhang, Yifeng Hu, Tangwei Li,

Organizations: Zhejiang University · Netease Yidun AI Lab

Abstract

Recent advances in Large Vision-Language Models (LVLMs) have enabled training-free Fine-Grained Visual Recognition (FGVR). However, effectively exploiting LVLMs for FGVR remains challenging due to the inherent visual ambiguity of subordinate-level categories. Existing methods predominantly adopt either retrieval-oriented or reasoning-oriented paradigms to tackle this challenge, but both are constrained by two fundamental limitations:(1) They apply the same inference pipeline to all samples without accounting for uneven recognition difficulty, thereby leading to suboptimal accuracy and efficiency; (2) The lack of mechanisms to consolidate and reuse error-specific experience causes repeated failures on similar challenging cases. To address these limitations, we propose SARE, a Sample-wise Adaptive textbfREasoning framework for training-free FGVR. Specifically, SARE adopts a cascaded design that combines fast candidate retrieval with fine-grained reasoning, invoking the latter only when necessary. In the reasoning process, SARE incorporates a self-reflective experience mechanism that leverages past failures to provide transferable discriminative guidance during inference, without any parameter updates. Extensive experiments across 14 datasets substantiate that SARE achieves state-of-the-art performance while substantially reducing computational overhead.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Fine-Grained Visual Recognition challenge: sub-categories (e.g., Siberian Husky vs. Alaskan Malamute) exhibit subtle visual differences, requiring attention to localized discriminative features such as forehead patterns and tail shape. (b) For a fixed overall retrieval accuracy (70%), different sub-categories require substantially different confidence thresholds, indicating that identical confidence scores imply varying reliability across categories.

Figure 2: Overview of SARE . The framework performs fast prototype-based retrieval to generate candidate categories for a query image, followed by a class-conditional trigger that adaptively invokes fine-grained reasoning when retrieval confidence is insufficient. Experience distilled from past errors is injected as contextual guidance, enabling accurate and efficient training-free FGVR.

Method	Fine-Grained Visual Recognition (FGVC)							General Recognition					Avg.	$\Delta$
Method	Dogs	Flowers	CUB	Cars	Pets	Air.	Bird.	Food	DTD	IN-1k	SUN	UCF	Avg.	$\Delta$
Backbone
CLIP-B/32	52.09	63.44	53.26	62.38	81.66	23.67	42.37	80.32	44.26	63.67	62.32	64.13	57.80	-2.80
Qwen2.5-VL-7B	65.90	68.50	43.70	75.31	85.12	54.01	54.48	69.90	68.83	68.40	58.11	73.30	65.46	+4.86
Training-free LVLM Methods
(Retrieval-based Methods)

Table 1: Comparison with state-of-the-art methods on 7 fine-grained and 5 general datasets. We highlight the best results in bold and the second-best results with underlining. “Avg.” denotes the average accuracy across all 12 datasets.

\Delta

indicates the performance gain compared to FineR.

Figure 3: Comparison of SARE against baselines on Stanford Cars dataset. SARE achieves the optimal balance, significantly outperforming baselines in accuracy with lower inference overhead.

System 1	System 2	Experience	Dogs	CUB
✓			58.24	73.02
	✓		78.61	78.43
✓	✓		76.01	81.36
	✓	✓	81.34	88.21
✓	✓	✓	84.29	90.76

Table 2: Ablation study of key components in SARE.

Method	ImageNet-V2		ImageNet-Sketch
Method	Acc. (%)	$\Delta$	Acc. (%)	$\Delta$
$SARE_{tgt}$	84.43	-	72.49	-
$SARE_{src}$	83.72	-0.69	71.96	-0.53

Table 3: Comparison of SARE with experience constructed on ImageNet-1K (

SARE_{src}

) versus experience constructed on target-domain experience (

SARE_{tgt}

Figure 4: The proportion of samples triggering System 2 across datasets with varying recognition difficulty. The x-axis measures dataset-level difficulty using $100\%-CLIP_{Top-1}$ accuracy.

Panel of Figure 5 — Figure 5: Performance of SARE across different backbone architectures. SARE consistently enhances performance across all backbones, with larger relative gains on lower-performing models.

Figure 6: Behavior of different inference strategies under increasing sample difficulty on Stanford Dogs.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Aircraft	Birdsnap	Dogs	DTD	CUB	SUN	IN-1K	Cars	Flowers	UCF	Food	Pets
Trigger Rate (%)	96.19	92.92	82.44	75.03	74.08	68.82	62.90	60.24	56.85	44.57	39.49	24.78

Appendix

Table 4: Analysis of Trigger Rates. The percentage of samples activating the System 2 varies significantly based on dataset difficulty.

Dataset	Aircraft	Birdsnap	Dogs	DTD	CUB	SUN	IN-1K	Cars	Flowers	UCF	Food	Pets
Sys-1 Acc. (%)	88.93	96.96	91.52	100	95.78	91.98	99.97	99.37	97.94	97.74	92.81	95.43

Appendix

Table 5: Reliability of System 1 Decisions. We report the recognition accuracy on the subset of samples that the dynamic trigger determined did not require System 2 reasoning. The consistently high accuracy across datasets demonstrates that our trigger effectively identifies easy samples where retrieval-based perception is sufficient.

Fine-Grained Visual Recognition
Dataset	# Cls.	# Test Size	Domain
CUB-200-2011	200	5,794	Birds
Stanford Dogs	120	8,580	Dogs
Stanford Cars	196	8,041	Cars
FGVC-Aircraft	100	3,333	Aircraft
Oxford-IIIT Pet	37	3,669	Animals

Appendix

Table 6: Detailed statistics of the 14 datasets used in our experiments. To save space, # Cls. denotes the number of categories.

Figure 7: Hyperparameter Sensitivity Analysis. We analyze the impact of candidate count ( $K_{c}$ ) and experience capacity ( $\mathcal{E}$ ) on average accuracy across seven fine-grained benchmarks. The line plots reveal a clear inverted-U trend for both parameters: performance improves as context increases, peaks at the optimal settings ( $K_{c}=10,\mathcal{E}=8$ ), and slightly declines beyond this point due to diminishing marginal utility and the introduction of semantic noise.

Figure 8: Comparison of SARE against baselines on Stanford Dogs dataset.

Sampling Strategy	Top-1 Acc (%)
Random	84.29
Diversity	84.59
Confusion	83.56
Centroid	85.15
Entropy	85.44
Boundary	85.68

Appendix

Table 7: Impact of Sampling Strategies. Performance of SARE on Stanford Dogs under different

k

-shot sampling strategies.

Figure 9: Sensitivity Analysis of Shot Number $k_{shot}$ . The results demonstrate that SARE maintains robust performance across different few-shot settings. As shown in the figure, the accuracy fluctuation is minimal as $k$ varies from 1 to 10, indicating that our method is not overly sensitive to the exact size of the support set.

Figure 10: Visualization of Experience Generation and Reuse. Left (Reflection): The model analyzes past misclassifications (e.g., confusing an Afghan Hound with a Collie or Golden Retriever) to distill a generalized decision rule: “Prioritize morphological features over color.” Right (Inference): When encountering a visually ambiguous Black-and-tan Coonhound , System 1 is misled by the coat color and retrieves Rottweiler . However, System 2 retrieves the accumulated rule, corrects the focus to morphological details (ears), and successfully identifies the correct category.

Figure 11: Visualization of attention heatmaps.

Fine-Grained Visual Perception

Fine-grained visual perception studies how models detect, distinguish, and ground subtle visual details, attributes, parts, and changes.

See all 153 papers

From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognition
Oct 9, 2026Yutong Yang, Zhuqi Liu, Mouxing Yang +1Multimodal Large Language Models Fine-Grained Visual Perception
College of Computer Science, Sichuan University, Chengdu 610065, China · School of Artificial Intelligence, Sichuan University, Chengdu 610065, China, and also with Tianfu Jincheng Laboratory, Chengdu 610093, China
Learning Adaptive Reasoning Paths for Efficient Visual Reasoning
Apr 16, 2026Yixu Huang, Tinghui Zhu, Muhao ChenEfficient VLM Inference Visual Reasoning
♠Fudan University · ♣University of California, Davis
Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning
Jul 1, 2026Hongxing Li, Xiufeng Huang, Dingming Li +11Visual Reasoning VLM Reasoning
1Zhejiang University · 2Alibaba Group

Oct 9, 2026J/K move · Enter open · S save

From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognition

Yutong Yang, Zhuqi Liu, Mouxing Yang, Xi Peng

College of Computer Science, Sichuan University, Chengdu 610065, China · School of Artificial Intelligence, Sichuan University, Chengdu 610065, China, and also with Tianfu Jincheng Laboratory, Chengdu 610093, China

Multimodal Large Language Models Fine-Grained Visual Perception

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) generally fall short of expectations in Fine-Grained Visual Recognition (FGVR), although they exhibit promising FGVR foundations, i.e., visual perception and broad category knowledge. Most existing studies seek to remedy this deficiency by adopting domain-specific post-training, at the risk of compromising generalization to unseen domains. Different from these efforts, in this paper, we revisit the FGVR mechanism underlying MLLMs from both the empirical and theoretical perspectives and uncover an inference-time phenomenon termed FGVR Laziness. In brief, FGVR Laziness refers to the insufficient association between visual cues and category concepts under certain candidate category lists and recognition guidance, leaving the FGVR foundations of MLLMs largely under-utilized. To mitigate FGVR Laziness, we propose Laziness-resIstant Visual rEcognition (LIVE), a training-free framework which jointly refines candidate lists and recognition guidance to elicit the latent FGVR capability of MLLMs. Specifically, LIVE first performs query-adaptive candidate inference to derive a compact yet reliable candidate list. Meanwhile, LIVE constructs category-specific FGVR skills by mining the intra-category characteristics and inter-category differences. Finally, LIVE selectively invokes detailed skill guidance according to recognition uncertainty and category confusion, shifting MLLM-based FGVR from a cursory glance to a focus on associating discriminative visual cues with category concepts. Extensive experiments on 13 benchmarks against 24 state-of-the-art methods spanning fine-grained category recognition, action recognition, and general category recognition, verify the effectiveness and generalizability of LIVE across different MLLM backbones. The code will be released upon acceptance.

Abstract

Fine-Grained Visual Perception

Explore similar work

From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognition

Learning Adaptive Reasoning Paths for Efficient Visual Reasoning

Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning