Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model's capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL.
Figures & tables
Figure 1: In open-world image classification, improving prediction specificity without compromising correctness remains challenging. Existing techniques, such as prompting to be specific, supervised fine-tuning ( sft ) or reinforcement fine-tuning ( rft ), promote specificity but reduce correctness. Instead, our proposed method ( SpeciaRL ) significantly improves the specificity of the base Qwen2.5VL-7B model without compromising correctness. Gray arrows indicate that training is performed on a single-domain (birds) dataset, which is disjoint from the domains in the test set, therefore illustrating cross-domain generalization.
Figure 2: Predictions distribution over categories for Qwen2.5VL-7B [ 3 ] and its BoN version with N=64 inference runs. The right side shows specificity, correctness and their harmonic mean (HM). The BoN-64 serves as an indication for the model’s potential capability.
Figure 3: Overview of SpeciaRL Given an input image I , the policy model generates N open-ended predictions {p1,…,pN} . Each prediction is categorized by a judge model (LLM verifier) as wrong or correct at different levels of specificity with respect to the ground-truth. A verifiable reward ri∗ is then assigned according to whether the prediction’s category ci meets the adaptive reference level c∗ , which is defined based on the best prediction within the N rollouts. The resulting graded rewards are aggregated through a Group Relative Policy Optimization (GRPO) update to reinforce policies that are maximally specific while remaining correct.
Fine-grained
Very fine-grained
Prediction categorization
Metrics
Prediction categorization
Metrics
Model
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
CaSED [ 8 ]
0.0%
43.7%
10.6%
24.2%
0.0%
21.5%
0.812
0.785
0.797
0.0%
0.9%
13.8%
56.0%
0.0%
29.3%
0.56
0.707
0.612
InternVL2.5-4B [ 6 ]
0.0%
11.4%
1.5%
54.4%
8.4%
24.1%
0.554
0.759
0.639
0.0%
0.1%
1.2%
62.7%
5.5%
30.5%
0.486
0.695
0.571
InternVL2.5-8B [ 6 ]
0.7%
16.7%
3.3%
30.6%
20.7%
27.9%
0.575
0.721
0.624
0.0%
1.2%
5.7%
54.5%
15.6%
22.9%
0.476
0.771
0.589
Qwen2.5VL-3B [ 3 ]
0.8%
17.3%
2.7%
53.4%
4.2%
21.5%
0.608
0.785
0.685
0.1%
1.1%
3.9%
75.1%
2.4%
17.4%
0.511
0.826
0.631
Table 1: Open-world image classification results of Ay zero-shot methods and models Ay fine-tuned out-of-domain . We report the ratio of the predictions within categories assigned by the LLM verifier, our measures of specificity and correctness and the harmonic mean of these two (HM). Results are averaged over all datasets within the fine-grained and very fine-grained sets. For reference, we report the performance of Ay inference out of 64 runs (BoN-64) . Best in bold ; second best underlined .
Fine-grained
Very fine-grained
Model
TI ↑
LI ↑
SS ↑
CS ↑
TI ↑
LI ↑
SS ↑
CS ↑
Retrieval-based baselines
CASED
27.4
46.6
60.7
61.7
0.7
47.1
38.5
38.5
CLIP retrieval
32.4
45.4
42.9
65.4
7.0
18.1
39.7
56.1
Non-reasoning LMMs
IDEFICS2 8B
3.0
49.9
38.0
41.7
0.0
67.0
29.6
33.6
Table 2: Comparison against state-of-the-art methods following the evaluation protocol of [ 9 ] . Key: TI:Text Inclusion; LI: Language Inclusion; SS: Semantic Similarity; CS: Concept Similarity. Best in bold ; second best underlined .
Figure 4: Qualitative examples of the think-answer output from the base model Qwen2.5VL-7B and our SpeciaRL, which steers the reasoning traces towards more specific prediction.
Prediction categorization
Metrics
Model
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
S+&S (1)
4.6%
52.2%
5.0%
16.2%
0.0%
21.5%
0.875
0.785
0.825
S+&S (1) S− (0.75)
4.9%
62.6%
5.6%
10.4%
0.0%
16.4%
0.919
0.836
0.875
S+&S (1) S− (0.75) G (0.5)
3.6%
61.1%
5.1%
17.7%
0.0%
12.6%
0.884
0.874
0.878
S+&S (1) S− (0.75) G (0.5) A (0.25)
1.4%
63.9%
6.7%
11.5%
0.0%
16.5%
0.911
0.835
0.871
SpeciaRL-7B (dynamic reward)
5.6%
63.4%
5.1%
10.7%
0.0%
15.2%
0.920
0.848
0.883
Table 3: SpeciaRL against rft with different static reward rules. Best in bold .
Prediction categorization
Metrics
N rollouts
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
5
5.5%
63.6%
5.8%
9.5%
0.0%
15.6%
0.925
0.844
0.883
10
5.6%
63.4%
5.1%
10.7%
0.0%
15.2%
0.920
0.848
0.883
15
4.6%
50.4%
3.7%
22.2%
0.0%
19.0%
0.848
0.810
0.824
Table 4: SpeciaRL with different rollouts size N . Best in bold .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: LMM default prompt for prediction.
Figure 6: LMM prompt for prediction for the “Be specific” baseline.
Figure 7: LMM prompt used in the evaluation protocol of [ 9 ] .
Figure 8: Prompt for the LLM-as-a-judge verifier categorizing a prediction given the target ground-truth.
Figure 9: Prompt for generating the reasoning traces used to train the supervised fine-tuning baseline model.
Figure 10: LLM-as-a-judge per-batch verification times during reinforcement learning training, showing the speedup obtained as cache hit rates increase when starting from an empty cache.
Figure 11: Additional qualitative examples of the think-answer output of the base model Qwen2.5VL-7B and SpeciaRL.
Figure 12: Failure cases. Qualitative examples of SpeciaRL providing a Wrong prediction (Top & Center) and of SpeciaRL unnecessarily using a scientific name for a generic concept (Bottom).
Fine-grained
Prediction categorization
Metrics
Dataset
Model
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
Flowers102 [ 35 ]
CaSED [ 8 ]
0.0%
57.4%
8.4%
14.7%
0.0%
19.4%
0.883
0.806
0.842
InternVL2.5-4B [ 6 ]
0.2%
15.8%
1.8%
29.8%
20.0%
32.4%
0.551
0.676
0.607
InternVL2.5-8B [ 6 ]
0.4%
26.4%
3.3%
15.1%
13.2%
41.6%
0.688
0.584
0.632
Qwen2.5VL-3B [ 3 ]
0.1%
26.6%
2.7%
49.6%
1.9%
19.2%
0.668
0.808
0.731
Appendix
Table 5: Results on the individual datasets composing the fine-grained set.
Very fine-grained
Prediction categorization
Metrics
Dataset
Model
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
FGVCAircraft [ 34 ]
CaSED [ 8 ]
0.0%
1.6%
13.9%
37.7%
0.0%
46.8%
0.580
0.532
0.555
InternVL2.5-4B [ 6 ]
0.0%
0.0%
0.2%
66.0%
8.5%
25.3%
0.472
0.747
0.579
InternVL2.5-8B [ 6 ]
0.1%
2.2%
1.3%
59.1%
13.0%
24.4%
0.476
0.756
0.584
Qwen2.5VL-3B [ 3 ]
0.2%
1.6%
1.4%
82.4%
0.3%
14.1%
0.514
0.859
0.643
Appendix
Table 6: Individual dataset results on the very fine-grained set.
In-domain
Prediction categorization
Metrics
Dataset
Model
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
CUB [ 50 ]
Qwen2.5VL-7B [ 3 ]
0.2%
23.0%
15.9%
48.1%
2.0%
11.0%
0.669
0.890
0.764
Qwen2.5VL-7B (“Be specific”)
0.2%
32.2%
13.7%
35.3%
2.6%
16.1%
0.726
0.839
0.779
Qwen2.5VL-7B ( sft )
0.1%
80.4%
0.7%
0.3%
0.0%
18.5%
0.996
0.815
0.896
Qwen2.5VL-7B ( rft )
1.0%
92.7%
0.0%
0.0%
0.0%
6.3%
1.000
0.937
0.968
Appendix
Table 7: In-domain evaluation of the training strategies.
Figure 13: Generated LMM prompt ( Pc (v1)).
Figure 14: Generated LMM prompt ( Pc (v2)).
Figure 15: Generated LMM prompt ( Pc (v3)).
Fine-grained
Very fine-grained
spec. ↑
corr. ↑
HM ↑
spec. ↑
corr. ↑
HM ↑
Pc (“Be specific”)
0.816
0.832
0.822
0.652
0.89
0.751
Pc (v1)
0.840
0.830
0.834
0.688
0.885
0.772
Pc (v2)
0.814
0.849
0.830
0.637
0.902
0.746
Pc (v3)
0.884
0.764
0.819
0.777
0.832
0.799
Appendix
Table 8: Performance comparison of additional prompting baseline.
Fine-grained
Prediction categorization
Metrics
Test set
Training set
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
Flowers102 [ 35 ]
Flowers102
0.0%
82.5%
2.7%
1.8%
0.0%
12.9%
0.982
0.871
0.923
Food101
0.1%
66.5%
4.3%
10.4%
0.0%
18.7%
0.923
0.813
0.864
OxfordPets
0.2%
72.8%
6.5%
4.7%
0.0%
15.8%
0.953
0.842
0.894
CUB
13.6%
69.2%
5.0%
1.7%
0.0%
10.5%
0.976
0.895
0.934
Appendix
Table 9: Individual dataset results for SpeciaRL-7B trained with different fine-grained datasets. In-domain performance is highlighted in Ay blue italic and best out-of-domain results on each test set is highlighted in bold . Note that CUB is an additional dataset, i.e . not part of the fine-grained test sets that are used in [ 9 ] and our main evaluation.
Prediction categorization
Metrics
Sample size
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
100
0.1%
53.1%
4.6%
8.7%
0.0%
33.5%
0.917
0.665
0.771
1000
0.2%
69.7%
5.4%
7.7%
0.0%
17.1%
0.938
0.829
0.880
2000
0.9%
91.6%
0.0%
0.0%
0.0%
7.5%
1.000
0.925
0.961
3000
0.6%
92.7%
0.0%
0.0%
0.0%
6.7%
1.000
0.933
0.965
Appendix
Table 10: In-domain results of SpeciaRL-7B trained with different dataset sizes sampled from CUB, and evaluated with CUB test set.
Prediction categorization
Metrics
Sample size
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
100
2.5%
64.9%
6.8%
7.6%
0.2%
18.0%
0.930
0.820
0.872
1000
3.2%
66.5%
6.2%
7.9%
0.0%
16.2%
0.933
0.838
0.883
2000
6.0%
64.8%
6.2%
6.4%
0.1%
16.6%
0.941
0.834
0.884
3000
5.6%
63.4%
5.1%
10.7%
0.0%
15.2%
0.920
0.848
0.883
Appendix
Table 11: Out-of-domain results of SpeciaRL-7B trained with different dataset sizes sampled from CUB. Results are averaged over fine-grained datasets.
CUB
Fine-grained
Very fine-grained
spec. ↑
corr. ↑
HM ↑
spec. ↑
corr. ↑
HM ↑
spec. ↑
corr. ↑
HM ↑
CUB
1.000
0.933
0.965
0.920
0.848
0.833
0.818
0.855
0.830
Mixed
0.995
0.889
0.939
0.963
0.878
0.918
0.863
0.860
0.852
Appendix
Table 12: Comparison between SpeciaRL trained on a single domain (CUB) versus a mixture of samples from all available domains.
Prediction categorization
Metrics
RL method
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
GRPO [ 40 ]
4.6%
52.2%
5.0%
16.2%
0.0%
21.5%
0.875
0.785
0.825
SpeciaRL (GRPO)
5.6%
63.4%
5.1%
10.7%
0.0%
15.2%
0.920
0.848
0.883
Dr.GRPO [ 32 ]
8.6%
59.3%
6.5%
5.3%
0.2%
20.1%
0.942
0.799
0.864
SpeciaRL (Dr.GRPO)
6.6%
64.4%
6.0%
4.9%
0.0%
18.2%
0.951
0.818
0.879
DAPO [ 55 ]
7.3%
61.0%
7.1%
3.2%
0.4%
21.0%
0.951
0.790
0.862
Appendix
Table 13: SpeciaRL compared to static reward rft across different on-policy RL algorithms. Best in bold . Results are averaged over fine-grained datasets.
Fine-grained
Very fine-grained
AR
κ
AR
κ
Qwen3-30B
0.90
0.84
0.92
0.82
Llama3-7B
0.75
0.64
0.69
0.48
Pj ( v1 )
0.94
0.91
0.95
0.89
Pj ( v2 )
0.91
0.87
0.91
0.80
Pj ( v3 )
0.90
0.85
0.90
0.76
Appendix
Table 14: LLM-as-a-judge validation across different models and prompt variants.
Prediction categorization
Metrics
ρe
S+
S
S−
G
A
W
spec. ↑
corr. ↑
HM ↑
0%
3.2%
66.5%
6.2%
7.9%
0.0%
16.2%
0.933
0.838
0.883
5%
5.6%
65.3%
6.4%
5.4%
0.0%
17.3%
0.946
0.827
0.882
10%
3.3%
64.7%
6.6%
8.4%
0.0%
16.9%
0.928
0.831
0.877
25%
2.0%
64.5%
6.6%
10.5%
0.0%
16.4%
0.916
0.836
0.874
Appendix
Table 15: Sensitivity of SpeciaRL to LLM-judge error. Results are averaged over fine-grained datasets.
Figure 16: Generated Prompt for the LLM-as-a-judge verifier.
Figure 17: Generated Prompt for the LLM-as-a-judge verifier.
Figure 18: Generated Prompt for the LLM-as-a-judge verifier.
Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately 4.0× inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.
Yuxiang Shen, Hailong Huang, Zhenkun Gao +6
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
Traditional vision-language models struggle with contrastive fine-grained taxonomic reasoning, particularly when distinguishing between visually similar species within the same genus or family. We introduce TaxonRL, a reinforcement learning approach using Group Relative Policy Optimization with intermediate rewards that decomposes the reasoning process into hierarchical taxonomic predictions. Our method incentivizes models to explicitly reason about species-level, genus-level, and family-level features before making final classifications. This structured approach is designed not only to boost accuracy but also to yield a transparent, verifiable decision-making process. On the challenging Birds-to-Words dataset, TaxonRL achieves 91.7% average accuracy, exceeding human performance (77.3%) while generating interpretable reasoning traces. We demonstrate strong cross-domain generalization, showing substantial gains in primate and marine species verification. Our results establish that enforcing structured, hierarchical reasoning provides a powerful and transferable framework for fine-grained visual discrimination.
Maximilian von Klinski, Maximilian Schall
Hasso Plattner Institute University of Potsdam, Germany
Fine-grained visual understanding is shifting from static classification to knowledge-augmented reasoning, where models must justify as well as recognise. Existing approaches remain limited by closed-set taxonomies and single-label prediction, leading to significant degradation under open-set or context-dependent conditions. We present the Knowledge-Augmented Fine-Grained Reasoning Agent (KFRA), a unified framework that transforms fine-grained perception into evidence-driven reasoning. KFRA operates through a three-stage closed reasoning loop that emulates expert analysis. It first performs open-vocabulary detection and web-scale retrieval to generate category hypotheses. It then conducts discriminative regions localisation by aligning textual knowledge with visual evidence through a global-to-local focusing mechanism. Finally, it integrates all multimodal evidence within a large multimodal model to perform interpretable reasoning. Unlike existing agents that treat retrieval and reasoning as independent processes, KFRA establishes a retrieval-grounding coupling that converts retrieved knowledge into spatially grounded evidence for verification. This design enables factual, interpretable, and task-agnostic reasoning across diverse fine-grained scenarios. To evaluate this capability, we construct FGExpertBench, a benchmark designed to assess reasoning depth and cross-task generalisation across six knowledge dimensions. Extensive experiments demonstrate that KFRA consistently surpasses both standalone large multimodal models and current agent frameworks, achieving up to 19 percent improvement in reasoning accuracy and delivering evidence-grounded interpretability in open-set fine-grained visual understanding.
Junhan Chen, Zilu Zhou, Yujun Tong +3
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China