Specificity-aware reinforcement learning for fine-grained open-world classification

Samuele Angheben; Davide Berasi; Alessandro Conti; Elisa Ricci; Yiming Wang

cs.CVMar 3, 2026

Specificity-aware reinforcement learning for fine-grained open-world classification

Authors: Samuele Angheben, Davide Berasi, Alessandro Conti, Elisa Ricci, Yiming Wang

Organizations: University of Trento · Fondazione Bruno Kessler

Abstract

Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model's capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL.

Figures & tables

Figure 1: In open-world image classification, improving prediction specificity without compromising correctness remains challenging. Existing techniques, such as prompting to be specific, supervised fine-tuning ( sft ) or reinforcement fine-tuning ( rft ), promote specificity but reduce correctness. Instead, our proposed method ( SpeciaRL ) significantly improves the specificity of the base Qwen2.5VL-7B model without compromising correctness. Gray arrows indicate that training is performed on a single-domain (birds) dataset, which is disjoint from the domains in the test set, therefore illustrating cross-domain generalization.

Figure 2: Predictions distribution over categories for Qwen2.5VL-7B [ 3 ] and its BoN version with

N=64

inference runs. The right side shows specificity, correctness and their harmonic mean (HM). The BoN-64 serves as an indication for the model’s potential capability.

Figure 3: Overview of SpeciaRL Given an input image $I$ , the policy model generates $N$ open-ended predictions $\{p_{1},\dots,p_{N}\}$ . Each prediction is categorized by a judge model (LLM verifier) as wrong or correct at different levels of specificity with respect to the ground-truth. A verifiable reward $r_{i}^{*}$ is then assigned according to whether the prediction’s category $c_{i}$ meets the adaptive reference level $c^{*}$ , which is defined based on the best prediction within the $N$ rollouts. The resulting graded rewards are aggregated through a Group Relative Policy Optimization (GRPO) update to reinforce policies that are maximally specific while remaining correct.

	Fine-grained									Very fine-grained
	Prediction categorization						Metrics			Prediction categorization						Metrics
Model	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
CaSED [ 8 ]	0.0%	43.7%	10.6%	24.2%	0.0%	21.5%	0.812	0.785	0.797	0.0%	0.9%	13.8%	56.0%	0.0%	29.3%	0.56	0.707	0.612
InternVL2.5-4B [ 6 ]	0.0%	11.4%	1.5%	54.4%	8.4%	24.1%	0.554	0.759	0.639	0.0%	0.1%	1.2%	62.7%	5.5%	30.5%	0.486	0.695	0.571
InternVL2.5-8B [ 6 ]	0.7%	16.7%	3.3%	30.6%	20.7%	27.9%	0.575	0.721	0.624	0.0%	1.2%	5.7%	54.5%	15.6%	22.9%	0.476	0.771	0.589
Qwen2.5VL-3B [ 3 ]	0.8%	17.3%	2.7%	53.4%	4.2%	21.5%	0.608	0.785	0.685	0.1%	1.1%	3.9%	75.1%	2.4%	17.4%	0.511	0.826	0.631

Table 1: Open-world image classification results of Ay zero-shot methods and models Ay fine-tuned out-of-domain . We report the ratio of the predictions within categories assigned by the LLM verifier, our measures of specificity and correctness and the harmonic mean of these two (HM). Results are averaged over all datasets within the fine-grained and very fine-grained sets. For reference, we report the performance of Ay inference out of 64 runs (BoN-64) . Best in bold ; second best underlined .

Retrieval-based baselines
	Fine-grained				Very fine-grained
Model	TI $\uparrow$	LI $\uparrow$	SS $\uparrow$	CS $\uparrow$	TI $\uparrow$	LI $\uparrow$	SS $\uparrow$	CS $\uparrow$
CASED	27.4	46.6	60.7	61.7	0.7	47.1	38.5	38.5
CLIP retrieval	32.4	45.4	42.9	65.4	7.0	18.1	39.7	56.1
Non-reasoning LMMs
IDEFICS2 8B	3.0	49.9	38.0	41.7	0.0	67.0	29.6	33.6

Table 2: Comparison against state-of-the-art methods following the evaluation protocol of [ 9 ] . Key: TI:Text Inclusion; LI: Language Inclusion; SS: Semantic Similarity; CS: Concept Similarity. Best in bold ; second best underlined .

Figure 4: Qualitative examples of the think-answer output from the base model Qwen2.5VL-7B and our SpeciaRL, which steers the reasoning traces towards more specific prediction.

	Prediction categorization						Metrics
Model	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
$S^{+}\&S$ (1)	4.6%	52.2%	5.0%	16.2%	0.0%	21.5%	0.875	0.785	0.825
$S^{+}\&S$ (1) $S^{-}$ (0.75)	4.9%	62.6%	5.6%	10.4%	0.0%	16.4%	0.919	0.836	0.875
$S^{+}\&S$ (1) $S^{-}$ (0.75) $G$ (0.5)	3.6%	61.1%	5.1%	17.7%	0.0%	12.6%	0.884	0.874	0.878
$S^{+}\&S$ (1) $S^{-}$ (0.75) $G$ (0.5) $A$ (0.25)	1.4%	63.9%	6.7%	11.5%	0.0%	16.5%	0.911	0.835	0.871
SpeciaRL-7B (dynamic reward)	5.6%	63.4%	5.1%	10.7%	0.0%	15.2%	0.920	0.848	0.883

Table 3: SpeciaRL against rft with different static reward rules. Best in bold .

	Prediction categorization						Metrics
N rollouts	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
5	5.5%	63.6%	5.8%	9.5%	0.0%	15.6%	0.925	0.844	0.883
10	5.6%	63.4%	5.1%	10.7%	0.0%	15.2%	0.920	0.848	0.883
15	4.6%	50.4%	3.7%	22.2%	0.0%	19.0%	0.848	0.810	0.824

Table 4: SpeciaRL with different rollouts size

N

. Best in bold .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: LMM default prompt for prediction.

Panel of Figure 6 — Figure 6: LMM prompt for prediction for the “Be specific” baseline.

Panel of Figure 7 — Figure 7: LMM prompt used in the evaluation protocol of [ 9 ] .

Panel of Figure 8 — Figure 8: Prompt for the LLM-as-a-judge verifier categorizing a prediction given the target ground-truth.

Panel of Figure 9 — Figure 9: Prompt for generating the reasoning traces used to train the supervised fine-tuning baseline model.

Figure 10: LLM-as-a-judge per-batch verification times during reinforcement learning training, showing the speedup obtained as cache hit rates increase when starting from an empty cache.

Figure 11: Additional qualitative examples of the think-answer output of the base model Qwen2.5VL-7B and SpeciaRL.

Figure 12: Failure cases. Qualitative examples of SpeciaRL providing a Wrong prediction (Top & Center) and of SpeciaRL unnecessarily using a scientific name for a generic concept (Bottom).

Fine-grained
		Prediction categorization						Metrics
Dataset	Model	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
Flowers102 [ 35 ]	CaSED [ 8 ]	0.0%	57.4%	8.4%	14.7%	0.0%	19.4%	0.883	0.806	0.842
	InternVL2.5-4B [ 6 ]	0.2%	15.8%	1.8%	29.8%	20.0%	32.4%	0.551	0.676	0.607
	InternVL2.5-8B [ 6 ]	0.4%	26.4%	3.3%	15.1%	13.2%	41.6%	0.688	0.584	0.632
	Qwen2.5VL-3B [ 3 ]	0.1%	26.6%	2.7%	49.6%	1.9%	19.2%	0.668	0.808	0.731

Appendix

Table 5: Results on the individual datasets composing the fine-grained set.

Very fine-grained
		Prediction categorization						Metrics
Dataset	Model	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
FGVCAircraft [ 34 ]	CaSED [ 8 ]	0.0%	1.6%	13.9%	37.7%	0.0%	46.8%	0.580	0.532	0.555
	InternVL2.5-4B [ 6 ]	0.0%	0.0%	0.2%	66.0%	8.5%	25.3%	0.472	0.747	0.579
	InternVL2.5-8B [ 6 ]	0.1%	2.2%	1.3%	59.1%	13.0%	24.4%	0.476	0.756	0.584
	Qwen2.5VL-3B [ 3 ]	0.2%	1.6%	1.4%	82.4%	0.3%	14.1%	0.514	0.859	0.643

Appendix

Table 6: Individual dataset results on the very fine-grained set.

In-domain
		Prediction categorization						Metrics
Dataset	Model	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
CUB [ 50 ]	Qwen2.5VL-7B [ 3 ]	0.2%	23.0%	15.9%	48.1%	2.0%	11.0%	0.669	0.890	0.764
	Qwen2.5VL-7B (“Be specific”)	0.2%	32.2%	13.7%	35.3%	2.6%	16.1%	0.726	0.839	0.779
	Qwen2.5VL-7B ( sft )	0.1%	80.4%	0.7%	0.3%	0.0%	18.5%	0.996	0.815	0.896
	Qwen2.5VL-7B ( rft )	1.0%	92.7%	0.0%	0.0%	0.0%	6.3%	1.000	0.937	0.968

Appendix

Table 7: In-domain evaluation of the training strategies.

Panel of Figure 13 — Figure 13: Generated LMM prompt ( $\mathtt{P}_{c}$ (v1)).

Panel of Figure 14 — Figure 14: Generated LMM prompt ( $\mathtt{P}_{c}$ (v2)).

Panel of Figure 15 — Figure 15: Generated LMM prompt ( $\mathtt{P}_{c}$ (v3)).

	Fine-grained			Very fine-grained
	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
$\mathtt{P}_{c}$ (“Be specific”)	0.816	0.832	0.822	0.652	0.89	0.751
$\mathtt{P}_{c}$ (v1)	0.840	0.830	0.834	0.688	0.885	0.772
$\mathtt{P}_{c}$ (v2)	0.814	0.849	0.830	0.637	0.902	0.746
$\mathtt{P}_{c}$ (v3)	0.884	0.764	0.819	0.777	0.832	0.799

Appendix

Table 8: Performance comparison of additional prompting baseline.

Fine-grained
		Prediction categorization						Metrics
Test set	Training set	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
Flowers102 [ 35 ]	Flowers102	0.0%	82.5%	2.7%	1.8%	0.0%	12.9%	0.982	0.871	0.923
	Food101	0.1%	66.5%	4.3%	10.4%	0.0%	18.7%	0.923	0.813	0.864
	OxfordPets	0.2%	72.8%	6.5%	4.7%	0.0%	15.8%	0.953	0.842	0.894
	CUB	13.6%	69.2%	5.0%	1.7%	0.0%	10.5%	0.976	0.895	0.934

Appendix

Table 9: Individual dataset results for SpeciaRL-7B trained with different fine-grained datasets. In-domain performance is highlighted in Ay blue italic and best out-of-domain results on each test set is highlighted in bold . Note that CUB is an additional dataset, i.e . not part of the fine-grained test sets that are used in [ 9 ] and our main evaluation.

	Prediction categorization						Metrics
Sample size	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
100	0.1%	53.1%	4.6%	8.7%	0.0%	33.5%	0.917	0.665	0.771
1000	0.2%	69.7%	5.4%	7.7%	0.0%	17.1%	0.938	0.829	0.880
2000	0.9%	91.6%	0.0%	0.0%	0.0%	7.5%	1.000	0.925	0.961
3000	0.6%	92.7%	0.0%	0.0%	0.0%	6.7%	1.000	0.933	0.965

Appendix

Table 10: In-domain results of SpeciaRL-7B trained with different dataset sizes sampled from CUB, and evaluated with CUB test set.

	Prediction categorization						Metrics
Sample size	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
100	2.5%	64.9%	6.8%	7.6%	0.2%	18.0%	0.930	0.820	0.872
1000	3.2%	66.5%	6.2%	7.9%	0.0%	16.2%	0.933	0.838	0.883
2000	6.0%	64.8%	6.2%	6.4%	0.1%	16.6%	0.941	0.834	0.884
3000	5.6%	63.4%	5.1%	10.7%	0.0%	15.2%	0.920	0.848	0.883

Appendix

Table 11: Out-of-domain results of SpeciaRL-7B trained with different dataset sizes sampled from CUB. Results are averaged over fine-grained datasets.

	CUB			Fine-grained			Very fine-grained
	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
CUB	1.000	0.933	0.965	0.920	0.848	0.833	0.818	0.855	0.830
Mixed	0.995	0.889	0.939	0.963	0.878	0.918	0.863	0.860	0.852

Appendix

Table 12: Comparison between SpeciaRL trained on a single domain (CUB) versus a mixture of samples from all available domains.

	Prediction categorization						Metrics
RL method	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
GRPO [ 40 ]	4.6%	52.2%	5.0%	16.2%	0.0%	21.5%	0.875	0.785	0.825
SpeciaRL (GRPO)	5.6%	63.4%	5.1%	10.7%	0.0%	15.2%	0.920	0.848	0.883
Dr.GRPO [ 32 ]	8.6%	59.3%	6.5%	5.3%	0.2%	20.1%	0.942	0.799	0.864
SpeciaRL (Dr.GRPO)	6.6%	64.4%	6.0%	4.9%	0.0%	18.2%	0.951	0.818	0.879
DAPO [ 55 ]	7.3%	61.0%	7.1%	3.2%	0.4%	21.0%	0.951	0.790	0.862

Appendix

Table 13: SpeciaRL compared to static reward rft across different on-policy RL algorithms. Best in bold . Results are averaged over fine-grained datasets.

	Fine-grained		Very fine-grained
	AR	$\kappa$	AR	$\kappa$
Qwen3-30B	0.90	0.84	0.92	0.82
Llama3-7B	0.75	0.64	0.69	0.48
$\mathtt{P}_{j}$ ( $v_{1}$ )	0.94	0.91	0.95	0.89
$\mathtt{P}_{j}$ ( $v_{2}$ )	0.91	0.87	0.91	0.80
$\mathtt{P}_{j}$ ( $v_{3}$ )	0.90	0.85	0.90	0.76

Appendix

Table 14: LLM-as-a-judge validation across different models and prompt variants.

	Prediction categorization						Metrics
$\rho_{e}$	$S^{+}$	$S$	$S^{-}$	$G$	$A$	$W$	spec. $\uparrow$	corr. $\uparrow$	HM $\uparrow$
0%	3.2%	66.5%	6.2%	7.9%	0.0%	16.2%	0.933	0.838	0.883
5%	5.6%	65.3%	6.4%	5.4%	0.0%	17.3%	0.946	0.827	0.882
10%	3.3%	64.7%	6.6%	8.4%	0.0%	16.9%	0.928	0.831	0.877
25%	2.0%	64.5%	6.6%	10.5%	0.0%	16.4%	0.916	0.836	0.874

Appendix

Table 15: Sensitivity of SpeciaRL to LLM-judge error. Results are averaged over fine-grained datasets.

Panel of Figure 16 — Figure 16: Generated Prompt for the LLM-as-a-judge verifier.

Panel of Figure 17 — Figure 17: Generated Prompt for the LLM-as-a-judge verifier.

Panel of Figure 18 — Figure 18: Generated Prompt for the LLM-as-a-judge verifier.

Image Classification

Image classification assigns semantic labels to images, enabling recognition of objects, scenes, conditions, and fine-grained categories.

See all 391 papers

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
TaxonRL: Reinforcement Learning with Intermediate Rewards for Interpretable Fine-Grained Visual Reasoning
Mar 4, 2026Maximilian von Klinski, Maximilian SchallVisual Reasoning Fine-Grained Image Classification
Hasso Plattner Institute University of Potsdam, Germany
Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding
Mar 4, 2026Junhan Chen, Zilu Zhou, Yujun Tong +3Fine-Grained Visual Perception Visually Grounded Reasoning
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China

Feb 26, 2026J/K move · Enter open · S save

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li+5

the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China+3

VLM Reasoning Large Vision-Language Models Fine-Grained Visual Perception Efficient VLM Inference

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately

4.0\times

inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.

Abstract

Image Classification

Explore similar work

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

TaxonRL: Reinforcement Learning with Intermediate Rewards for Interpretable Fine-Grained Visual Reasoning

Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding