Fine-grained visual understanding is shifting from static classification to knowledge-augmented reasoning, where models must justify as well as recognise. Existing approaches remain limited by closed-set taxonomies and single-label prediction, leading to significant degradation under open-set or context-dependent conditions. We present the Knowledge-Augmented Fine-Grained Reasoning Agent (KFRA), a unified framework that transforms fine-grained perception into evidence-driven reasoning. KFRA operates through a three-stage closed reasoning loop that emulates expert analysis. It first performs open-vocabulary detection and web-scale retrieval to generate category hypotheses. It then conducts discriminative regions localisation by aligning textual knowledge with visual evidence through a global-to-local focusing mechanism. Finally, it integrates all multimodal evidence within a large multimodal model to perform interpretable reasoning. Unlike existing agents that treat retrieval and reasoning as independent processes, KFRA establishes a retrieval-grounding coupling that converts retrieved knowledge into spatially grounded evidence for verification. This design enables factual, interpretable, and task-agnostic reasoning across diverse fine-grained scenarios. To evaluate this capability, we construct FGExpertBench, a benchmark designed to assess reasoning depth and cross-task generalisation across six knowledge dimensions. Extensive experiments demonstrate that KFRA consistently surpasses both standalone large multimodal models and current agent frameworks, achieving up to 19 percent improvement in reasoning accuracy and delivering evidence-grounded interpretability in open-set fine-grained visual understanding.
Figures & tables
Figure 2: Comparison of Fine-Grained Paradigms. KFRA advances fine-grained understanding from static classification to evidence-driven reasoning, leveraging dynamic knowledge and retrieval–grounding coupling to achieve factual, interpretable, and task-agnostic understanding.
Figure 3: Overview of FGExpertBench . The benchmark introduces a structured taxonomy of fine-grained visual understanding tasks that span multiple perception and reasoning dimensions. Each category includes representative examples illustrating its task type, while the overall design enables systematic evaluation of large multimodal models in terms of reasoning depth, cross-task generalisation, and evidence grounding.
Figure 4: The construction pipeline of FGExpertBench . We first collect diverse fine-grained categories followed by retrieving corresponding images through web search. GPT-4o is then used to identify potential professional communities relevant to the image content, and domain experts check these professions while providing the corresponding knowledge scopes they focus on. Based on the curated knowledge, GPT-4o generates domain-oriented question-answer pairs. After data filtering and quality control, the finalized benchmark is produced for evaluating large multimodal models on fine-grained visual understanding.
Dimensions
FOCI Bench
FG BMK
KVG Bench
Naut Data
FGExpert Bench
Object
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
Attribute
✗
\Block [fill=mygreen]1-1 ✓
✗
✗
\Block [fill=mygreen]1-1 ✓
Action
✗
✗
✗
✗
\Block [fill=mygreen]1-1 ✓
Count
✗
✗
✗
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
Reason
✗
✗
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
\Block [fill=mygreen]1-1 ✓
Knowledge
✗
✗
✗
✗
\Block [fill=mygreen]1-1 ✓
Table 1: Comparison with Similar Benchmarks. FGExpertBench provides broader and more comprehensive coverage of fine-grained understanding dimensions than existing benchmarks.
Models
Obj.
Attr.
Act.
Cnt.
Rsn.
Know.
Average
\Block 12-1 Open-Source
Phi-3.5-4B [ 1 ]
36.45
46.94
51.02
47.33
56.00
45.76
47.47
Gemma3-it-4B [ 41 ]
45.81
51.02
55.10
51.33
45.33
54.00
50.43
LLaVA-OneVision-7B [ 23 ]
31.03
36.73
48.98
40.67
34.67
44.27
39.39
LLaVA-Next-8B [ 24 ]
31.53
51.02
57.14
44.00
36.67
42.24
43.77
InternVL3-8B [ 56 ]
42.36
52.04
61.22
58.00
55.33
52.24
53.53
InternVL3.5-8B [ 45 ]
39.41
47.96
65.31
61.33
54.66
51.18
53.31
Table 2: Quantitative results (%) on FGExpertBench. KFRA achieves strong and consistent performance across all dimensions, showing superior fine-grained perception and reasoning ability. Bold indicates the best performance in each column.
Figure 5: Qualitative results of all comparison methods on FGExpertBench . KFRA produces the most accurate answers across diverse fine-grained scenarios, highlighting the strong capability in visual reasoning and multimodal understanding.
Figure 6: Visualisation of the “Fine-Grained Anything” Effect. Starting from coarse detections guided by LMM-generated tags and Grounding-DINO, KFRA performs knowledge-driven reasoning, refining coarse predictions into precise fine-grained categories.
Models
Bird
Car
Dog
Air.
Flwr.
Pet
Avg.
\Block 12-1 Open-Source
Phi-3.5-4B [ 1 ]
40.66
44.31
41.52
30.57
48.02
61.87
44.49
Gemma3-it-4B [ 41 ]
50.05
91.07
66.66
66.82
81.25
86.35
73.70
LLaVA-OneVision-7B [ 23 ]
34.14
61.77
51.99
53.56
51.21
62.06
52.46
LLaVA-Next-8B [ 24 ]
31.58
37.56
29.27
18.36
40.15
46.09
33.84
InternVL3-8B [ 56 ]
51.28
70.80
58.47
57.40
65.47
80.49
63.99
InternVL3.5-8B [ 45 ]
52.50
71.38
52.27
58.78
58.74
78.82
62.13
Table 3: Experimental results (%) on six conventional FGIC datasets. KFRA achieves the competitive performance across all datasets. Bold indicates the best performance in each column.
KR
VS
OD
GF
SR
Acc. %
Imp. %
48.64
-
✓
✓
✓
✓
49.56
+0.92
✓
✓
✓
✓
50.21
+1.57
✓
✓
✓
✓
52.83
+4.19
✓
✓
✓
✓
57.15
+8.51
✓
✓
✓
✓
63.42
+14.78
Table 4: Ablation study of tools within KFRA on FGExpertBench . #Acc. denotes the average accuracy. #Imp. denotes the improvement above baseline. Bold indicates the best performance.
Fine-grained visual reasoning remains challenging for vision-language models, especially when small but critical visual cues are buried in high-resolution images. Existing approaches rely on repeated cropping or test-time visual search to introduce local evidence, but they typically do not explicitly distinguish perception from reasoning. In this paper, we propose Perceive-to-Reason (P2R), a unified framework that formulates fine-grained visual reasoning as a two-stage process: the model first localizes question-relevant evidence as a Perceiver, and then answers the question as a Reasoner based on the annotated image and cropped regions. To better align training with this decoupled formulation, we further introduce Perception-Reasoning Alternating GRPO (PRA-GRPO), a role-aware reinforcement learning strategy that alternates between perception-focused and reasoning-focused updates using only final-answer supervision. Built on top of Qwen3-VL-Instruct-2B/4B/8B, P2R consistently improves performance across model scales. In particular, P2R-4B achieves 93.2% on V-Star, 81.9% on HR-Bench-4K, and 80.5% on HR-Bench-8K, substantially outperforming its corresponding backbone. Further experiments show that the benefits of P2R extend beyond high-resolution benchmarks to broader multimodal reasoning tasks. These results suggest that explicitly decoupling perception from reasoning provides an effective framework for fine-grained visual reasoning.
Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model's capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL.
When video reasoning requires external knowledge, many systems with large multimodal models (LMMs) adopt retrieval augmentation to supply the missing context. Appending textual or multi-clip evidence, however, forces heterogeneous signals into a single attention space. We observe diluted attention and higher cognitive load even on non-long videos. The bottleneck is not only what to retrieve but how to represent and fuse external knowledge with the video backbone.We present Graph-to-Frame RAG (G2F-RAG), a training free and auditable paradigm that delivers knowledge in the visual space. On the offline stage, an agent builds a problem-agnostic video knowledge graph that integrates entities, events, spatial relations, and linked world knowledge. On the online stage, a hierarchical multi-agent controller decides whether external knowledge is needed, retrieves a minimal sufficient subgraph, and renders it as a single reasoning frame appended to the video. LMMs then perform joint reasoning in a unified visual domain. This design reduces cognitive load and leaves an explicit, inspectable evidence trail.G2F-RAG is plug-and-play across backbones and scales. It yields consistent gains on diverse public benchmarks, with larger improvements in knowledge-intensive settings. Ablations further confirm that knowledge representation and delivery matter. G2F-RAG reframes retrieval as visual space knowledge fusion for robust and interpretable video reasoning.
Songyuan Yang, Weijiang Yu, Ziyu Liu +4
National University of Defense Technology · Sun Yat-sen University