Fine-grained visual understanding requires multimodal large language models (MLLMs) to recognize small visual entities and understand their relationships. Scene graphs offer a natural way to represent this information, organizing visual entities as nodes and their relationships as edges. However, we identify two challenges: existing scene graph benchmarks provide insufficient coverage of fine-grained entities and the relationships among them, and existing methods for fine-grained visual understanding have yet to effectively leverage such graph-structured information in MLLMs. To address these challenges, we introduce Fine-Grained Multimodal Scene Graph Benchmark (FG-GraphBench) and FiGRA. FG-GraphBench is a benchmark comprising three complementary tasks: fine-grained visual question answering (FG-QA), image captioning (FG-Caption), and image description generation (FG-Description). We develop a two-stage construction pipeline that discovers and grounds fine-grained entities and establishes their relations to construct the fine-grained multimodal scene graphs (MSGs) in FG-GraphBench. Building on these MSGs, we propose FiGRA, a Fine-grained Graph Representation Augmentation method for MLLMs. FiGRA adopts a two-stage training strategy. It encodes multimodal entity and relation features as continuous graph tokens, and integrates them into the MLLM. Experiments show that FiGRA significantly outperforms strong baselines across three tasks of FG-GraphBench. Further analyses confirm the effectiveness of FG-GraphBench and highlight the importance of preserving fine-grained entity and relation information for visual understanding.
Figures & tables
Figure 1: Overview of the FG-GraphBench construction pipeline.
Dataset / Question Type
Graphs
Avg. Nodes
Avg. Edges
FG-Description (AG)
9,595
16.19
35.08
FG-Caption
14,026
35.05
101.32
FG-QA (Overall)
43,384
28.67
81.22
queryRel
24,824
28.93
81.52
verifyRel
6,977
28.23
80.04
existRel
6,974
28.15
80.24
Table 1: Statistics of FG-GraphBench. We report the number of graphs and the average numbers of nodes and edges per graph.
Figure 2: Overview of FiGRA and its two-stage training strategy.
Method
LLM
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
ROUGE-L
METEOR
CLIP-Score
Open-source Multimodal Large Language Models
1
LLaVA-1.5-7B [ 27 ]
7B
0.6754
0.4888
0.3456
0.2494
0.1854
0.3561
0.3992
0.5655
2
ALLaVA-3B [ 2 ]
3B
0.6385
0.4363
0.2887
0.1953
0.1911
0.5435
0.3863
0.5898
3
Llama-3.2-11B-Vision-Instruct [ 10 ]
11B
0.6624
0.4656
0.3215
0.2269
0.2448
0.5691
0.4241
0.6042
4
InternVL3.5-8B [ 44 ]
8B
0.7203
0.5394
0.3976
0.2985
0.2892
0.5963
0.4542
0.6049
5
Qwen2-VL-7B-Instruct [ 43 ]
7B
0.6969
0.5135
0.3692
0.2694
0.2291
0.3723
0.4207
0.6020
Table 2: Results on the FG-Description. Methods evaluated without fine-tuning, including training-free methods and some pretrained models used directly for inference, are highlighted in gray . Best results are highlighted in bold , while second-best results are underlined .
Table 3: Performance comparison on the FG-QA benchmark. Results are reported in percentage (%). Alt. Acc. denotes Alternative Answer Set Accuracy, which measures the proportion of questions for which the predicted answer matches at least one of the alternative answers provided in the benchmark.
Method
LLM
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
ROUGE-L
METEOR
CLIP-Score
Open-source Multimodal Large Language Models
1
LLaVA-1.5-7B [ 27 ]
7B
0.7294
0.5493
0.3938
0.2774
0.5381
0.4114
0.4164
0.6890
2
ALLaVA-3B [ 2 ]
3B
0.7009
0.5199
0.3643
0.2526
0.5079
0.4175
0.4085
0.6786
3
Llama-3.2-11B-Vision-Instruct [ 10 ]
11B
0.7476
0.5837
0.4160
0.3108
0.5554
0.4641
0.4688
0.6857
4
InternVL3.5-8B [ 44 ]
8B
0.7334
0.5790
0.4080
0.3047
0.5203
0.4476
0.4742
0.6967
5
Qwen2-VL-7B-Instruct [ 43 ]
7B
0.7380
0.5611
0.4117
0.3263
0.6041
0.4294
0.4608
0.6993
Table 4: Results on the FG-Caption. Methods evaluated without fine-tuning, including training-free methods and some pretrained models used directly for inference, are highlighted in gray . Best results are highlighted in bold , while second-best results are underlined .
Configuration
FG- Description BLEU-4
FG- Caption BLEU-4
FG-QA Overall Acc. (%)
A. w/o Shortest-Path Prediction
0.3594
0.3228
58.20
B. w/o Pre-training
0.3522
0.3217
57.64
C. w/o Subgraph Retrieval
0.3611
0.3002
53.09
D. w/o Graph Encoder
0.3509
0.3257
57.91
E. Full FiGRA
0.3611
0.3293
59.59
Table 5: Ablation study of FiGRA on the three tasks of FG-GraphBench. BLEU-4 is reported for FG-Description and FG-Caption, while overall accuracy (%) is reported for FG-QA.
Figure 3: Effectiveness of FG-GraphBench. (a) Graph linearization evaluation experiment on FG-Description. (b) Node ablation experiment on FG-Description (left) and FG-QA (right).
Figure 4: Performance comparison of different graph information injection methods.
Method
gr.
ge.
sa.
Avg. input tokens ↓
training hours/epoch ↓
Inference hours ↓
Graph Linearization
✓
–
–
1063
4.09
5.48
Compressed Graph Token
✓
✓
✓
668
3.20
5.00
Graph Semantic Sequence (Ours)
✓
✓
✓
782
3.35
5.20
Table 6: Computational cost and module configurations of different graph information injection methods. gr. , ge. , and sa. denote graph retrieval, graph encoder, and semantic adapter, respectively.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Original
Ours
Relative change
Avg. nodes (original/ours)
FG-Description
35,942
155,373
+332.29%
3.75 / 16.19
FG-Caption
228,671
491,677
+115.02%
16.30 / 35.05
FG-QA
740,088
1,244,016
+68.09%
17.06 / 28.67
Appendix
Table 7: Comparison of node counts between the original scene graphs and the graphs constructed by our data construction pipeline. Relative change is calculated with respect to the original count. Average node counts are reported per graph.
Dataset
Original
Ours
Relative change
Avg. edges (original/ours)
FG-Description
84,410
336,586
+298.75%
8.80 / 35.08
FG-Caption
685,803
1,421,056
+107.21%
48.90 / 101.32
FG-QA
2,320,725
3,523,486
+51.83%
53.49 / 81.22
Appendix
Table 8: Comparison of edge counts between the original scene graphs and the graphs constructed by our data construction pipeline. Relative change is calculated with respect to the original count. Average edge counts are reported per graph.
Question type
Questions
1-hop
2-hop
3-hop
queryRel
24,824
5,665 (22.82%)
5,837 (23.51%)
13,322 (53.67%)
verifyRel
6,977
522 (7.48%)
918 (13.16%)
5,537 (79.36%)
existRel
6,974
1,049 (15.04%)
842 (12.07%)
5,083 (72.89%)
chooseRel
4,609
4,609 (100.00%)
0 (0.00%)
0 (0.00%)
All
43,384
11,845 (27.30%)
7,597 (17.51%)
23,942 (55.19%)
Appendix
Table 9: Reasoning-hop distribution of FG-QA . Percentages are computed within each question type.
Figure 5: Question type and reasoning-hop distributions in FG-QA (a), and the composition of our fine-grained scene graph nodes and edges (b). The inner ring in (a) shows the proportions of question types, the outer ring shows the hop distribution within each type.
Question type
Questions
Yes
No
verifyRel
6,977
3,422 (49.05%)
3,555 (50.95%)
existRel
6,974
3,422 (49.07%)
3,552 (50.93%)
Appendix
Table 10: Yes/no answer distributions in FG-QA . Percentages are computed within each question type.
Top- K categories
Questions covered
Coverage (%)
Top-1
739
2.98
Top-5
2,251
9.07
Top-10
3,679
14.82
Top-20
5,576
22.46
Top-50
8,497
34.23
Top-100
11,446
46.11
Appendix
Table 11: Cumulative coverage of the most frequent answer categories in queryRel.
Figure 6: Answer distribution of queryRel in FG-QA. (a) shows the frequencies of the 20 most common answers. (b) shows the cumulative coverage of FG-QA queryRel answers.
Dataset
Train
Validation
Test
FG-Description
5,757
1,918
1,920
FG-Caption
8,415
2,805
2,806
FG-QA
26,030
8,676
8,678
Appendix
Table 12: Dataset split statistics.
Dataset
Group
#Samples
Accentity
Accrelation
FG-QA (GQA)
Simple
50
90.1%
87.7%
High-risk
50
89.3%
84.2%
Overall
100
89.6%
85.6%
FG-Description (AG)
Simple
24
87.4%
88.9%
High-risk
50
88.1%
82.0%
Overall
74
88.0%
83.4%
Appendix
Table 13: Data quality check results for newly introduced entities and relations in FG-GraphBench. Overall accuracies are computed by aggregating the inspected entities or relations across the two groups.
Dataset
Group
#Samples
Accfg-entity
Accfg-relation
FG-QA (GQA)
Simple
50
85.4%
82.6%
High-risk
50
86.3%
81.7%
Overall
100
86.0%
82.0%
FG-Description (AG)
Simple
24
84.9%
83.9%
High-risk
50
85.1%
78.2%
Overall
74
85.1%
79.8%
Appendix
Table 14: Data quality check results for fine-grained entities and relations. Fine-grained entities are defined by a mask area smaller than 1% of the image area, and fine-grained relations are edges involving at least one such entity.
Table 15: Prompt used for first-round fine-grained entity discovery.
Table 16: Prompt used for second-round global entity discovery.
Table 17: Prompt used for candidate name selection.
Table 18: Prompt for verifying a name against a mask.
Table 19: Prompt used for supervised fine-tuning of FiGRA.
Table 20: Prompt used for shortest-path prediction during pre-training.
Hyperparameter
Value
Model Architecture
Graph hidden dimension ( d )
768
MLLM token embedding dimension ( dMLLM )
4096
Graph Transformer layers ( L )
2
Attention heads per layer
8
Graph dropout
0.1
Appendix
Table 21: Key hyperparameters for graph pre-training.
Hyperparameter
Value
Model Architecture
Graph hidden dimension ( d )
768
MLLM token embedding dimension ( dMLLM )
4096
Graph Transformer layers ( L )
2
Attention heads per layer
8
Graph dropout
0.1
Appendix
Table 22: Key hyperparameters for supervised fine-tuning. Triple retrieval is applied to FG-Caption and FG-QA.
Figure 7: Case Study of FiGRA and LLaVA-1.5-7B on FG-GraphBench. Green text highlights fine-grained details captured by FiGRA that are missing from LLaVA’s answers. Red text indicates incorrect details generated by the model.
Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.
Zhiwei Yang, Yuanchen Wu, Nan Zhang +3
Tencent Youtu Lab, Shanghai, China · Fudan University, Shanghai, China
Recent Multimodal Large Language Models (MLLMs) have achieved remarkable progress across diverse vision-language tasks, creating an urgent need for more challenging benchmarks. Yet existing evaluations still provide limited insight into whether these models can truly reason over structured visual information. Visual Graph Reasoning (VGR) offers a compelling testbed for this challenge, requiring models to integrate perception, structural understanding, and multi-step reasoning over graph-based visual inputs. However, prior VGR benchmarks often reduce the task to visual perception followed by text-based reasoning, restrict evaluation to single-image settings, rely on answer-only metrics, and underrepresent realistic graph-centric scenarios. To bridge the gap, we introduce GraphVerse, a unified benchmark that jointly evaluates perception, visual reasoning, and text-based graph reasoning in MLLMs under both single-image and paired-image settings. At its core is a suite of Graph-centric Image Editing (GIE) strategies that modify graph images while preserving their semantics, turning them into active tests of visual reasoning. We further propose VGR-Score, a process-sensitive metric that evaluates reasoning quality beyond final-answer accuracy. Extensive experiments reveal several key limitations of current MLLMs in VGR, while also validating the effectiveness of GIE strategies and the transferability of GraphVerse to broader multimodal reasoning capabilities. The code is available at https://github.com/sunyuanfu/GraphVerse.
Yuanfu Sun, Yuanhang Ren, Kang Li +5
New York University · Sensetime Research · Tsinghua University +3
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.
Jiajin Liu, Dongzhe Fan, Chuanhao Ji +2
NYU Shanghai · New York University · East China Normal University +1