Fine-grained visual understanding requires multimodal large language models (MLLMs) to recognize small visual entities and understand their relationships. Scene graphs offer a natural way to represent this information, organizing visual entities as nodes and their relationships as edges. However, we identify two challenges: existing scene graph benchmarks provide insufficient coverage of fine-grained entities and the relationships among them, and existing methods for fine-grained visual understanding have yet to effectively leverage such graph-structured information in MLLMs. To address these challenges, we introduce Fine-Grained Multimodal Scene Graph Benchmark (FG-GraphBench) and FiGRA. FG-GraphBench is a benchmark comprising three complementary tasks: fine-grained visual question answering (FG-QA), image captioning (FG-Caption), and image description generation (FG-Description). We develop a two-stage construction pipeline that discovers and grounds fine-grained entities and establishes their relations to construct the fine-grained multimodal scene graphs (MSGs) in FG-GraphBench. Building on these MSGs, we propose FiGRA, a Fine-grained Graph Representation Augmentation method for MLLMs. FiGRA adopts a two-stage training strategy. It encodes multimodal entity and relation features as continuous graph tokens, and integrates them into the MLLM. Experiments show that FiGRA significantly outperforms strong baselines across three tasks of FG-GraphBench. Further analyses confirm the effectiveness of FG-GraphBench and highlight the importance of preserving fine-grained entity and relation information for visual understanding.
Figures & tables
Figure 1: Overview of the FG-GraphBench construction pipeline.
Dataset / Question Type
Graphs
Avg. Nodes
Avg. Edges
FG-Description (AG)
9,595
16.19
35.08
FG-Caption
14,026
35.05
101.32
FG-QA (Overall)
43,384
28.67
81.22
queryRel
24,824
28.93
81.52
verifyRel
6,977
28.23
80.04
existRel
6,974
28.15
80.24
Table 1: Statistics of FG-GraphBench. We report the number of graphs and the average numbers of nodes and edges per graph.
Figure 2: Overview of FiGRA and its two-stage training strategy.
Method
LLM
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
ROUGE-L
METEOR
CLIP-Score
Open-source Multimodal Large Language Models
1
LLaVA-1.5-7B [ 27 ]
7B
0.6754
0.4888
0.3456
0.2494
0.1854
0.3561
0.3992
0.5655
2
ALLaVA-3B [ 2 ]
3B
0.6385
0.4363
0.2887
0.1953
0.1911
0.5435
0.3863
0.5898
3
Llama-3.2-11B-Vision-Instruct [ 10 ]
11B
0.6624
0.4656
0.3215
0.2269
0.2448
0.5691
0.4241
0.6042
4
InternVL3.5-8B [ 44 ]
8B
0.7203
0.5394
0.3976
0.2985
0.2892
0.5963
0.4542
0.6049
5
Qwen2-VL-7B-Instruct [ 43 ]
7B
0.6969
0.5135
0.3692
0.2694
0.2291
0.3723
0.4207
0.6020
Table 2: Results on the FG-Description. Methods evaluated without fine-tuning, including training-free methods and some pretrained models used directly for inference, are highlighted in gray . Best results are highlighted in bold , while second-best results are underlined .
Table 3: Performance comparison on the FG-QA benchmark. Results are reported in percentage (%). Alt. Acc. denotes Alternative Answer Set Accuracy, which measures the proportion of questions for which the predicted answer matches at least one of the alternative answers provided in the benchmark.
Method
LLM
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
ROUGE-L
METEOR
CLIP-Score
Open-source Multimodal Large Language Models
1
LLaVA-1.5-7B [ 27 ]
7B
0.7294
0.5493
0.3938
0.2774
0.5381
0.4114
0.4164
0.6890
2
ALLaVA-3B [ 2 ]
3B
0.7009
0.5199
0.3643
0.2526
0.5079
0.4175
0.4085
0.6786
3
Llama-3.2-11B-Vision-Instruct [ 10 ]
11B
0.7476
0.5837
0.4160
0.3108
0.5554
0.4641
0.4688
0.6857
4
InternVL3.5-8B [ 44 ]
8B
0.7334
0.5790
0.4080
0.3047
0.5203
0.4476
0.4742
0.6967
5
Qwen2-VL-7B-Instruct [ 43 ]
7B
0.7380
0.5611
0.4117
0.3263
0.6041
0.4294
0.4608
0.6993
Table 4: Results on the FG-Caption. Methods evaluated without fine-tuning, including training-free methods and some pretrained models used directly for inference, are highlighted in gray . Best results are highlighted in bold , while second-best results are underlined .
Configuration
FG- Description BLEU-4
FG- Caption BLEU-4
FG-QA Overall Acc. (%)
A. w/o Shortest-Path Prediction
0.3594
0.3228
58.20
B. w/o Pre-training
0.3522
0.3217
57.64
C. w/o Subgraph Retrieval
0.3611
0.3002
53.09
D. w/o Graph Encoder
0.3509
0.3257
57.91
E. Full FiGRA
0.3611
0.3293
59.59
Table 5: Ablation study of FiGRA on the three tasks of FG-GraphBench. BLEU-4 is reported for FG-Description and FG-Caption, while overall accuracy (%) is reported for FG-QA.
Figure 3: Effectiveness of FG-GraphBench. (a) Graph linearization evaluation experiment on FG-Description. (b) Node ablation experiment on FG-Description (left) and FG-QA (right).
Figure 4: Performance comparison of different graph information injection methods.
Method
gr.
ge.
sa.
Avg. input tokens ↓
training hours/epoch ↓
Inference hours ↓
Graph Linearization
✓
–
–
1063
4.09
5.48
Compressed Graph Token
✓
✓
✓
668
3.20
5.00
Graph Semantic Sequence (Ours)
✓
✓
✓
782
3.35
5.20
Table 6: Computational cost and module configurations of different graph information injection methods. gr. , ge. , and sa. denote graph retrieval, graph encoder, and semantic adapter, respectively.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Original
Ours
Relative change
Avg. nodes (original/ours)
FG-Description
35,942
155,373
+332.29%
3.75 / 16.19
FG-Caption
228,671
491,677
+115.02%
16.30 / 35.05
FG-QA
740,088
1,244,016
+68.09%
17.06 / 28.67
Appendix
Table 7: Comparison of node counts between the original scene graphs and the graphs constructed by our data construction pipeline. Relative change is calculated with respect to the original count. Average node counts are reported per graph.
Dataset
Original
Ours
Relative change
Avg. edges (original/ours)
FG-Description
84,410
336,586
+298.75%
8.80 / 35.08
FG-Caption
685,803
1,421,056
+107.21%
48.90 / 101.32
FG-QA
2,320,725
3,523,486
+51.83%
53.49 / 81.22
Appendix
Table 8: Comparison of edge counts between the original scene graphs and the graphs constructed by our data construction pipeline. Relative change is calculated with respect to the original count. Average edge counts are reported per graph.
Question type
Questions
1-hop
2-hop
3-hop
queryRel
24,824
5,665 (22.82%)
5,837 (23.51%)
13,322 (53.67%)
verifyRel
6,977
522 (7.48%)
918 (13.16%)
5,537 (79.36%)
existRel
6,974
1,049 (15.04%)
842 (12.07%)
5,083 (72.89%)
chooseRel
4,609
4,609 (100.00%)
0 (0.00%)
0 (0.00%)
All
43,384
11,845 (27.30%)
7,597 (17.51%)
23,942 (55.19%)
Appendix
Table 9: Reasoning-hop distribution of FG-QA . Percentages are computed within each question type.
Figure 5: Question type and reasoning-hop distributions in FG-QA (a), and the composition of our fine-grained scene graph nodes and edges (b). The inner ring in (a) shows the proportions of question types, the outer ring shows the hop distribution within each type.
Question type
Questions
Yes
No
verifyRel
6,977
3,422 (49.05%)
3,555 (50.95%)
existRel
6,974
3,422 (49.07%)
3,552 (50.93%)
Appendix
Table 10: Yes/no answer distributions in FG-QA . Percentages are computed within each question type.
Top- K categories
Questions covered
Coverage (%)
Top-1
739
2.98
Top-5
2,251
9.07
Top-10
3,679
14.82
Top-20
5,576
22.46
Top-50
8,497
34.23
Top-100
11,446
46.11
Appendix
Table 11: Cumulative coverage of the most frequent answer categories in queryRel.
Figure 6: Answer distribution of queryRel in FG-QA. (a) shows the frequencies of the 20 most common answers. (b) shows the cumulative coverage of FG-QA queryRel answers.
Dataset
Train
Validation
Test
FG-Description
5,757
1,918
1,920
FG-Caption
8,415
2,805
2,806
FG-QA
26,030
8,676
8,678
Appendix
Table 12: Dataset split statistics.
Dataset
Group
#Samples
Accentity
Accrelation
FG-QA (GQA)
Simple
50
90.1%
87.7%
High-risk
50
89.3%
84.2%
Overall
100
89.6%
85.6%
FG-Description (AG)
Simple
24
87.4%
88.9%
High-risk
50
88.1%
82.0%
Overall
74
88.0%
83.4%
Appendix
Table 13: Data quality check results for newly introduced entities and relations in FG-GraphBench. Overall accuracies are computed by aggregating the inspected entities or relations across the two groups.
Dataset
Group
#Samples
Accfg-entity
Accfg-relation
FG-QA (GQA)
Simple
50
85.4%
82.6%
High-risk
50
86.3%
81.7%
Overall
100
86.0%
82.0%
FG-Description (AG)
Simple
24
84.9%
83.9%
High-risk
50
85.1%
78.2%
Overall
74
85.1%
79.8%
Appendix
Table 14: Data quality check results for fine-grained entities and relations. Fine-grained entities are defined by a mask area smaller than 1% of the image area, and fine-grained relations are edges involving at least one such entity.
Table 15: Prompt used for first-round fine-grained entity discovery.
Table 16: Prompt used for second-round global entity discovery.
Table 17: Prompt used for candidate name selection.
Table 18: Prompt for verifying a name against a mask.
Table 19: Prompt used for supervised fine-tuning of FiGRA.
Table 20: Prompt used for shortest-path prediction during pre-training.
Hyperparameter
Value
Model Architecture
Graph hidden dimension ( d )
768
MLLM token embedding dimension ( dMLLM )
4096
Graph Transformer layers ( L )
2
Attention heads per layer
8
Graph dropout
0.1
Appendix
Table 21: Key hyperparameters for graph pre-training.
Hyperparameter
Value
Model Architecture
Graph hidden dimension ( d )
768
MLLM token embedding dimension ( dMLLM )
4096
Graph Transformer layers ( L )
2
Attention heads per layer
8
Graph dropout
0.1
Appendix
Table 22: Key hyperparameters for supervised fine-tuning. Triple retrieval is applied to FG-Caption and FG-QA.
Figure 7: Case Study of FiGRA and LLaVA-1.5-7B on FG-GraphBench. Green text highlights fine-grained details captured by FiGRA that are missing from LLaVA’s answers. Red text indicates incorrect details generated by the model.