Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R3, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R3 achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.
Figures & tables
Figure 1: Pipeline comparison of Scene Graph Generation between the two-stage classification framework and the end-to-end generative MLLM method.
Figure 2: Overview of the SGG-R 3 framework. The “R 3 ” denotes three key contributions: R elation augmentation, structured R easoning, and R eward alignment. First, candidate relations generated by the Qwen2.5-VL-32B with CoT prompt are filtered via Sentence-BERT embedding similarity against the original data. The model is then supervised fine-tuned on the CoT-formatted augmented data with CoT prompt, followed by reward-driven reinforcement learning aligned with the original dataset.
Figure 3: Coarse-grained semantic clustering reward. A triplet is matched if its semantic embedding aligns with any ground-truth cluster centroid beyond a threshold, relaxing the strict matching requirement.
Method
Params
VG150 Dataset
Recall
mRecall
Mean
zsRecall
Non-VLM Methods
MOTIFS
–
30.3
6.6
18.5
0.2
VCTree
–
31.3
8.0
19.7
0.7
GPS-Net
–
31.7
9.8
20.8
–
PE-Net
–
35.2
14.5
24.9
3.6
Table 1: Performance ( % ) comparison of our proposed method against non-VLM and VLM-based baselines on the VG150 test set. Bold text indicates best performance, underlined text indicates second best. The “Mean” denotes the average of Recall and mean Recall. † and ‡ denote Qwen2-VL and Qwen2.5-VL as the base model respectively.
Method
Params
PSG Dataset
Recall
mRecall
Mean
zsRecall
Non-VLM Methods
MOTIFS
–
22.0
9.1
15.6
–
VCTree
–
20.6
9.7
15.2
–
GPS-Net
–
20.6
7.2
13.9
–
PSGFormer
–
21.0
19.8
20.4
4.9
Table 2: Performance ( % ) comparison of our proposed method against non-VLM and VLM-based baselines on the PSG test set. Bold text indicates best performance, underlined text indicates second best. The “Mean” denotes the average of Recall and mean Recall. † and ‡ denote Qwen2-VL and Qwen2.5-VL as the base model respectively.
Method
Training
Params
VG150 Dataset
PSG Dataset
Failure Rate
Recall
mRecall
Mean
Failure Rate
Recall
mRecall
Mean
Baselines
R1-SGG †
SFT
2B
75.01
4.82
1.04
2.93
9.79
21.83
13.79
17.81
R1-SGG ‡
SFT
3B
63.16
6.60
1.61
4.11
6.30
22.63
14.60
18.62
R1-SGG †
SFT
7B
39.54
9.62
3.30
6.46
0.96
24.73
17.11
20.92
R1-SGG †
SFT + RL
2B
0.10
21.09
7.48
14.29
2.70
38.49
31.21
34.85
Table 3: Performance ( % ) comparison with MLLM-based end-to-end SGG baselines on the VG150 validation set and PSG test set. The “Mean” denotes the average of Recall and mean Recall. † and ‡ denote Qwen2-VL and Qwen2.5-VL as the base model, respectively.
Figure 4: Quantitative analysis of the average number of relations generated per image on the VG150 and PSG test sets: SFT, SFT + RL; with/without relation augmentation (RA).
Method
Training
Params
VG150
PSG
Recall
mRecall
Recall
mRecall
Open-sourced MLLMs
Qwen2.5-VL
zero-shot
32B
24.84
25.39
35.66
39.52
Baselines
R1-SGG †
SFT
2B
10.72
14.76
47.43
46.72
R1-SGG ‡
SFT
3B
14.98
18.89
51.97
49.94
Table 4: Object detection performance ( % ) comparison of our method against the baselines on the VG150 and PSG test sets. † and ‡ indicate the use of Qwen2-VL and Qwen2.5-VL as the base models.
Setup
VG150 Dataset
PSG Dataset
RA
DGR
Training
Recall
mRecall
zsRecall
Head
Body
Tail
Recall
mRecall
zsRecall
Head
Body
Tail
Relation Augmentation (RA) Ablation
✕
✕
SFT
12.88
2.41
0.90
13.99
1.13
0.53
22.27
11.45
0.00
22.32
14.92
1.21
✓
✕
SFT
17.74
8.23
2.35
18.67
9.90
4.96
33.31
26.13
0.00
32.24
33.04
16.01
✕
✓
SFT + RL
31.57
12.62
5.03
32.95
14.90
4.76
47.63
38.33
0.00
46.98
43.10
24.77
✓
✓
SFT + RL
35.95
14.78
6.11
36.98
15.58
8.13
52.49
44.30
7.69
50.95
50.84
34.14
Table 5: Performance ( % ) comparison for ablation studies of relation augmentation (RA) and dual-granularity reward (DGR) modules. ✓ indicates the component is enabled, ✕ indicates the component is disabled.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Precision
bf16
Max completion length
2048
Per-device batch size
8
Gradient accumulation steps
2
Scheduler type
Cosine
Temperature
1.0
Appendix
Table 6: Hyperparameter settings for reinforcement learning training.
Category
VG150
PSG
Count
Freq.%
Count
Freq.%
Head
15
94.60
16
93.97
Body
15
3.95
16
5.32
Tail
20
1.45
24
0.71
Appendix
Table 7: Distribution of long-tail predicate types across two datasets. The table shows the number and frequency of predicates in each category: Head (top 30% ), Body (middle 30% ), and Tail (bottom 40% ).
Dataset
O/I
Before RA
After RA
Total
Head
Body
Tail
Total
Head
Body
Tail
Count
Type
Count
Type
Count
Type
Count
Type
Count
Type
Count
Type
Count
Type
Count
Type
VG150
11.7
269196
25151
254656
20592
10645
2981
3985
1578
976465
30928
827105
24742
62374
3320
86986
2866
PSG
11.0
261666
18126
245886
16430
13932
1287
1848
409
810219
23530
742387
20542
55220
2150
12612
838
Appendix
Table 8: Statistics of relation augmentation on VG150 and PSG training sets, with breakdowns for head/body/tail categories. “O/I” denotes average objects per image; “Count” refers to the number of relation triplets; “Type” denotes the number of relation triple combination patterns.
Figure 5: Comparison between original and CoT data. The original data are transformed into our three-stage format through a structured curation process.
Figure 6: Visualization of relation augmentation. (a) Original images with ground-truth object bounding boxes and category labels. (b) Visualization of original sparse relation annotations. (c) Visualization of the augmented relation annotations.
Model
Params
VG150 Dataset
PSG Dataset
Failure Rate
Recall
mRecall
zsRecall
Head
Body
Tail
Failure Rate
Recall
mRecall
zsRecall
Head
Body
Tail
Zero-shot Performance
Qwen2.5-VL
3B
76.95
0.46
0.34
0.12
0.46
0.31
0.13
51.05
3.69
3.69
0.00
2.66
4.15
2.42
Qwen2.5-VL
7B
24.89
3.40
2.38
0.78
3.09
3.26
1.98
8.69
8.78
11.02
0.00
6.82
10.45
7.25
Qwen2.5-VL
32B
9.95
8.40
5.66
1.68
8.20
6.13
4.96
5.67
15.69
17.38
0.00
14.03
18.60
13.90
Fine-tuned Performance
Appendix
Table 9: Performance ( % ) comparison of Qwen2.5-VL and our fine-tuned approach on the VG150 and PSG test sets.
Figure 7: Qualitative comparison of scene graph generation under different fine-tuning strategies. We visualize the ground truth, SFT-only outputs, and SFT + RL outputs. The comparisons highlight the performance gaps in both object detection (a) and relation recognition (b).
Figure 8: Qualitative comparison of scene graph generation between our method and the baseline. We visualize the ground truth, baseline reasoning outputs, and our method’s CoT-based reasoning results. The comparisons highlight the performance gaps in both object detection (a) and relation recognition (b).
Figure 9: Three-stage CoT prompt template for end-to-end scene graph generation. The template is applied to both VG150 and PSG datasets, with the predefined object classes and predicate categories adjusted accordingly for each benchmark.
Figure 10: Two-stage CoT prompt template for relation augmentation. The template is applied to both VG150 and PSG datasets, with the predefined object list and predicate categories adjusted accordingly for each benchmark.
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible relation phrases beyond a fixed predicate set. Existing methods usually treat annotated triplets as positives and all unannotated object-pair relations as negatives. However, scene graph annotations are inherently incomplete: many valid relations are missing, and the same interaction can be described at different granularities, e.g., on, standing on, resting on, and supported by. This issue becomes more severe in open-vocabulary SGG due to the much larger relation space. We propose ReLIC-SGG, a relation-incompleteness-aware framework that treats unannotated relations as latent variables rather than definite negatives. ReLIC-SGG builds a semantic relation lattice to model similarity, entailment, and contradiction among open-vocabulary predicates, and uses it to infer missing positive relations from visual-language compatibility, graph context, and semantic consistency. A positive-unlabeled graph learning objective further reduces false-negative supervision, while lattice-guided decoding produces compact and semantically consistent scene graphs. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that ReLIC-SGG improves rare and unseen predicate recognition and better recovers missing relations.
3D scene graphs provide a structured representation of object entities and their relationships, enabling high-level interpretation and reasoning for robots while remaining intuitively understandable to humans. Existing approaches for 3D scene graph generation typically combine scene reconstruction with graph neural networks (GNNs). However, such pipelines require multi-modal data that may not always be available, and their reliance on heuristic graph construction can constrain the prediction of relationship triplets. In this work, we introduce a Scene Graph Retrieval-Reasoning Model in 3D (SGR3 Model), a training-free framework that leverages multi-modal large language models (MLLMs) with retrieval-augmented generation (RAG) for semantic scene graph generation. SGR3 Model bypasses the need for explicit 3D reconstruction. Instead, it enhances relational reasoning by incorporating semantically aligned scene graphs retrieved via a ColPali-style cross-modal framework. To improve retrieval robustness, we further introduce a weighted patch-level similarity selection mechanism that mitigates the negative impact of blurry or semantically uninformative regions. Experiments demonstrate that SGR3 Model achieves competitive performance compared to training-free baselines and on par with GNN-based expert models. Moreover, an ablation study on the retrieval module and knowledge base scale reveals that retrieved external information is explicitly integrated into the token generation process, rather than being implicitly internalized through abstraction.
Zirui Wang, Ruiping Liu, Yufan Chen +7
Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany. · Shenzhen University, Shenzhen, China. · Hunan University, Changsha, China.
Learning-driven Scene Graph Generation (SGG) models excel on frequent relation types but degrade sharply under annotation sparsity, failing to capture reliable visual commonsense knowledge. We propose a model-agnostic, semantically-guided knowledge refinement framework that systematically mines commonsense-grounded constraints from training data - capturing spatial, functional, and qualitative relational regularities - and uses general declarative commonsense reasoning to correct and refine ranked SGG predictions at inference time. The framework requires no manual rule authoring, no model retraining, and transfers across datasets and architectures. On three standard benchmarks, we obtain consistent improvements over strong baselines, demonstrating that structured visual commonsense reasoning over deep scene semantics is a practical and effective complement to purely learning-based scene graph generation.
Maëlic Neau, Salim Baloch, Jakob Suchan +2
Computing Science Department, Umeå University, Umeå, Sweden · School of Computer Science & Engineering, Constructor University, Bremen, Germany · CoDesign Lab EU., Örebro, Sweden +1