Organizations: ShanghaiTech University, Shanghai, China · Lingang Laboratory, Shanghai, China · Shanghai Engineering Research Center of Intelligent Vision and Imaging
Open-Vocabulary Scene Graph Generation (OV-SGG) requires models to recognize visual relationships beyond the training vocabulary, yet existing methods often rely on dataset-specific object--predicate co-occurrence patterns. Under long-tailed distributions and noisy object localization, such reliance leads to biased, entangled relation representations, limiting both rare-relation recognition and generalization to unseen predicates. To address these challenges, we propose RaRe, a reasoning-aware relational representation learning framework that transforms MLLM-generated chain-of-thought (CoT) rationales into structured relation embeddings. Rather than using CoT only as an intermediate reasoning trace for final prediction, RaRe directly optimizes the rationale-derived representation space to improve inter-relation discriminability. Specifically, RaRe first aligns rationale embeddings with textualized relation descriptions through self-supervised contrastive learning, and then refines rationale generation with a specificity-oriented reinforcement learning objective that encourages semantically object-pair-specific relational evidence. Experiments on VG150 and PSG demonstrate improved category-balanced recognition over existing baselines, while the additional gains from learning vary across datasets and proposal settings. Code available at: https://artanic30.github.io/project_pages/RaRe.
Figures & tables
Figure 1: Left: our method achieves higher mR@50/100 on both novel and all predicate categories compared with existing OV-SGG methods on SGDet. Right: our method generates more object-pair-specific CoT relational evidence, leading to a better relation prediction than the baseline MLLM.
Figure 2: Training pipeline of our method. Step 1 we introduce Self-supervised training for the reward model. Step 2 we adopt GRPO for optimizing the MLLM. Step 3 we adopt supervised tuning for the final retrieval model.
Figure 3: Inference pipeline of our method. (1)CoT-based Visual Description: The MLLM generates a structural CoT Description as the image’s textual representation.(2)Textual Prototype Construction: Build up a Multi-template textual prototype for each SGG predicate.(3)Embedding Retrieval: We utilize an embedding model to compute the similarities between the CoT and the text prototypes to yield the final results.
Model
Total (Relation)
Base (Relation)
Novel (Relation)
R@50/100
mR@50/100
R@50/100
mR@50/100
R@50/100
mR@50/100
OvSGTR [ 4 ]
20.46/23.86
3.91/4.62
26.14/30.16
4.81/5.60
13.45/16.19
1.82/2.32
OvSGTR+RAHP [ 23 ]
21.50/25.74
4.51/5.37
26.29/30.16
5.15/5.94
15.59/19.92
3.01/4.04
OwSGG(Qwen72b) [ 5 ]
-/-
-/-
-/-
-/-
2.19/3.06
2.19/3.07
OpenSGen [ 15 ]
18.0/20.5
6.2/7.7
-/-
-/-
15.7/17.9
-/-
ACC(Swin-B) [ 19 ] †
24.04 / 28.50
4.30/5.11
27.41/31.53
5.08/5.95
19.15 / 23.72
2.48/3.17
Table 1: OVR-SGG results under SGDet on the full VG150 test set. All results use graph-constraint evaluation. † denotes our re-evaluation of ACC. ACC+RaRe uses the same detected boxes, object labels, and candidate pairs as ACC.
Model
Total (Relation)
Base (Relation)
Novel (Relation)
R@50/100
mR@50/100
R@50/100
mR@50/100
R@50/100
mR@50/100
PGSG [ 22 ]
26.90/33.90
10.80/13.90
-
-
-
5.20/7.70
SGTR+ [ 21 ]
36.32/41.51
13.30/17.60
40.50/46.67
19.57/24.96
0.00/0.00
0.00/0.00
SGTR+ + RAHP [ 23 ]
39.92/46.03
16.88/22.18
41.29/47.65
20.51/25.18
15.46/20.37
11.82/15.46
SVPR [ 11 ]
33.50/35.90
8.30/10.80
-/-
-/-
-/-
-/-
Epic [ 38 ]
-/-
16.50/21.80
28.30/31.10
-/-
13.90/18.30
-/-
Table 2: Experimental results of OVR-SGG on PredCls setting.
Model
Total (Relation)
Novel (Relation)
R@50/100
mR@50/100
R@50/100
mR@50/100
SGTR+ [ 21 ]
14.20/18.20
6.40/8.40
–/–
0.00/0.00
PGSG [ 22 ]
18.00/20.20
13.50/16.40
–/–
7.40/11.30
OvSGTR [ 4 ]
17.76/19.50
–/–
6.93/8.08
–/–
ACC [ 19 ]
20.01/21.71
–/–
8.78/9.70
–/–
PGSG+RaRe
16.34/18.65
17.52/20.59
9.11/10.34
14.76/16.10
Table 3: OVR-SGG results under SGDet on PSG. PGSG+RaRe uses the same object proposals and candidate pairs as PGSG.
Figure 4: Distribution of CoT samples in the similarity plane.
#
TP
PR
RL
SSL
ST
Total mR@50/100
Base mR@50/100
Novel mR@50/100
1
19.42/20.25
17.94/19.02
22.88/23.13
2
✓
19.50/20.80
18.38/19.41
22.10/24.05
3
✓
✓
20.68/23.54
23.37/25.93
14.39/17.95
4
✓
✓
✓
20.80/23.25
21.39/23.61
19.41/22.41
5
✓
✓
✓
21.54/24.23
23.61/26.08
16.72/19.92
6
✓
✓
✓
✓
27.29/29.14
33.11/34.61
13.71/16.38
Table 4: Ablation study of RaRe under PredCls on the full VG150 test set. We evaluate the impact of different components: TP (predicate template for multi-template prototype construction), PR (prior calibration), SSL (self-supervised embedding learning in Stage 1), RL (GRPO in Stage 2), and ST (supervised embedding tuning). We report category-balanced mR@50/100 in the main paper.
#
RL
SSL
ST
Total mR@50/100
Novel mR@50/100
1
12.56/14.39
12.69/14.20
2
✓
14.02/16.68
14.25/15.88
3
✓
✓
15.12/17.04
9.47/10.49
4
✓
12.63/14.45
12.83/14.31
5
✓
✓
13.89/15.92
14.75/ 16.34
6
✓
✓
✓
17.52/20.59
14.76 /16.10
Table 5: Same-proposal SGDet ablation on PSG. All variants use identical PGSG proposals and evaluation protocols, with TP and PR fixed. SSL, RL, and ST denote self-supervised embedding learning, policy optimization, and supervised embedding tuning, respectively.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
MLLM Backbone
Total
Base
Novel
R@50/100
mR@50/100
R@50/100
mR@50/100
R@50/100
mR@50/100
Qwen2.5-VL-3B-Instruct
48.86/52.11
17.35/19.96
49.29/52.57
19.68/22.35
9.64/11.29
11.92/14.39
Qwen3-VL-4B-Instruct
52.90/55.68
20.68/23.54
53.40/56.21
23.37/25.93
12.71/13.67
14.39/17.95
Qwen3-VL-8B-Instruct
53.13/55.96
21.60/23.94
53.57/56.42
23.63/25.89
15.56/17.34
16.87/19.41
Qwen3.5-4B
53.22/56.08
21.72/23.96
53.69/56.58
23.75/26.30
13.22/14.34
16.98/18.50
RaRe
49.90/51.33
24.62/25.80
50.36/51.81
25.53/26.45
12.75/13.60
22.49/24.27
Appendix
Table 6: PredCls results of four off-the-shelf MLLM backbones using TP+PR without parameter updates, with trained RaRe included as a reference. PR uses base-training statistics.
K
Total mR@50/100
Base mR@50/100
Novel mR@50/100
1
22.86/24.53
24.94/26.60
18.02/19.72
3
23.61/25.31
24.80/26.48
20.82/22.58
5
24.62/25.80
25.53 /26.45
22.49/24.27
10
23.95/25.48
24.94/ 26.69
21.65/22.65
Appendix
Table 7: Sensitivity analysis of the number of transferred base predicates K . All results are reported under the PredCls setting with η=0.05 and uniform averaging transfer.
η
Total mR@50/100
Base mR@50/100
Novel mR@50/100
0
23.03/24.33
22.12/23.32
25.15/26.71
0.01
24.50/25.65
23.54/24.85
26.75/27.50
0.05
24.62/25.80
25.53/26.45
22.49/24.27
0.1
19.39/21.06
23.03/24.79
10.92/12.37
Appendix
Table 8: Sensitivity analysis of the subject–object prior weight η . All results are reported under the PredCls setting with K=5 and uniform averaging transfer.
Method
Total mR@50/100
Base mR@50/100
Novel mR@50/100
Uniform
24.62/25.80
25.53/26.45
22.49/24.27
Distance-weighted
23.60/25.30
24.82/26.56
20.73/22.35
No novel-prior transfer
18.99/20.13
26.14/27.56
2.33/2.78
Appendix
Table 9: Comparison of novel-predicate prior transfer strategies. All results are reported under the PredCls setting with K=5 and η=0.05 .
Novel predicate
Top-5 nearest base predicates
across
1) to, 2) over, 3) in front of, 4) against, 5) behind
laying on
1) on, 2) sitting on, 3) under, 4) covered in, 5) attached to
on back of
1) on, 2) behind, 3) against, 4) over, 5) attached to
Appendix
Table 10: Examples of the top-5 nearest base predicates used for novel-prior transfer under the VG150 PredCls split.
Total (Relation)
Base (Relation)
Novel (Relation)
Model
R@50/100
mR@50/100
R@50/100
mR@50/100
R@50/100
mR@50/100
SFT
55.92/58.49
16.83/18.70
56.53/59.13
20.69/22.80
7.33/8.13
7.84/9.14
GRPO
55.05/57.64
20.18/22.52
55.60/58.22
23.56/26.03
10.34/10.97
12.30/14.33
RaRe (Ours)
49.90/51.33
24.62/25.80
50.36/51.81
25.53/26.45
12.75/13.60
22.49/24.27
Appendix
Table 11: Same-backbone comparison of MLLM optimization strategies under the PredCls setting on the full VG150 test set. GRPO uses a binary predicate-correctness reward adapted from Relation-R1, while RaRe uses the specificity reward in Eq. (8).
Model
Total (Relation)
Base (Relation)
Novel (Relation)
R@50/100
mR@50/100
R@50/100
mR@50/100
R@50/100
mR@50/100
SGTR+
21.42/26.60
9.13/11.58
32.18/36.11
13.12/16.88
0.00/0.00
0.00/0.00
SGTR+ + RAHP
26.65/30.83
12.98/ 16.81
32.63/36.83
13.86/ 17.14
14.37/ 21.72
9.91/15.04
ACC+Ours
30.46 / 33.56
13.76 /16.62
33.79 / 37.39
14.30 /17.02
15.50 /17.38
12.51 / 15.67
Appendix
Table 12: Experimental results of OVR-SGG on SGCls setting.
Method
Full
Rare
Non-rare
UPT (TF)
13.64
9.66
14.85
ADA-CM (ICCV 2023)
25.19
27.24
24.58
DA-HOI (ICLR 2026)
31.50
-
-
Ours
27.71
29.06
27.31
Appendix
Table 13: Training-Free Zero-Shot HOI results on HICO-DET (metric: mAP). All methods are under the fully training-free setting.
Figure 5: t-SNE visualization of CoT representations projected by Baseline (Qwen3-VL-Embedding-2B), and +CoT+GRPO.
Metric
Baseline
+CoT
+CoT+GRPO
Silhouette (Global)
-0.1313
-0.0935 ( ↑ 0.0378)
-0.0924 ( ↑ 0.0389)
Silhouette (Base)
-0.1040
-0.0689 ( ↑ 0.0351)
-0.0657 ( ↑ 0.0383)
Silhouette (Novel)
+0.0298
+0.0514 ( ↑ 0.0216)
+0.0511 ( ↑ 0.0213)
macro_mean_ratio (Global)
0.0944
0.2876 ( ↑ 0.1932)
0.3177 ( ↑ 0.2233)
macro_mean_ratio (Base)
0.0863
0.2766 ( ↑ 0.1903)
0.3071 ( ↑ 0.2208)
macro_mean_ratio (Novel)
0.9683
1.4294 ( ↑ 0.4611)
1.4717 ( ↑ 0.5034)
Appendix
Table 14: Discriminability of Unsupervised-trained Embedding We report Silhouette Score and macro_mean_ratio for Global, Base, and Novel subsets. ↑Δ denotes improvement over the baseline. Baseline uses the off-the-shelf embedding model and CoTs from the unoptimized MLLM. +CoT replaces only the embedding model with its SSL-trained counterpart. +CoT+GRPO retains the same SSL-trained embedding model and uses CoTs from the GRPO-optimized MLLM.
Model
Average Length (tokens)
Qwen3-VL-4B-Instruct
392.8
RaRe
297.1
Appendix
Table 15: Average CoT length on the full VG150 test set under PredCls.
Configuration
Total mR@50/100
Novel mR@50/100
Training-free retrieval
10.08/12.19
10.21/12.27
+ RL + SSL + PR
10.38/12.32
6.69/8.02
Appendix
Table 16: Fixed-proposal SGDet control on the full VG150 test set. Both configurations use identical ACC object proposals and candidate pairs. The RaRe configuration adds RL, SSL, and PR to training-free retrieval; ST is not used.
Variant
Total mR@50/100
Base mR@50/100
Novel mR@50/100
RaRe w/o image
16.78/18.02
18.10/19.50
13.71/14.56
RaRe
24.62/25.80
25.53/26.45
22.49/24.27
Appendix
Table 17: Object-only control under the PredCls setting on the full VG150 test set. Only the image is removed; object labels, boxes, prompt, and inference pipeline are unchanged.
Method
Total mR@100
Base mR@100
Novel mR@100
Direct MLLM output
12.20
13.25
9.76
Embedding retrieval
20.25
19.02
23.13
Improvement
+8.05
+5.77
+13.37
Appendix
Table 18: Training-free comparison between direct MLLM output and embedding retrieval under the PredCls setting on the full VG150 test set. Both variants use the same MLLM and prompt.
Component
Time (s/image)
Share
Object detection (ACC)
1.247
22.9%
CoT reasoning generation
3.526
64.7%
Embedding retrieval
0.677
12.4%
Total
5.450
100.0%
Appendix
Table 19: Inference-time breakdown of the SGDet pipeline on four NVIDIA L40 GPUs, reported per image.
Method
Head mR@50/100
Body mR@50/100
Tail mR@50/100
ACC
24.0/28.0
0.7/0.9
0.0/0.0
RaRe
22.8/26.7
14.3/16.3
4.2/5.7
Appendix
Table 20: Frequency-stratified results on VG150 base predicates under SGDet. Head, body, and tail contain 7, 14, and 14 predicates, respectively, ranked by training frequency.
Method
Base pred. (%)
Head (%)
Body (%)
Tail (%)
ACC
64.36
99.4
0.6
≈0.0
RaRe
70.07
72.9
19.2
7.9
Appendix
Table 21: Prediction distribution across base-predicate frequency groups. Base pred. denotes the share of all top-100 predictions assigned to base predicates. Head, body, and tail shares are normalized within base predictions.
Bucket
GT share
Ours@50
ACC@50
Ours@100
ACC@100
Pred. (Ours/ACC)
Head
92.4%
0.083
0.124
0.112
0.159
644k/942k
Body
6.9%
0.062
0.000
0.074
0.000
77k/0
Tail
0.7%
0.064
0.000
0.070
0.000
70k/124
Appendix
Table 22: Novel-predicate recall and prediction counts by head/body/tail group under SGDet with fixed ACC proposals. All results are reported at mR@50/100 on the full VG test set.
R@50
R@100
Prediction (Ours/ACC)
Predicate
Bucket
Test#
Ours
ACC
Ours
ACC
#
%
on
Head
50947
0.103
0.273
0.119
0.334
409k/827k
15.47/31.26
of
Head
17074
0.078
0.072
0.129
0.110
81k/95k
3.06/3.58
in
Head
9138
0.068
0.027
0.089
0.031
154k/21k
5.82/0.79
wears
Body
2059
0.000
0.000
0.000
0.000
1.7k/0
0.06/0.00
riding
Body
1348
0.310
0.000
0.352
0.000
26k/0
0.99/0.00
Appendix
Table 23: Per-predicate novel recall, test-set frequency, and prediction counts under the VG150 SGDet split with fixed ACC proposals. Pred. # reports the number of predictions, and paired entries correspond to Ours/ACC
Figure 29Figure 30Figure 31Figure 32Figure 33
Figure 6: Failure case of our method. The generated CoT contains hallucinated visual evidence and describes an incorrect object pair, leading to a mismatched relation prediction.
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible relation phrases beyond a fixed predicate set. Existing methods usually treat annotated triplets as positives and all unannotated object-pair relations as negatives. However, scene graph annotations are inherently incomplete: many valid relations are missing, and the same interaction can be described at different granularities, e.g., on, standing on, resting on, and supported by. This issue becomes more severe in open-vocabulary SGG due to the much larger relation space. We propose ReLIC-SGG, a relation-incompleteness-aware framework that treats unannotated relations as latent variables rather than definite negatives. ReLIC-SGG builds a semantic relation lattice to model similarity, entailment, and contradiction among open-vocabulary predicates, and uses it to infer missing positive relations from visual-language compatibility, graph context, and semantic consistency. A positive-unlabeled graph learning objective further reduces false-negative supervision, while lattice-guided decoding produces compact and semantically consistent scene graphs. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that ReLIC-SGG improves rare and unseen predicate recognition and better recovers missing relations.
Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R3, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R3 achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.
Jiaye Feng, Qixiang Yin, Yuankun Liu +2
School of Software and Microelectronics, Peking University, Beijing, China · Zhongguancun Academy, Beijing, China
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible and fine-grained relation phrases beyond a fixed predicate vocabulary. While recent vision-language models greatly expand the semantic coverage of SGG, they also introduce a critical reliability issue: predicted relations may be driven by language priors or object co-occurrence rather than grounded visual evidence. In this paper, we propose an evidence-rounded open-vocabulary SGG framework based on counterfactual relation verification. Instead of directly accepting plausible relation proposals, our method verifies whether each candidate relation is supported by relation-pecific visual, geometric, and contextual evidence. Specifically, we first generate open-vocabulary relation candidates with a vision-language proposer, then decompose predicate phrases into soft evidence bases such as support, contact, containment, depth and state. A relation-conditioned evidence encoder extracts predicate-relevant cues, while a counterfactual verifier tests whether the relation score decreases when necessary vidence is removed and remains stable under irrelevant perturbations. We further introduce contradiction-aware predicate learning and graph-level preference optimization to improve fine-grained discrimination and global graph consistency. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that our method consistently improves standard recall-based metrics, unseen predicate generalization, and counterfactual grounding quality. These results demonstrate that moving from relation generation to relation verification leads to more reliable, interpretable, and evidence-grounded scene graphs.
Suiyang Guang, Chenyu Liu, Ruohan Zhang +1
Institute of Intelligent Vision and Embodied Cognition