SGG-R$^{\rm 3}$: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation

Jiaye Feng; Qixiang Yin; Yuankun Liu; Tong Mo; Weiping Li

cs.CVMar 9, 2026

SGG-R $^{\rm 3}$ : From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation

Authors: Jiaye Feng, Qixiang Yin, Yuankun Liu, Tong Mo, Weiping Li

Organizations: School of Software and Microelectronics, Peking University, Beijing, China · Zhongguancun Academy, Beijing, China

Abstract

Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R $^{\rm 3}$ , a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R $^{\rm 3}$ achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.

Figures & tables

Figure 1: Pipeline comparison of Scene Graph Generation between the two-stage classification framework and the end-to-end generative MLLM method.

Figure 2: Overview of the SGG-R 3 framework. The “R 3 ” denotes three key contributions: R elation augmentation, structured R easoning, and R eward alignment. First, candidate relations generated by the Qwen2.5-VL-32B with CoT prompt are filtered via Sentence-BERT embedding similarity against the original data. The model is then supervised fine-tuned on the CoT-formatted augmented data with CoT prompt, followed by reward-driven reinforcement learning aligned with the original dataset.

Figure 3: Coarse-grained semantic clustering reward. A triplet is matched if its semantic embedding aligns with any ground-truth cluster centroid beyond a threshold, relaxing the strict matching requirement.

Non-VLM Methods
Method	Params	VG150 Dataset
Method	Params	Recall	mRecall	Mean	zsRecall
MOTIFS	–	$30.3$	$6.6$	$18.5$	$0.2$
VCTree	–	$31.3$	$8.0$	$19.7$	$0.7$
GPS-Net	–	$31.7$	$9.8$	$20.8$	–
PE-Net	–	$35.2$	$14.5$	$24.9$	$3.6$

Table 1: Performance (

\%

) comparison of our proposed method against non-VLM and VLM-based baselines on the VG150 test set. Bold text indicates best performance, underlined text indicates second best. The “Mean” denotes the average of Recall and mean Recall. † and ‡ denote Qwen2-VL and Qwen2.5-VL as the base model respectively.

Non-VLM Methods
Method	Params	PSG Dataset
Method	Params	Recall	mRecall	Mean	zsRecall
MOTIFS	–	$22.0$	$9.1$	$15.6$	–
VCTree	–	$20.6$	$9.7$	$15.2$	–
GPS-Net	–	$20.6$	$7.2$	$13.9$	–
PSGFormer	–	$21.0$	$19.8$	$20.4$	$4.9$

Table 2: Performance (

\%

) comparison of our proposed method against non-VLM and VLM-based baselines on the PSG test set. Bold text indicates best performance, underlined text indicates second best. The “Mean” denotes the average of Recall and mean Recall. † and ‡ denote Qwen2-VL and Qwen2.5-VL as the base model respectively.

Baselines
Method	Training	Params	VG150 Dataset				PSG Dataset
Method	Training	Params	Failure Rate	Recall	mRecall	Mean	Failure Rate	Recall	mRecall	Mean
R1-SGG †	SFT	2B	$75.01$	$4.82$	$1.04$	$2.93$	$9.79$	$21.83$	$13.79$	$17.81$
R1-SGG ‡	SFT	3B	$63.16$	$6.60$	$1.61$	$4.11$	$6.30$	$22.63$	$14.60$	$18.62$
R1-SGG †	SFT	7B	$39.54$	$9.62$	$3.30$	$6.46$	$0.96$	$24.73$	$17.11$	$20.92$
R1-SGG †	SFT + RL	2B	$0.10$	$21.09$	$7.48$	$14.29$	$2.70$	$38.49$	$31.21$	$34.85$

Table 3: Performance (

\%

) comparison with MLLM-based end-to-end SGG baselines on the VG150 validation set and PSG test set. The “Mean” denotes the average of Recall and mean Recall. † and ‡ denote Qwen2-VL and Qwen2.5-VL as the base model, respectively.

Figure 4: Quantitative analysis of the average number of relations generated per image on the VG150 and PSG test sets: SFT, SFT + RL; with/without relation augmentation (RA).

Method	Training	Params	VG150		PSG
Method	Training	Params	Recall	mRecall	Recall	mRecall
Open-sourced MLLMs
Qwen2.5-VL	zero-shot	32B	$24.84$	$25.39$	$35.66$	$39.52$
Baselines
R1-SGG †	SFT	2B	$10.72$	$14.76$	$47.43$	$46.72$
R1-SGG ‡	SFT	3B	$14.98$	$18.89$	$51.97$	$49.94$

Table 4: Object detection performance (

\%

) comparison of our method against the baselines on the VG150 and PSG test sets. † and ‡ indicate the use of Qwen2-VL and Qwen2.5-VL as the base models.

Relation Augmentation (RA) Ablation
Setup			VG150 Dataset						PSG Dataset
RA	DGR	Training	Recall	mRecall	zsRecall	Head	Body	Tail	Recall	mRecall	zsRecall	Head	Body	Tail
✕	✕	SFT	$12.88$	$2.41$	$0.90$	$13.99$	$1.13$	$0.53$	$22.27$	$11.45$	$0.00$	$22.32$	$14.92$	$1.21$
✓	✕	SFT	$17.74$	$8.23$	$2.35$	$18.67$	$9.90$	$4.96$	$33.31$	$26.13$	$0.00$	$32.24$	$33.04$	$16.01$
✕	✓	SFT + RL	$31.57$	$12.62$	$5.03$	$32.95$	$14.90$	$4.76$	$47.63$	$38.33$	$0.00$	$46.98$	$43.10$	$24.77$
✓	✓	SFT + RL	$35.95$	$14.78$	$6.11$	$36.98$	$15.58$	$8.13$	$52.49$	$44.30$	$7.69$	$50.95$	$50.84$	$34.14$

Table 5: Performance (

\%

) comparison for ablation studies of relation augmentation (RA) and dual-granularity reward (DGR) modules. ✓ indicates the component is enabled, ✕ indicates the component is disabled.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Precision	bf16
Max completion length	2048
Per-device batch size	8
Gradient accumulation steps	2
Scheduler type	Cosine
Temperature	1.0

Appendix

Table 6: Hyperparameter settings for reinforcement learning training.

Category	VG150		PSG
Category	Count	Freq.%	Count	Freq.%
Head	$15$	$94.60$	$16$	$93.97$
Body	$15$	$3.95$	$16$	$5.32$
Tail	$20$	$1.45$	$24$	$0.71$

Appendix

Table 7: Distribution of long-tail predicate types across two datasets. The table shows the number and frequency of predicates in each category: Head (top

30\%

), Body (middle

30\%

), and Tail (bottom

40\%

Dataset	O/I	Before RA								After RA
		Total		Head		Body		Tail		Total		Head		Body		Tail
		Count	Type	Count	Type	Count	Type	Count	Type	Count	Type	Count	Type	Count	Type	Count	Type
VG150	11.7	$269\,196$	$25\,151$	$254\,656$	$20\,592$	$10\,645$	$2981$	$3985$	$1578$	$976\,465$	$30\,928$	$827\,105$	$24\,742$	$62\,374$	$3320$	$86\,986$	$2866$
PSG	11.0	$261\,666$	$18\,126$	$245\,886$	$16\,430$	$13\,932$	$1287$	$1848$	$409$	$810\,219$	$23\,530$	$742\,387$	$20\,542$	$55\,220$	$2150$	$12\,612$	$838$

Appendix

Table 8: Statistics of relation augmentation on VG150 and PSG training sets, with breakdowns for head/body/tail categories. “O/I” denotes average objects per image; “Count” refers to the number of relation triplets; “Type” denotes the number of relation triple combination patterns.

Figure 5: Comparison between original and CoT data. The original data are transformed into our three-stage format through a structured curation process.

Panel of Figure 6 — Figure 6: Visualization of relation augmentation. (a) Original images with ground-truth object bounding boxes and category labels. (b) Visualization of original sparse relation annotations. (c) Visualization of the augmented relation annotations.

Model	Params	VG150 Dataset							PSG Dataset
Model	Params	Failure Rate	Recall	mRecall	zsRecall	Head	Body	Tail	Failure Rate	Recall	mRecall	zsRecall	Head	Body	Tail
Zero-shot Performance
Qwen2.5-VL	3B	$76.95$	$0.46$	$0.34$	$0.12$	$0.46$	$0.31$	$0.13$	$51.05$	$3.69$	$3.69$	$0.00$	$2.66$	$4.15$	$2.42$
Qwen2.5-VL	7B	$24.89$	$3.40$	$2.38$	$0.78$	$3.09$	$3.26$	$1.98$	$8.69$	$8.78$	$11.02$	$0.00$	$6.82$	$10.45$	$7.25$
Qwen2.5-VL	32B	$9.95$	$8.40$	$5.66$	$1.68$	$8.20$	$6.13$	$4.96$	$5.67$	$15.69$	$17.38$	$0.00$	$14.03$	$18.60$	$13.90$
Fine-tuned Performance

Appendix

Table 9: Performance (

\%

) comparison of Qwen2.5-VL and our fine-tuned approach on the VG150 and PSG test sets.

Panel of Figure 7 — Figure 7: Qualitative comparison of scene graph generation under different fine-tuning strategies. We visualize the ground truth, SFT-only outputs, and SFT + RL outputs. The comparisons highlight the performance gaps in both object detection (a) and relation recognition (b).

Panel of Figure 8 — Figure 8: Qualitative comparison of scene graph generation between our method and the baseline. We visualize the ground truth, baseline reasoning outputs, and our method’s CoT-based reasoning results. The comparisons highlight the performance gaps in both object detection (a) and relation recognition (b).

Figure 9: Three-stage CoT prompt template for end-to-end scene graph generation. The template is applied to both VG150 and PSG datasets, with the predefined object classes and predicate categories adjusted accordingly for each benchmark.

Figure 10: Two-stage CoT prompt template for relation augmentation. The template is applied to both VG150 and PSG datasets, with the predefined object list and predicate categories adjusted accordingly for each benchmark.

Explore similar work

ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generation
Apr 24, 2026Amir Hosseini, Sara Farahani, Xinyi Li +1Scene Graph Generation Positive-Unlabeled Learning
Amirkabir University of Technology
SGR3 Model: Scene Graph Retrieval-Reasoning Model in 3D
Mar 4, 2026Zirui Wang, Ruiping Liu, Yufan Chen +73D SGG Retrieval-Augmented Generation
Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany. · Shenzhen University, Shenzhen, China. · Hunan University, Changsha, China.
Visual Commonsense Driven Knowledge Refinements for Scene Graph Generation
Jun 4, 2026Maëlic Neau, Salim Baloch, Jakob Suchan +2Scene Graph Generation Multimodal Reasoning
Computing Science Department, Umeå University, Umeå, Sweden · School of Computer Science & Engineering, Constructor University, Bremen, Germany · CoDesign Lab EU., Örebro, Sweden +1

Apr 24, 2026J/K move · Enter open · S save

ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generation

Amir Hosseini, Sara Farahani, Xinyi Li, Suiyang Guang

Amirkabir University of Technology

Scene Graph Generation Positive-Unlabeled Learning

Open paper AskPDF arXiv

Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible relation phrases beyond a fixed predicate set. Existing methods usually treat annotated triplets as positives and all unannotated object-pair relations as negatives. However, scene graph annotations are inherently incomplete: many valid relations are missing, and the same interaction can be described at different granularities, e.g., on, standing on, resting on, and supported by. This issue becomes more severe in open-vocabulary SGG due to the much larger relation space. We propose ReLIC-SGG, a relation-incompleteness-aware framework that treats unannotated relations as latent variables rather than definite negatives. ReLIC-SGG builds a semantic relation lattice to model similarity, entailment, and contradiction among open-vocabulary predicates, and uses it to infer missing positive relations from visual-language compatibility, graph context, and semantic consistency. A positive-unlabeled graph learning objective further reduces false-negative supervision, while lattice-guided decoding produces compact and semantically consistent scene graphs. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that ReLIC-SGG improves rare and unseen predicate recognition and better recovers missing relations.

Abstract

Explore similar work

ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generation

SGR3 Model: Scene Graph Retrieval-Reasoning Model in 3D

Visual Commonsense Driven Knowledge Refinements for Scene Graph Generation