Reasoning-Aware Relational Representation Learning for Open-Vocabulary Scene Graph Generation
Organizations: ShanghaiTech University, Shanghai, China · Lingang Laboratory, Shanghai, China · Shanghai Engineering Research Center of Intelligent Vision and Imaging
Abstract
Open-Vocabulary Scene Graph Generation (OV-SGG) requires models to recognize visual relationships beyond the training vocabulary, yet existing methods often rely on dataset-specific object--predicate co-occurrence patterns. Under long-tailed distributions and noisy object localization, such reliance leads to biased, entangled relation representations, limiting both rare-relation recognition and generalization to unseen predicates. To address these challenges, we propose RaRe, a reasoning-aware relational representation learning framework that transforms MLLM-generated chain-of-thought (CoT) rationales into structured relation embeddings. Rather than using CoT only as an intermediate reasoning trace for final prediction, RaRe directly optimizes the rationale-derived representation space to improve inter-relation discriminability. Specifically, RaRe first aligns rationale embeddings with textualized relation descriptions through self-supervised contrastive learning, and then refines rationale generation with a specificity-oriented reinforcement learning objective that encourages semantically object-pair-specific relational evidence. Experiments on VG150 and PSG demonstrate improved category-balanced recognition over existing baselines, while the additional gains from learning vary across datasets and proposal settings. Code available at: https://artanic30.github.io/project_pages/RaRe.
Figures & tables
| Model | Total (Relation) | Base (Relation) | Novel (Relation) | |||
|---|---|---|---|---|---|---|
| R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | |
| OvSGTR [ 4 ] | 20.46/23.86 | 3.91/4.62 | 26.14/30.16 | 4.81/5.60 | 13.45/16.19 | 1.82/2.32 |
| OvSGTR+RAHP [ 23 ] | 21.50/25.74 | 4.51/5.37 | 26.29/30.16 | 5.15/5.94 | 15.59/19.92 | 3.01/4.04 |
| OwSGG(Qwen72b) [ 5 ] | -/- | -/- | -/- | -/- | 2.19/3.06 | 2.19/3.07 |
| OpenSGen [ 15 ] | 18.0/20.5 | 6.2/7.7 | -/- | -/- | 15.7/17.9 | -/- |
| ACC(Swin-B) [ 19 ] † | 24.04 / 28.50 | 4.30/5.11 | 27.41/31.53 | 5.08/5.95 | 19.15 / 23.72 | 2.48/3.17 |
| Model | Total (Relation) | Base (Relation) | Novel (Relation) | |||
|---|---|---|---|---|---|---|
| R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | |
| PGSG [ 22 ] | 26.90/33.90 | 10.80/13.90 | - | - | - | 5.20/7.70 |
| SGTR+ [ 21 ] | 36.32/41.51 | 13.30/17.60 | 40.50/46.67 | 19.57/24.96 | 0.00/0.00 | 0.00/0.00 |
| SGTR+ + RAHP [ 23 ] | 39.92/46.03 | 16.88/22.18 | 41.29/47.65 | 20.51/25.18 | 15.46/20.37 | 11.82/15.46 |
| SVPR [ 11 ] | 33.50/35.90 | 8.30/10.80 | -/- | -/- | -/- | -/- |
| Epic [ 38 ] | -/- | 16.50/21.80 | 28.30/31.10 | -/- | 13.90/18.30 | -/- |
| Model | Total (Relation) | Novel (Relation) | ||
|---|---|---|---|---|
| R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | |
| SGTR+ [ 21 ] | 14.20/18.20 | 6.40/8.40 | –/– | 0.00/0.00 |
| PGSG [ 22 ] | 18.00/20.20 | 13.50/16.40 | –/– | 7.40/11.30 |
| OvSGTR [ 4 ] | 17.76/19.50 | –/– | 6.93/8.08 | –/– |
| ACC [ 19 ] | 20.01/21.71 | –/– | 8.78/9.70 | –/– |
| PGSG+RaRe | 16.34/18.65 | 17.52/20.59 | 9.11/10.34 | 14.76/16.10 |
| # | TP | PR | RL | SSL | ST | Total mR@50/100 | Base mR@50/100 | Novel mR@50/100 |
|---|---|---|---|---|---|---|---|---|
| 1 | 19.42/20.25 | 17.94/19.02 | 22.88/23.13 | |||||
| 2 | ✓ | 19.50/20.80 | 18.38/19.41 | 22.10/24.05 | ||||
| 3 | ✓ | ✓ | 20.68/23.54 | 23.37/25.93 | 14.39/17.95 | |||
| 4 | ✓ | ✓ | ✓ | 20.80/23.25 | 21.39/23.61 | 19.41/22.41 | ||
| 5 | ✓ | ✓ | ✓ | 21.54/24.23 | 23.61/26.08 | 16.72/19.92 | ||
| 6 | ✓ | ✓ | ✓ | ✓ | 27.29/29.14 | 33.11/34.61 | 13.71/16.38 |
| # | RL | SSL | ST | Total mR@50/100 | Novel mR@50/100 |
|---|---|---|---|---|---|
| 1 | 12.56/14.39 | 12.69/14.20 | |||
| 2 | ✓ | 14.02/16.68 | 14.25/15.88 | ||
| 3 | ✓ | ✓ | 15.12/17.04 | 9.47/10.49 | |
| 4 | ✓ | 12.63/14.45 | 12.83/14.31 | ||
| 5 | ✓ | ✓ | 13.89/15.92 | 14.75/ 16.34 | |
| 6 | ✓ | ✓ | ✓ | 17.52/20.59 | 14.76 /16.10 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| MLLM Backbone | Total | Base | Novel | |||
|---|---|---|---|---|---|---|
| R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | |
| Qwen2.5-VL-3B-Instruct | 48.86/52.11 | 17.35/19.96 | 49.29/52.57 | 19.68/22.35 | 9.64/11.29 | 11.92/14.39 |
| Qwen3-VL-4B-Instruct | 52.90/55.68 | 20.68/23.54 | 53.40/56.21 | 23.37/25.93 | 12.71/13.67 | 14.39/17.95 |
| Qwen3-VL-8B-Instruct | 53.13/55.96 | 21.60/23.94 | 53.57/56.42 | 23.63/25.89 | 15.56/17.34 | 16.87/19.41 |
| Qwen3.5-4B | 53.22/56.08 | 21.72/23.96 | 53.69/56.58 | 23.75/26.30 | 13.22/14.34 | 16.98/18.50 |
| RaRe | 49.90/51.33 | 24.62/25.80 | 50.36/51.81 | 25.53/26.45 | 12.75/13.60 | 22.49/24.27 |
| Total mR@50/100 | Base mR@50/100 | Novel mR@50/100 | |
|---|---|---|---|
| 1 | 22.86/24.53 | 24.94/26.60 | 18.02/19.72 |
| 3 | 23.61/25.31 | 24.80/26.48 | 20.82/22.58 |
| 5 | 24.62/25.80 | 25.53 /26.45 | 22.49/24.27 |
| 10 | 23.95/25.48 | 24.94/ 26.69 | 21.65/22.65 |
| Total mR@50/100 | Base mR@50/100 | Novel mR@50/100 | |
|---|---|---|---|
| 0 | 23.03/24.33 | 22.12/23.32 | 25.15/26.71 |
| 0.01 | 24.50/25.65 | 23.54/24.85 | 26.75/27.50 |
| 0.05 | 24.62/25.80 | 25.53/26.45 | 22.49/24.27 |
| 0.1 | 19.39/21.06 | 23.03/24.79 | 10.92/12.37 |
| Method | Total mR@50/100 | Base mR@50/100 | Novel mR@50/100 |
|---|---|---|---|
| Uniform | 24.62/25.80 | 25.53/26.45 | 22.49/24.27 |
| Distance-weighted | 23.60/25.30 | 24.82/26.56 | 20.73/22.35 |
| No novel-prior transfer | 18.99/20.13 | 26.14/27.56 | 2.33/2.78 |
| Novel predicate | Top-5 nearest base predicates |
|---|---|
| across | 1) to, 2) over, 3) in front of, 4) against, 5) behind |
| laying on | 1) on, 2) sitting on, 3) under, 4) covered in, 5) attached to |
| on back of | 1) on, 2) behind, 3) against, 4) over, 5) attached to |
| Total (Relation) | Base (Relation) | Novel (Relation) | ||||
|---|---|---|---|---|---|---|
| Model | R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | R@50/100 | mR@50/100 |
| SFT | 55.92/58.49 | 16.83/18.70 | 56.53/59.13 | 20.69/22.80 | 7.33/8.13 | 7.84/9.14 |
| GRPO | 55.05/57.64 | 20.18/22.52 | 55.60/58.22 | 23.56/26.03 | 10.34/10.97 | 12.30/14.33 |
| RaRe (Ours) | 49.90/51.33 | 24.62/25.80 | 50.36/51.81 | 25.53/26.45 | 12.75/13.60 | 22.49/24.27 |
| Model | Total (Relation) | Base (Relation) | Novel (Relation) | |||
|---|---|---|---|---|---|---|
| R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | R@50/100 | mR@50/100 | |
| SGTR+ | 21.42/26.60 | 9.13/11.58 | 32.18/36.11 | 13.12/16.88 | 0.00/0.00 | 0.00/0.00 |
| SGTR+ + RAHP | 26.65/30.83 | 12.98/ 16.81 | 32.63/36.83 | 13.86/ 17.14 | 14.37/ 21.72 | 9.91/15.04 |
| ACC+Ours | 30.46 / 33.56 | 13.76 /16.62 | 33.79 / 37.39 | 14.30 /17.02 | 15.50 /17.38 | 12.51 / 15.67 |
| Method | Full | Rare | Non-rare |
|---|---|---|---|
| UPT (TF) | 13.64 | 9.66 | 14.85 |
| ADA-CM (ICCV 2023) | 25.19 | 27.24 | 24.58 |
| DA-HOI (ICLR 2026) | 31.50 | - | - |
| Ours | 27.71 | 29.06 | 27.31 |
| Metric | Baseline | +CoT | +CoT+GRPO |
|---|---|---|---|
| Silhouette (Global) | -0.1313 | -0.0935 ( 0.0378) | -0.0924 ( 0.0389) |
| Silhouette (Base) | -0.1040 | -0.0689 ( 0.0351) | -0.0657 ( 0.0383) |
| Silhouette (Novel) | +0.0298 | +0.0514 ( 0.0216) | +0.0511 ( 0.0213) |
| macro_mean_ratio (Global) | 0.0944 | 0.2876 ( 0.1932) | 0.3177 ( 0.2233) |
| macro_mean_ratio (Base) | 0.0863 | 0.2766 ( 0.1903) | 0.3071 ( 0.2208) |
| macro_mean_ratio (Novel) | 0.9683 | 1.4294 ( 0.4611) | 1.4717 ( 0.5034) |
| Model | Average Length (tokens) |
|---|---|
| Qwen3-VL-4B-Instruct | 392.8 |
| RaRe | 297.1 |
| Configuration | Total mR@50/100 | Novel mR@50/100 |
|---|---|---|
| Training-free retrieval | 10.08/12.19 | 10.21/12.27 |
| + RL + SSL + PR | 10.38/12.32 | 6.69/8.02 |
| Variant | Total mR@50/100 | Base mR@50/100 | Novel mR@50/100 |
|---|---|---|---|
| RaRe w/o image | 16.78/18.02 | 18.10/19.50 | 13.71/14.56 |
| RaRe | 24.62/25.80 | 25.53/26.45 | 22.49/24.27 |
| Method | Total mR@100 | Base mR@100 | Novel mR@100 |
|---|---|---|---|
| Direct MLLM output | 12.20 | 13.25 | 9.76 |
| Embedding retrieval | 20.25 | 19.02 | 23.13 |
| Improvement | +8.05 | +5.77 | +13.37 |
| Component | Time (s/image) | Share |
|---|---|---|
| Object detection (ACC) | 1.247 | 22.9% |
| CoT reasoning generation | 3.526 | 64.7% |
| Embedding retrieval | 0.677 | 12.4% |
| Total | 5.450 | 100.0% |
| Method | Head mR@50/100 | Body mR@50/100 | Tail mR@50/100 |
|---|---|---|---|
| ACC | 24.0/28.0 | 0.7/0.9 | 0.0/0.0 |
| RaRe | 22.8/26.7 | 14.3/16.3 | 4.2/5.7 |
| Method | Base pred. (%) | Head (%) | Body (%) | Tail (%) |
|---|---|---|---|---|
| ACC | 64.36 | 99.4 | 0.6 | |
| RaRe | 70.07 | 72.9 | 19.2 | 7.9 |
| Bucket | GT share | Ours@50 | ACC@50 | Ours@100 | ACC@100 | Pred. (Ours/ACC) |
|---|---|---|---|---|---|---|
| Head | 92.4% | 0.083 | 0.124 | 0.112 | 0.159 | 644k/942k |
| Body | 6.9% | 0.062 | 0.000 | 0.074 | 0.000 | 77k/0 |
| Tail | 0.7% | 0.064 | 0.000 | 0.070 | 0.000 | 70k/124 |
| R@50 | R@100 | Prediction (Ours/ACC) | ||||||
| Predicate | Bucket | Test# | Ours | ACC | Ours | ACC | # | % |
| on | Head | 50947 | 0.103 | 0.273 | 0.119 | 0.334 | 409k/827k | 15.47/31.26 |
| of | Head | 17074 | 0.078 | 0.072 | 0.129 | 0.110 | 81k/95k | 3.06/3.58 |
| in | Head | 9138 | 0.068 | 0.027 | 0.089 | 0.031 | 154k/21k | 5.82/0.79 |
| wears | Body | 2059 | 0.000 | 0.000 | 0.000 | 0.000 | 1.7k/0 | 0.06/0.00 |
| riding | Body | 1348 | 0.310 | 0.000 | 0.352 | 0.000 | 26k/0 | 0.99/0.00 |