Scene Graph Generation (SGG) is a task that encodes visual relationships between objects in images as graph structures. SGG shows significant promise as a foundational component for downstream tasks, such as reasoning for embodied agents. To enable real-time applications, SGG must address the trade-off between performance and inference speed. However, current methods tend to focus on one of the following: (1) improving relation prediction accuracy, (2) enhancing object detection accuracy, or (3) reducing latency, without aiming to balance all three objectives simultaneously. To address this limitation, we build on the powerful Real-time Efficiency and Accuracy Compromise for Tradeoffs in Scene Graph Generation (REACT) architecture and propose REACT++, a new state-of-the-art model for real-time SGG. By leveraging efficient feature extraction and subject-to-object cross-attention within the prototype space, REACT++ balances latency and representational power. REACT++ achieves the highest inference speed among existing SGG models, improving relation prediction accuracy without sacrificing object detection performance. Compared to the previous REACT version, REACT++ is 20% faster with a gain of 10% in relation prediction accuracy on average. The code is available at https://github.com/Maelic/SGG-Benchmark.
Figures & tables
Figure 1: Comparing PE-NET architecture Zheng et al. (2023) (top) with our REACT++ architeture (botton) in Stage 1. The dashed boxes in orange [ - - - ] represent modified or added components, ⨂ denotes element-wise concatenation.
Backbone
Relation Head
mAP@50
Faster-RCNN
-
36.4
Neural-Motifs Zellers et al. (2018)
35.9
PE-NET Zheng et al. (2023)
35.4
Transformer Tang et al. (2020)
34.6
VC-Tree Tang et al. (2019)
32.6
GPS-NET Lin et al. (2020)
31.5
Table 1: Object Detection accuracy for two-stage models, PSG dataset Yang et al. (2022a) .
Figure 2: Our REACT++ architecture is based on REACT Neau et al. (2025) . The dashed boxes in purple [ - - - ] represent modified or added components to REACT. The dashed boxes in orange [ - - - ] represent modified or added components to the original PE-NET (see Figure 1 ). ⨂ denotes element-wise concatenation.
B
D
Relation Head
mR@20/50/100
R@20/50/100
F1@K
mAP
Lat.
Params
CSPDarkNet-53
YOLOV8m-DTS
REACT++ (ours) †
22.2 / 24.9 / 27.1
29.1 / 34.2 / 37.5
28.4
53.1
19.4
35.8M
REACT++ (ours)
22.2 / 24.9 / 27.1
29.1 / 34.2 / 37.5
28.4
53.1
25.9
35.8M
REACT † Neau et al. (2025)
18.3 / 20.0 / 20.8
27.5 / 30.9 / 32.2
23.9
53.1
23.0
43.3M
REACT Neau et al. (2025)
18.3 / 20.1 / 20.9
27.6 / 30.9 / 32.3
23.9
53.1
32.5
43.3M
PE-NET † Zheng et al. (2023)
17.1 / 19.0 / 19.8
28.0 / 31.3 / 32.9
23.2
53.1
51.7
187M
PE-NET Zheng et al. (2023)
17.1 / 19.0 / 19.9
28.0 / 31.3 / 33.0
23.2
53.1
103.0
187M
Table 2: Results on the PSG dataset. † represents evaluation with our DCS strategy. Bold and underlined represent the best and second best for each metric.
B
Relation Head
mR@K
R@K
F1@K
mAP 50
DTS
REACT++
20.7
28.1
23.9
37.2
REACT
18.0
30.9
22.8
37.2
PE-NET
16.0
29.4
20.7
37.2
GPS-NET
9.9
29.2
14.8
37.2
Neural-Motifs
11.1
30.5
16.2
37.2
VCTree
14.9
18.9
16.7
37.2
Table 3: Our architecture REACT++ compared to previous work on: (a) results on IndoorVG and (b) VG150 Xu et al. (2017) ; Krishna et al. (2017) (reported results).
Figure 3: Left - latency comparison of the REACT++ model with the different Feature Extraction components. Right - evolution of the F1@K metric across different stages of training, for the same components.
Method
R@ K
mR@ K
F1@ K
Params (M)
Latency (ms)
Global context (AIFI)
28.13
20.70
23.85
9.82
20.76
No global context
27.84
20.23
23.43
7.85
17.94
Δ (AIFI − no context)
+ 0.29
+ 0.47
+ 0.42
− 1.97
− 2.82
Table 4: Ablation of the AIFI component of REACT++. Params counts only unfrozen relation-head parameters. Metrics are averaged over @20, @50, and @100 thresholds. Latency is the median model forward time over 200 images.
Figure 4: Left: Latency for the REACT++ model using a different number of proposals per image, with batch size 1. Right: Average F1@k for the REACT++ model with different number fo proposals.
Figure 5: Average Recall@K, meanRecall@K, and mAP@50 performance for REACT++ against the corresponding latency using different variants of the YOLO12 model Tian et al. (2025) .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
REACT
REACT++
MLP hidden dim
1024
512
Loss function
CrossEntropy
Logit Adjustment + Focal Loss
Optimizer
SGD
AdamW
Learning Rate
10−2
10−4
Scheduler
ReduceLROnPlateau
CosineAnnealing
Gradient accumulation
1
4
Appendix
Table 5: Main hyperparameter differences between REACT and REACT++ on PSG.
Extractor
mR@K
R@K
F1@K
Lat. (ms)
RoIAlign (baseline)
4.9
21.1
8.0
29.2
DA
18.2
27.4
21.9
19.5
DAP
19.4
27.9
22.9
19.6
DAM
19.1
27.0
22.4
19.6
DAMP (ours)
20.7
28.1
23.8
20.8
Appendix
Table 6: Ablation study on the box feature extractor on the IndoorVG dataset. DA: Detection-Anchored. DAM: DA + multi-scale (P3+P4+P5, no gather). DAMP: DAM + 3×3 Gaussian-weighted neighbourhood gather (ours).
Model
mR@20/50/100
R@20/50/100
F1@K
mAP 50
Lat. (ms)
Params
YOLOv8m
22.2 / 24.9 / 27.0
29.1 / 34.2 / 37.5
28.5
51.1
25.9
35.9M
YOLO12n
17.1 / 20.9 / 22.4
25.6 / 30.0 / 33.2
24.0
38.2
26.6
12.4M
YOLO12s
21.2 / 23.9 / 25.8
27.7 / 32.7 / 36.0
27.2
48.2
27.1
19.2M
YOLO12m
23.5 / 26.4 / 28.3
30.9 / 36.1 / 39.3
30.0
53.5
28.0
30.3M
YOLO12l
23.6 / 26.1 / 27.9
29.5 / 34.9 / 37.8
29.4
56.6
33.3
36.5M
Appendix
Table 7: Comparison of REACT++ with different YOLO backbone sizes on the PSG dataset.
Scene Graph Generation (SGG) aims to extract a detailed graph structure from an image, a representation that holds significant promise as a robust intermediate step for complex downstream tasks like reasoning for embodied agents. However, practical deployment in real-world applications - especially on resource constrained edge devices - requires speed and resource efficiency, challenges that have received limited attention in existing research. To bridge this gap, we introduce DSFlash, a low-latency model for panoptic scene graph generation designed to overcome these limitations. DSFlash can process a video stream at 56 frames per second on a standard RTX 3090 GPU, without compromising performance against existing state-of-the-art methods. Crucially, unlike prior approaches that often restrict themselves to salient relationships, DSFlash computes comprehensive scene graphs, offering richer contextual information while maintaining its superior latency. Furthermore, DSFlash is light on resources, requiring less than 24 hours to train on a single, nine-year-old GTX 1080 GPU. This accessibility makes DSFlash particularly well-suited for researchers and practitioners operating with limited computational resources, empowering them to adapt and fine-tune SGG models for specialized applications.
Julian Lorenz, Vladyslav Kovganko, Elias Kohout +3
Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.
Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R3, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R3 achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.
Jiaye Feng, Qixiang Yin, Yuankun Liu +2
School of Software and Microelectronics, Peking University, Beijing, China · Zhongguancun Academy, Beijing, China