DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtime

Julian Lorenz; Vladyslav Kovganko; Elias Kohout; Mrunmai Phatak; Daniel Kienzle; Rainer Lienhart

cs.CVMar 11, 2026

DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtime

Authors: Julian Lorenz, Vladyslav Kovganko, Elias Kohout, Mrunmai Phatak, Daniel Kienzle, Rainer Lienhart

Organizations: University of Augsburg

Abstract

Scene Graph Generation (SGG) aims to extract a detailed graph structure from an image, a representation that holds significant promise as a robust intermediate step for complex downstream tasks like reasoning for embodied agents. However, practical deployment in real-world applications - especially on resource constrained edge devices - requires speed and resource efficiency, challenges that have received limited attention in existing research. To bridge this gap, we introduce DSFlash, a low-latency model for panoptic scene graph generation designed to overcome these limitations. DSFlash can process a video stream at 56 frames per second on a standard RTX 3090 GPU, without compromising performance against existing state-of-the-art methods. Crucially, unlike prior approaches that often restrict themselves to salient relationships, DSFlash computes comprehensive scene graphs, offering richer contextual information while maintaining its superior latency. Furthermore, DSFlash is light on resources, requiring less than 24 hours to train on a single, nine-year-old GTX 1080 GPU. This accessibility makes DSFlash particularly well-suited for researchers and practitioners operating with limited computational resources, empowering them to adapt and fine-tune SGG models for specialized applications.

Figures & tables

Figure 1: Performance comparison between our approach and previous work in terms of performance ( mR@50 ) and latency (ms) on the PSG dataset [ 4 ] .

Figure 2: Overview of the DSFlash architecture for inference. Part A is executed once per image. Part B is executed for each combination of two segmentation masks. We use EoMT [ eomt ] as the segmentation backbone which is kept frozen throughout the whole training. We use the mask embedding module from DSFormer [ 2 ] . The relation predictor head is described in Sec. 3.3 . The red numbers indicate which components are covered in which section.

Panel of Figure 3 — Figure 3: Qualitative comparison of the segmentation masks produced by EoMT with and without upsampling the logits.

Figure 4: Schematic of DSFlash’s gating mechanism and the enforced feature consistency loss during training. Given two segmentation masks and an image, DSFlash computes a class token $x$ using various modules, summarized here as $\Phi$ . To train the consistency loss, DSFlash performs two forward passes through the model head with flipped segmentation masks.

Method	mR@50 $\uparrow$	Latency $\downarrow$	# Params
MotifNet-R50	9.56	100 ms	109M
MotifNet-MD	16.32	504 ms	332M
VCTree-R50	10.14	116 ms	105M
VCTree-MD	17.58	520 ms	327M
HiLo-R50	16.34	277 ms	59M
HiLo-L	19.08	427 ms	230M

Table 1: Performance comparison on the PSG dataset [ 4 ] . All models are evaluated using a batch size of 1 on an RTX 3090 GPU and the SGDet protocol. Models marked with a star (*) use low-resolution segmentation masks, discussed in Sec. 3.2

Figure 5: Impact of the segmentation model’s capability on the final performance. mR@inf [ sgbench ] is the best possible mR@k that a hypothetical perfect PSGG model could achieve, given the extracted segmentation masks from the segmentation model.

Method	mR@50 $\uparrow$	Latency (ms) $\downarrow$	RPS $\uparrow$
Baseline	30.7	445	435
Unified Backbone	25.0	41 (-91%)	5,745
Eff. Mask Embed	25.0	37 (-10%)	7,132
Gated Bidir. Pred. 1	28.8	29 (-22%)	11,491
No Seg. Upscaling	28.5	23 (-21%)	12,928
EoMT-3S	25.1	18 (-22%)	17,897

Table 2: Impact of the optimizations on the overall latency, measured on a NVIDIA GeForce RTX 3090 GPU and a batch size of 1. We also report RPS as the processed relations per second when processing batched data. The rows are read from top to bottom with each row adding an incremental optimization to the evaluated model. The last row is an exception and is an improvement from the row marked with 1 .

Figure 6: Comparison of the effect of low-resolution segmentation masks on the final performance and latency when using different segmentation backbones.

Prune	ToMe	H100	3090	1080	mR@50
$\times$	0%	19 ms	29 ms	230 ms	28.80
✓	0%	20 ms	29 ms	205 ms	26.67
✓	30%	20 ms	30 ms	173 ms	26.51
$\times$	40%	21 ms	29 ms	180 ms	25.82
$\times$	50%	20 ms	29 ms	167 ms	24.87
$\times$	60%	21 ms	29 ms	155 ms	21.93

Table 3: Impact of Token Merging ( ToMe ) and mask-based dynamic patch pruning ( Prune , Sec. 3.4 ) on DSFlash’s latency with batch size 1, measured on a H100, RTX 3090, and GTX 1080 GPU.

Figure 7: Illustration of the impact of multiple masks for the same ground truth mask. The model in figure B predicts multiple very similar masks together with separate relation predictions. However, this gives the model multiple attempts to predict the ground truth relation, essentially ignoring the definition for mR@k . Adapted from [ 2 ] .

Figure 8: Illustration of the mask embedding. Subject, object, and background tokens are learnable tokens that are added to the patch embedding with a weighted sum. The weights are determined from the proportion of the patch area that is covered by the respective segmentation mask. Adapted from [ 2 ] .

Figure 9: DSFormer’s inefficient ratio computation can be represented by a simple average pooling layer.

Method	PredCls	SGDet
DSFlash-S	39.27	26.62
DSFlash-B	41.30	28.80
DSFlash-L	41.69	30.90

Table 4: Scene graph performance, evaluated as mR@50 on the PSG dataset [ 4 ] using Predicate Classification ( PredCls ) and Scene Graph Detection ( SGDet , aka SGGen ).

Panel of Figure 10 — Figure 10: Correlation of scene graph performance ( mR@50 ) and backbone capabilities (measured in mR@inf and panoptic quality).

Figure 11: Comparison of the effect of low-resolution segmentation masks on the final performance and latency when using different segmentation backbones.

Prune	ToMe	H100	3090	1080	mR@50
$\times$	0%	28,663	11,491	671	28.80
✓	0%	26,734	11,933	727	26.67
✓	30%	25,817	11,667	852	26.51
$\times$	40%	26,298	11,154	823	25.82
$\times$	50%	26,475	11,266	845	24.87
$\times$	60%	27,062	11,559	810	21.93

Table 5: Impact of Token Merging ( ToMe ) and mask-based dynamic patch pruning ( Prune , Sec. 3.4 ) on DSFlash’s RPS, measured on a H100, RTX 3090, and GTX 1080 GPU.

Figure 12: Comparison of a model trained with pruned patches and one without when pruning patches during evaluation.

Figure 13: Qualitative results using DSFlash.

Figure 14: Failure cases when using DSFlash. Shown are predictions that are in the top 50 predictions for the respective image. The color in the boxes indicates to which subject the prediction is related to.

Explore similar work

REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation
Mar 6, 2026Maëlic Neau, Zoe FalomirScene Graph Generation
Department of Computing Science, Umeå University, Universitetstorget 4, Umeå, 907 36, Sweden.
SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX
A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models
Mar 20, 2025Xuanming Cui, Jaiminkumar Ashokbhai Bhoi, Chionh Wei Peng +2Scene Graph Generation Fine-Tuning
University of Central Florida · DSO National Laboratories

Mar 6, 2026J/K move · Enter open · S save

REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation

Maëlic Neau, Zoe Falomir

Department of Computing Science, Umeå University, Universitetstorget 4, Umeå, 907 36, Sweden.

Scene Graph Generation

Open paper AskPDF arXiv

Scene Graph Generation (SGG) is a task that encodes visual relationships between objects in images as graph structures. SGG shows significant promise as a foundational component for downstream tasks, such as reasoning for embodied agents. To enable real-time applications, SGG must address the trade-off between performance and inference speed. However, current methods tend to focus on one of the following: (1) improving relation prediction accuracy, (2) enhancing object detection accuracy, or (3) reducing latency, without aiming to balance all three objectives simultaneously. To address this limitation, we build on the powerful Real-time Efficiency and Accuracy Compromise for Tradeoffs in Scene Graph Generation (REACT) architecture and propose REACT++, a new state-of-the-art model for real-time SGG. By leveraging efficient feature extraction and subject-to-object cross-attention within the prototype space, REACT++ balances latency and representational power. REACT++ achieves the highest inference speed among existing SGG models, improving relation prediction accuracy without sacrificing object detection performance. Compared to the previous REACT version, REACT++ is 20% faster with a gain of 10% in relation prediction accuracy on average. The code is available at https://github.com/Maelic/SGG-Benchmark.

Abstract

Explore similar work

REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models