Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as ⟨subject, predicate, object⟩ triplets, and underpin downstream tasks such as video captioning, video question answering, and action analysis. However, end-to-end dynamic scene graph generation (DSGG) methods are closed-set: they recognize only objects and predicates from a fixed training vocabulary and struggle with the long-tailed distribution of rare concepts, severely limiting their real-world applicability. Existing open-vocabulary models typically inherit pretrained large language models, resulting in multi-stage training and inference with substantial cost. We introduce OvDSGG, the first end-to-end framework for open-vocabulary DSGG. OvDSGG builds on top of an open-vocabulary Spatial Backbone and a Temporal Backbone; we further propose a Triplet Feature Extraction Module that bridges them, and a Visual-Language Alignment Module that preserves open-vocabulary recognition by learning an adaptive decision boundary in the joint visual-language feature space, without expensive knowledge distillation in existing methods. We further introduce a rigorous open-vocabulary DSGG benchmark adapted from Action Genome, with disjoint Base/Novel splits for both objects and predicates. OvDSGG significantly outperforms open-vocabulary baselines across all metrics, with zero-shot Recall@K scores 10.0--20.4 percentage point higher than the next-best baseline, while on closed-set DSGG remaining competitive with state-of-the-art models. Code and benchmark are publicly available at https://github.com/jhelsby/OvDSGG/.
Figures & tables
Figure 1: Comparison between existing DSGG methods and our OvDSGG. (a) Closed-vocabulary DSGG fails to detect objects and predicates that are unseen in training data. (b) Per-frame open-vocabulary models do not capture temporal dynamics. (c) Our OvDSGG is an end-to-end, open-vocabulary model for DSGG.
Figure 2: Overview of our OvDSGG architecture. The Spatial Backbone extracts per-frame visual and language features, which are processed by Triplet Feature Extraction to form triplet representations. The Temporal Backbone further aggregates temporal information from reference frames. The Visual–Language Alignment Module uses temporally enhanced triplet features and language embeddings to produce open-vocabulary classification results.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
zR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
With Constr.
No Constr.
Method
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
Ov-OED (spatial) [ 34 ]
22.9
26.4
29.6
21.5
26.7
34.6
11.5
13.6
15.2
14.5
21.4
32.3
2.0
2.8
3.9
3.6
8.3
13.8
Ov-OED (temporal)
23.7
27.8
31.5
21.9
27.5
36.0
11.2
13.5
15.9
14.5
22.0
33.5
0.5
0.7
1.1
3.9
8.6
14.9
OvSGTR [ 4 , 3 ]
12.9
16.4
20.8
12.2
16.8
23.1
7.9
9.8
11.9
9.8
14.2
20.0
3.6
4.8
6.2
3.4
5.1
8.7
OvDSGG (spatial)
32.0
37.6
43.6
32.9
40.5
50.0
14.1
16.6
19.1
19.0
25.7
36.5
15.5
20.4
25.6
17.7
24.6
33.1
Table 1: Comparison with baseline methods on Action Genome for Scene Graph Detection ( SGDET ), in the open-vocabulary setting. Best and second-best are bold and underlined , respectively.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
Method
10
20
50
10
20
50
10
20
50
10
20
50
RelDN [ 40 ]
9.1
9.1
9.1
13.6
23.0
36.6
3.3
3.3
3.3
7.5
18.8
33.7
VCTree [ 32 ]
24.4
32.6
34.7
23.9
35.3
46.8
-
-
-
-
-
-
TRACE [ 33 ]
13.9
14.5
14.5
26.5
35.6
45.3
8.2
8.2
8.2
22.8
31.3
41.8
GPS-Net [ 21 ]
24.7
33.1
35.1
24.4
35.7
47.3
-
-
-
-
-
-
Table 2: Comparison with state-of-the-art DSGG methods on Action Genome for Scene Graph Detection ( SGDET ), in the closed-set setting. Best and second-best are bold and underlined , respectively.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
zR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
With Constr.
No Constr.
Spat.
Trp.
Temp.
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
✓
22.3
28.0
35.7
22.3
29.1
38.1
10.0
12.4
15.7
12.5
17.6
25.9
9.0
12.3
17.9
10.2
15.0
22.6
✓
✓
32.0
37.6
43.6
32.9
40.5
50.0
14.1
16.6
19.1
19.0
25.7
36.5
15.5
20.4
25.6
17.7
24.6
33.1
✓
✓
✓
33.3
40.1
48.1
33.9
41.5
51.0
16.5
19.8
24.0
19.9
25.8
34.8
13.6
18.8
26.6
15.1
21.3
30.9
Table 3: Ablation on Action Genome SGDET , open-vocabulary setting, assessing the contributions of the Spatial (Spat.), Triplet (Trp.) and Temporal (Temp.) modules.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
zR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
With Constr.
No Constr.
Method
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
OvSGTR [ 3 ]
12.9
16.4
20.8
12.2
16.8
23.1
7.9
9.8
11.9
9.8
14.2
20.0
3.6
4.8
6.2
3.4
5.1
8.7
OvDSGG–frozen
3.6
5.2
7.8
3.6
5.3
7.9
1.0
1.5
2.3
1.0
1.5
2.3
3.1
3.8
4.7
3.1
3.8
4.7
OvDSGG (temporal)
33.3
40.1
48.1
33.9
41.5
51.0
16.5
19.8
24.0
19.9
25.8
34.8
13.6
18.8
26.6
15.1
21.3
30.9
Table 4: Learnable vs. frozen classification heads for visual–language alignment, on Action Genome SGDET in the open-vocabulary setting. Best and second-best are bold and underlined , respectively.
Figure 3: Qualitative results for OvDSGG in the open-vocabulary setting on selected Action Genome frames. Base classes are black; novel classes are red. Ground-truth frames are labeled GT1–GT4; corresponding predictions are labeled 1–4 chronologically. Although not present in the GT scene graph, the model detects novel predicates that are reasonable.
Figure A1 : Object and predicate class distribution on our Base/Novel split of Action Genome dataset.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
Spat.
Trp.
Temp.
10
20
50
10
20
50
10
20
50
10
20
50
✓
28.9
35.8
42.9
30.6
39.3
50.0
16.9
22.0
27.0
21.7
31.8
43.8
✓
✓
29.5
36.4
43.3
31.6
40.8
52.0
19.5
24.2
28.3
24.9
33.6
44.9
✓
✓
✓
31.3
38.2
44.7
33.4
42.4
53.2
20.1
25.0
29.3
25.5
35.3
47.1
Table A1 : Module-effectiveness ablation on Action Genome SGDET in the closed-set setting, assessing the contributions of the Spatial (Spat.), Triplet (Trp.) and Temporal (Temp.) modules. Counterpart to Table 3 of the main paper.
Figure A2 : Qualitative results for OvDSGG in the closed-set setting on selected Action Genome frames. Ground-truth frames are labeled GT1–GT4; corresponding predictions are labeled 1–4 chronologically. Although not present in the GT scene graph, the model detects predicates that are plausible. Compare with the open-vocabulary results in the main paper.
Dynamic Scene Graph Generation (DSGG) aims to capture objects and their evolving relations in videos. Despite recent progress, the practicality and quality of generated scene graphs remain limited compared to the rapid advances in Multimodal Large Language Models (MLLMs). In this work, we revisit DSGG from two fundamental perspectives: task setup and model design. From the task setup perspective, we identify two key limitations of the current recall-oriented evaluation protocol: (i) a severe precision-recall trade-off, and (ii) uninformative and redundant relation generation. To better assess practical usefulness, we introduce five additional metrics that provide a more comprehensive evaluation of the quality of generated dynamic scene graphs. From the model design perspective, we explore directly using MLLMs for scene graph generation and establish a strong MLLM-based DSGG baseline through three design changes. First, we replace the conventional bottom-up pipeline with a top-down reason-then-locate strategy. Second, we reformulate frame-wise dynamic graphs as Temporal Relation Set (TRS) prediction, improving both efficiency and performance. Third, we introduce Importance-Aware Finetuning (IAF) to encourage more relevant and diverse relation generation. Extensive experiments on Action Genome, VidVRD, and PVSG show that our approach consistently achieves state-of-the-art performance.
Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible relation phrases beyond a fixed predicate set. Existing methods usually treat annotated triplets as positives and all unannotated object-pair relations as negatives. However, scene graph annotations are inherently incomplete: many valid relations are missing, and the same interaction can be described at different granularities, e.g., on, standing on, resting on, and supported by. This issue becomes more severe in open-vocabulary SGG due to the much larger relation space. We propose ReLIC-SGG, a relation-incompleteness-aware framework that treats unannotated relations as latent variables rather than definite negatives. ReLIC-SGG builds a semantic relation lattice to model similarity, entailment, and contradiction among open-vocabulary predicates, and uses it to infer missing positive relations from visual-language compatibility, graph context, and semantic consistency. A positive-unlabeled graph learning objective further reduces false-negative supervision, while lattice-guided decoding produces compact and semantically consistent scene graphs. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that ReLIC-SGG improves rare and unseen predicate recognition and better recovers missing relations.