Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as ⟨subject, predicate, object⟩ triplets, and underpin downstream tasks such as video captioning, video question answering, and action analysis. However, end-to-end dynamic scene graph generation (DSGG) methods are closed-set: they recognize only objects and predicates from a fixed training vocabulary and struggle with the long-tailed distribution of rare concepts, severely limiting their real-world applicability. Existing open-vocabulary models typically inherit pretrained large language models, resulting in multi-stage training and inference with substantial cost. We introduce OvDSGG, the first end-to-end framework for open-vocabulary DSGG. OvDSGG builds on top of an open-vocabulary Spatial Backbone and a Temporal Backbone; we further propose a Triplet Feature Extraction Module that bridges them, and a Visual-Language Alignment Module that preserves open-vocabulary recognition by learning an adaptive decision boundary in the joint visual-language feature space, without expensive knowledge distillation in existing methods. We further introduce a rigorous open-vocabulary DSGG benchmark adapted from Action Genome, with disjoint Base/Novel splits for both objects and predicates. OvDSGG significantly outperforms open-vocabulary baselines across all metrics, with zero-shot Recall@K scores 10.0--20.4 percentage point higher than the next-best baseline, while on closed-set DSGG remaining competitive with state-of-the-art models. Code and benchmark are publicly available at https://github.com/jhelsby/OvDSGG/.
Figures & tables
Figure 1: Comparison between existing DSGG methods and our OvDSGG. (a) Closed-vocabulary DSGG fails to detect objects and predicates that are unseen in training data. (b) Per-frame open-vocabulary models do not capture temporal dynamics. (c) Our OvDSGG is an end-to-end, open-vocabulary model for DSGG.
Figure 2: Overview of our OvDSGG architecture. The Spatial Backbone extracts per-frame visual and language features, which are processed by Triplet Feature Extraction to form triplet representations. The Temporal Backbone further aggregates temporal information from reference frames. The Visual–Language Alignment Module uses temporally enhanced triplet features and language embeddings to produce open-vocabulary classification results.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
zR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
With Constr.
No Constr.
Method
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
Ov-OED (spatial) [ 34 ]
22.9
26.4
29.6
21.5
26.7
34.6
11.5
13.6
15.2
14.5
21.4
32.3
2.0
2.8
3.9
3.6
8.3
13.8
Ov-OED (temporal)
23.7
27.8
31.5
21.9
27.5
36.0
11.2
13.5
15.9
14.5
22.0
33.5
0.5
0.7
1.1
3.9
8.6
14.9
OvSGTR [ 4 , 3 ]
12.9
16.4
20.8
12.2
16.8
23.1
7.9
9.8
11.9
9.8
14.2
20.0
3.6
4.8
6.2
3.4
5.1
8.7
OvDSGG (spatial)
32.0
37.6
43.6
32.9
40.5
50.0
14.1
16.6
19.1
19.0
25.7
36.5
15.5
20.4
25.6
17.7
24.6
33.1
Table 1: Comparison with baseline methods on Action Genome for Scene Graph Detection ( SGDET ), in the open-vocabulary setting. Best and second-best are bold and underlined , respectively.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
Method
10
20
50
10
20
50
10
20
50
10
20
50
RelDN [ 40 ]
9.1
9.1
9.1
13.6
23.0
36.6
3.3
3.3
3.3
7.5
18.8
33.7
VCTree [ 32 ]
24.4
32.6
34.7
23.9
35.3
46.8
-
-
-
-
-
-
TRACE [ 33 ]
13.9
14.5
14.5
26.5
35.6
45.3
8.2
8.2
8.2
22.8
31.3
41.8
GPS-Net [ 21 ]
24.7
33.1
35.1
24.4
35.7
47.3
-
-
-
-
-
-
Table 2: Comparison with state-of-the-art DSGG methods on Action Genome for Scene Graph Detection ( SGDET ), in the closed-set setting. Best and second-best are bold and underlined , respectively.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
zR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
With Constr.
No Constr.
Spat.
Trp.
Temp.
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
✓
22.3
28.0
35.7
22.3
29.1
38.1
10.0
12.4
15.7
12.5
17.6
25.9
9.0
12.3
17.9
10.2
15.0
22.6
✓
✓
32.0
37.6
43.6
32.9
40.5
50.0
14.1
16.6
19.1
19.0
25.7
36.5
15.5
20.4
25.6
17.7
24.6
33.1
✓
✓
✓
33.3
40.1
48.1
33.9
41.5
51.0
16.5
19.8
24.0
19.9
25.8
34.8
13.6
18.8
26.6
15.1
21.3
30.9
Table 3: Ablation on Action Genome SGDET , open-vocabulary setting, assessing the contributions of the Spatial (Spat.), Triplet (Trp.) and Temporal (Temp.) modules.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
zR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
With Constr.
No Constr.
Method
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
10
20
50
OvSGTR [ 3 ]
12.9
16.4
20.8
12.2
16.8
23.1
7.9
9.8
11.9
9.8
14.2
20.0
3.6
4.8
6.2
3.4
5.1
8.7
OvDSGG–frozen
3.6
5.2
7.8
3.6
5.3
7.9
1.0
1.5
2.3
1.0
1.5
2.3
3.1
3.8
4.7
3.1
3.8
4.7
OvDSGG (temporal)
33.3
40.1
48.1
33.9
41.5
51.0
16.5
19.8
24.0
19.9
25.8
34.8
13.6
18.8
26.6
15.1
21.3
30.9
Table 4: Learnable vs. frozen classification heads for visual–language alignment, on Action Genome SGDET in the open-vocabulary setting. Best and second-best are bold and underlined , respectively.
Figure 3: Qualitative results for OvDSGG in the open-vocabulary setting on selected Action Genome frames. Base classes are black; novel classes are red. Ground-truth frames are labeled GT1–GT4; corresponding predictions are labeled 1–4 chronologically. Although not present in the GT scene graph, the model detects novel predicates that are reasonable.
Figure A1 : Object and predicate class distribution on our Base/Novel split of Action Genome dataset.
SGDET , IoU ≥0.5
R@ K↑
mR@ K↑
With Constr.
No Constr.
With Constr.
No Constr.
Spat.
Trp.
Temp.
10
20
50
10
20
50
10
20
50
10
20
50
✓
28.9
35.8
42.9
30.6
39.3
50.0
16.9
22.0
27.0
21.7
31.8
43.8
✓
✓
29.5
36.4
43.3
31.6
40.8
52.0
19.5
24.2
28.3
24.9
33.6
44.9
✓
✓
✓
31.3
38.2
44.7
33.4
42.4
53.2
20.1
25.0
29.3
25.5
35.3
47.1
Table A1 : Module-effectiveness ablation on Action Genome SGDET in the closed-set setting, assessing the contributions of the Spatial (Spat.), Triplet (Trp.) and Temporal (Temp.) modules. Counterpart to Table 3 of the main paper.
Figure A2 : Qualitative results for OvDSGG in the closed-set setting on selected Action Genome frames. Ground-truth frames are labeled GT1–GT4; corresponding predictions are labeled 1–4 chronologically. Although not present in the GT scene graph, the model detects predicates that are plausible. Compare with the open-vocabulary results in the main paper.