Dynamic Scene Graph Generation (DSGG) aims to capture objects and their evolving relations in videos. Despite recent progress, the practicality and quality of generated scene graphs remain limited compared to the rapid advances in Multimodal Large Language Models (MLLMs). In this work, we revisit DSGG from two fundamental perspectives: task setup and model design. From the task setup perspective, we identify two key limitations of the current recall-oriented evaluation protocol: (i) a severe precision-recall trade-off, and (ii) uninformative and redundant relation generation. To better assess practical usefulness, we introduce five additional metrics that provide a more comprehensive evaluation of the quality of generated dynamic scene graphs. From the model design perspective, we explore directly using MLLMs for scene graph generation and establish a strong MLLM-based DSGG baseline through three design changes. First, we replace the conventional bottom-up pipeline with a top-down reason-then-locate strategy. Second, we reformulate frame-wise dynamic graphs as Temporal Relation Set (TRS) prediction, improving both efficiency and performance. Third, we introduce Importance-Aware Finetuning (IAF) to encourage more relevant and diverse relation generation. Extensive experiments on Action Genome, VidVRD, and PVSG show that our approach consistently achieves state-of-the-art performance.
Figures & tables
Figure 1
Figure 3: Samples of our curated SGQA dataset based on Action Genome.
Figure 3
Figure 5: Conventional DSGG pipelines (top) and ours (bottom). Existing methods typically follow a bottom-up approach: the detector extracts regional features, which are then processed by specialized modules. In contrast, our method adopts a top-down reason-then-locate paradigm, where a MLLM first perform holistic reasoning, then grounded using a open-vocabulary detector.
Figure 6: Illustration of traditional frame-wise dynamic graphs (left) and the proposed Temporal Relation Set (TRS, right).
Table 6
With Constraint
No Constraints
Method
R@10
R@20
R@50
R@10
R@20
R@50
VRD Lu et al. (2016)
19.2
24.5
26.0
19.1
28.8
40.5
GPS-Net Lin et al. (2020)
24.7
33.1
35.1
24.4
35.7
47.3
STTran Cong et al. (2021)
25.2
34.1
37.0
24.6
36.2
48.8
TRACE Teng et al. (2021)
13.9
14.5
14.5
26.5
35.6
45.3
TEMPURA Nag et al. (2023)
28.1
33.4
34.9
29.8
38.1
46.4
Table 4: Results under With/No Constraints settings for SGDet on AG. Deltas are computed based on the SoTA baseline (UNO).
Table 8
vIoU = 0.5
vIoU = 0.1
Method
R/mR@20
R/mR@50
R/mR@20
R/mR@50
VPS Yang et al. (2023)
0.42 / 0.61
0.73 / 0.76
6.50 / 5.75
9.64 / 8.25
IPS+T Yang et al. (2023)
3.88 / 2.81
5.66 / 4.12
9.01 / 6.69
14.9 / 11.3
VISA Li et al. (2025)
- / 8.82
- / 8.94
-
-
MCL Nguyen et al. (2024)
0.84 / 0.98
1.26 / 1.22
8.18 / 8.00
12.9 / 11.5
UNO Le et al. (2026)
9.44 / 8.25
10.8 / 9.72
17.5 / 14.8
23.8 / 18.3
Table 7: Performance comparison of PVSG under different vIoUs.
SGDet
SGCls+
Method
P@1
P@10
P@20
P@1
P@10
P@20
STTran
26.3
17.6
13.1
32.7
22.6
16.9
TRACE
27.5
15.3
12.7
26.5
18.7
13.7
TEMPURA
38.1
21.4
13.7
49.1
27.3
18.0
DSG-DETR
37.4
22.3
14.4
51.3
30.6
20.2
OED
43.6
27.2
17.4
54.1
35.2
22.8
Table 8: Precision on SGDet and SGCls+ under AG. Deltas are computed based on the SoTA baseline (OED).
Figure 7: Per relation category Recall/Precision performance between our MLLM-based method (purple) and TEMPURA (blue) on Action Genome dataset, under the SGCls+ setting. We show the 15 least -frequent relation categories.
Figure 12
Figure 9: SGRet’s recall@5 performance on Action Genome dataset as a function of the number of top- k triplets used during retrieval.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Prompt for generating the proposed SGQA dataset.
Figure 11: Prompt for zero-shot DSGG experiments.
Figure 12: Top-5 triplets’ informativeness score generated by STTran [ 5 ] , TEMPURA [ 41 ] , ours, and ours with Importance-Aware Finetuning (IAF) [ 19 ] on Action Genome and VidVRD dataset.
Model
Obj
Spatial
Temporal
Event
Multi-hop
Overall
STTran
26.5
25.8
25.1
25.5
24.8
25.5
TEMPURA
36.9
33.6
27.9
33.6
25.7
31.5
OED
44.1
40.2
28.5
40.3
27.2
36.1
Ours (InternVL3.5)
52.4 (+8.3)
51.3 (+11.1)
38.4 (+9.9)
52.6 (+12.3)
44.4 (+17.2)
47.8 (+11.7)
Ours (Qwen3-VL)
53.2 (+9.1)
51.8 (+11.6)
39.6 (+11.1)
53.4 (+13.1)
46.6 (+19.4)
48.9 (+12.8)
Appendix
Table 10: Performance across different MCQ question types. Task abbreviation is as follow: Spatial : spatial understanding; Temporal : temporal understanding; Event : event understanding; Multi-hop : multi-hop reasoning.
Figure 13: Qualitative comparison between Tempura, STTran, and Ours. Predictions/objects highlighted in red represent false predictions. Triplets are displayed (top-down) in the order of their model predictions (sorted by probability for STTran and TEMPURA, or by their order of generation.
Figure 14: Qualitative examples of generations from traditional DSGG method (STTran [ 5 ] ), versus ours, and ours with Importance-Aware Finetuning (IAF). Predictions/objects highlighted in red represent false predictions. Triplets are displayed (top-down) in the order of their model predictions (sorted by probability for STTran and TEMPURA, or by their order of generation.