FDSG: Forecasting Dynamic Scene Graphs

Yi Yang; Yuren Cong; Hao Cheng; Bodo Rosenhahn; Michael Ying Yang

cs.CVJun 2, 2025

FDSG: Forecasting Dynamic Scene Graphs

Authors: Yi Yang, Yuren Cong, Hao Cheng, Bodo Rosenhahn, Michael Ying Yang

Organizations: Institute of Information Processing, Leibniz University Hannover, Germany. · Faculty of Geo-Information Science and Earth Observation, University of Twente, The Netherlands. · Department of Computer Science and Research Centre for Spatial Intelligence (RCSI), University of Bath, UK.

Abstract

Dynamic scene graph generation extends scene graph generation from images to videos by modeling entity relationships and their temporal evolution. However, existing methods either generate scene graphs from observed frames without explicitly modeling temporal dynamics or predict only relationships while assuming static entity labels and locations. These limitations hinder effective extrapolation of both entity and relationship dynamics, restricting video scene understanding. We propose Forecasting Dynamic Scene Graphs (FDSG), a novel framework that predicts future entity labels, bounding boxes, and relationships, for unobserved frames while also generating scene graphs for observed frames. Our scene graph forecast module leverages query decomposition and neural stochastic differential equations to model entity and relationship dynamics. A temporal aggregation module further refines predictions by integrating observed and forecasted information via cross-attention. To benchmark FDSG, we introduce Scene Graph Forecasting, a new task for full future scene graph prediction. Extensive experiments on Action Genome and VidOR show that FDSG outperforms state-of-the-art methods on dynamic scene graph generation, scene graph anticipation, and scene graph forecasting. Codes will be released upon publication.

Figures & tables

Fig. 1 : Comparison between existing tasks and our FDSG framework. (a) DSGG generates scene graphs and entity bounding boxes from observed video. (b) SGA predicts scene graphs without bounding boxes for future unobserved video. In contrast, FDSG forecasts complete scene graphs, including both (b) relationships, and (c) entity labels with bounding boxes in the future. FDSG also (d) aggregates observed-frame information to enhance both generation and forecasting. Our FDSG employs a one-stage end-to-end design, without explicit object tracking, and incorporates forecasting error as an auxiliary loss, enabling joint learning of DSGG and SGA .

Symbol	Description
Video and Time
$V_{t}$	Video frame at time $t$
$H$	Total number of frames (time horizon)
Scene Graphs
$o_{t}(k)$	Entity $k$ at frame $t$
$c_{t}(k)\in\mathcal{C}$	Entity category label

TABLE 1: Summary of key notation used throughout the method section.

Fig. 2 : Illustration of the proposed SGF task.

Fig. 3 : Our FDSG Framework carries out Dynamic Scene Graph Generation ( DSGG ) and Scene Graph Forecasting ( SGF ) simultaneously to fully exploit temporal dependencies. Video features are extracted using a CNN backbone and a DINO encoder. Then, (a) Scene Graph Generation Module generates scene graph representations for each observed frame. Subsequently, (b) Forecast Module forecasts a scene graph from a reference frame (gray representations) to the target frame (orange representations). Finally, (c) Temporal Aggregation Module aggregates information from DSGG and SGF to jointly enhance model performance. The output enhanced scene graph representations are shown in dark green.

Task	Protocol	Observation / Prediction Window			Input			Evaluated Output
Task	Protocol	Input	Reference	Output	Video	Entity Label	Entity BBox	Entity Label	Entity BBox	Predicate
DSGG	PredCLS	1 $\rightarrow$ H	1 $\rightarrow$ H	1 $\rightarrow$ H	✓	✓	✓	GT	GT	✓
	SGCLS				✓	-	✓	✓	GT	✓
	SGDET				✓	-	-	✓	✓	✓
SGA	GAGS	1 $\rightarrow$ T	1 $\rightarrow$ T	T+1 $\rightarrow$ H	✓	✓	✓	$\times$	$\times$	✓
	PGAGS				✓	-	✓

TABLE 2: Comparison of evaluation settings for tasks DSGG , SGA and SGF . GT means the output uses ground truth from the input and is therefore not evaluated.

SGDET, $IoU\geq 0.5$	Recall@K (R@K) $\uparrow$						Mean Recall@K (mR@K) $\uparrow$
	With Constraint			No Constraint			With Constraint			No Constraint
Method	10	20	50	10	20	50	10	20	50	10	20	50
RelDN [ 47 ]	9.1	9.1	9.1	13.6	23.0	36.6	3.3	3.3	3.3	7.5	18.8	33.7
VCTree [ 48 ]	24.4	32.6	34.7	23.9	35.3	46.8	-	-	-	-	-	-
TRACE [ 8 ]	13.9	14.5	14.5	26.5	35.6	45.3	8.2	8.2	8.2	22.8	31.3	41.8
GPS-Net [ 23 ]	24.7	33.1	35.1	24.4	35.7	47.3	-	-	-	-	-	-

TABLE 3: Comparison with state-of-the-art Dynamic Scene Graph Generation ( DSGG ) methods on Action Genome for Scene Graph Detection (SGDET). Best values are highlighted in boldface .

SGCLS	Recall@K (R@K) $\uparrow$						Mean Recall@K (mR@K) $\uparrow$
	With Constraint			No Constraint			With Constraint			No Constraint
Method	10	20	50	10	20	50	10	20	50	10	20	50
RelDN [ 47 ]	44.3	45.4	45.4	52.9	62.4	65.1	3.3	3.3	3.3	7.5	18.8	33.7
VCTree [ 48 ]	44.1	45.3	45.3	52.4	62.0	65.1	-	-	-	-	-	-
TRACE [ 8 ]	-	45.7	46.8	-	-	-	8.9	8.9	8.9	31.9	42.7	46.3
GPS-Net [ 23 ]	45.3	46.5	46.5	53.6	63.3	66.0	-	-	-	-	-	-

TABLE 4: Comparison with state-of-the-art Dynamic Scene Graph Generation methods on Action Genome for Scene Graph Classification (SGCLS) . Best and second-best values are highlighted in boldface and underlined , respectively.

PredCLS	Recall@K (R@K) $\uparrow$						Mean Recall@K (mR@K) $\uparrow$
	With Constraint			No Constraint			With Constraint			No Constraint
Method	10	20	50	10	20	50	10	20	50	10	20	50
RelDN [ 47 ]	66.3	69.5	69.5	75.7	93.0	99.0	6.2	6.2	6.2	31.2	63.1	75.5
VCTree [ 48 ]	66.0	69.3	69.3	75.5	92.9	99.3	-	-	-	-	-	-
TRACE [ 8 ]	27.5	27.5	27.5	72.6	91.6	96.4	15.2	15.2	15.2	50.9	73.6	82.7
GPS-Net [ 23 ]	66.8	69.9	69.9	76.0	93.6	99.5	-	-	-	-	-	-

TABLE 5: Comparison with state-of-the-art Dynamic Scene Graph Generation methods on Action Genome for Predicate Classification (PredCLS) . Best and second-best values are highlighted in boldface and underlined , respectively.

Protocol		Method	Recall (R@K) $\uparrow$						Mean Recall (mR@K) $\uparrow$
			With Constraint			No Constraint			With Constraint			No Constraint
			10	20	50	10	20	50	10	20	50	10	20	50
AGS	$\mathcal{F}=0.5$	STTran++ [ 6 , 16 ]	19.7	30.2	31.8	16.6	29.1	51.5	6.3	11.3	12.3	6.6	14.7	33.4
		DSG-DETR++ [ 32 , 16 ]	20.7	30.3	31.6	17.4	30.5	51.9	6.4	11.0	11.7	8.4	17.0	33.9
		SceneSayerODE [ 16 ]	25.9	32.6	34.8	26.4	36.6	49.8	11.6	15.2	16.4	14.3	21.4	36.0
		SceneSayerSDE [ 16 ]	27.3	34.8	37.0	28.4	38.6	51.4	12.4	16.6	18.0	16.3	25.1	39.9

TABLE 6: Comparison with state-of-the-art Scene Graph Anticipation methods on Action Genome Scenes (AGS), Partially Grounded Action Genome Scenes (PGAGS), and Grounded Action Genome Scenes (PGAGS). Best and second-best values are highlighted in boldface and underlined , respectively.

$\mathcal{F}$	Method	Recall@K (R@K) $\uparrow$						Mean Recall@K (mR@K) $\uparrow$
		With Constraint			No Constraint			With Constraint			No Constraint
		10	20	50	10	20	50	10	20	50	10	20	50
0.3	SceneSayerODE+ [ 16 ]	3.6	4.3	4.4	3.8	4.6	5.3	2.4	2.9	2.9	2.8	3.2	3.9
	SceneSayerSDE+ [ 16 ]	4.3	4.7	5.1	4.5	5.3	6.2	2.5	3.5	3.5	2.9	4.0	4.7
	FDSG (ours)	9.2	11.4	13.1	10.2	13.1	16.2	7.6	9.7	11.3	9.2	14.2	18.5
0.5	SceneSayerODE+ [ 16 ]	4.8	5.4	5.4	5.1	6.3	6.9	2.3	3.0	3.0	2.6	3.9	4.4

TABLE 7: Comparison with baseline methods on Action Genome for the Scene Graph Forecasting ( SGF ) task. Best values are highlighted in boldface .

Observed Fraction $\mathcal{F}$	Object Detection Recall R@20 $\uparrow$						AP50 $\uparrow$
	SGA (IoU $\geq$ 0)			SGF (IoU $\geq$ 0.5)			AP50 $\uparrow$
	Baseline	Oracle	Ours	Baseline	Oracle	Ours	Baseline	Oracle	Ours
0.3	77.00	77.05	81.57	37.07	39.23	40.67	8.7	13.3	10.2
0.5	80.11	82.75	83.36	40.33	44.01	43.85	10.0	15.9	11.6
0.7	84.44	89.76	87.38	45.67	51.27	47.87	12.3	22.5	13.8
0.9	86.99	96.01	91.89	55.25	64.95	58.51	17.4	36.1	20.3

TABLE 8: Object detection recall and average precision (AP) for SGA and SGF with different observation fraction

\mathcal{F}

Task	Method	Recall (R@K) $\uparrow$						Mean Recall (mR@K) $\uparrow$
		With Constraint			No Constraint			With Constraint			No Constraint
		10	20	50	10	20	50	10	20	50	10	20	50
DSGG	STTran [ 6 ]	19.2	21.6	27.7	22.1	27.4	31.7	5.7	6.5	8.0	6.8	7.0	8.7
SGDET	BIG-C [ 55 ]	21.6	25.9	31.3	23.3	28.9	34.6	6.6	7.2	8.4	7.4	8.8	10.7
	OED [ 11 ]	24.7	28.9	33.4	26.5	30.9	35.0	6.9	8.8	10.7	8.2	10.1	13.6
	FDSG (ours)	26.4	31.5	37.2	27.0	32.5	38.6	7.6	10.1	13.0	8.4	11.6	16.8

TABLE 9: Comparison with state-of-the-art methods on VidOR. Best and second-best values are highlighted in boldface and underlined , respectively.

Task	#	Method				Recall@K (R@K) $\uparrow$						Mean Recall@K (mR@K) $\uparrow$
		Method				With Constraint			No Constraint			With Constraint			No Constraint
		S	T	TDM	LDM	10	20	50	10	20	50	10	20	50	10	20	50
DSGG	1	✓				33.4	41.0	48.1	35.2	45.0	54.0	19.7	25.8	30.9	24.5	38.2	52.0
	2	✓	✓			34.6	42.2	49.2	36.5	46.4	55.5	21.5	27.3	33.0	26.2	39.3	50.8
	3	✓	✓	✓		34.9	42.4	49.3	36.7	46.6	55.7	21.0	27.2	32.0	26.0	39.5	52.5
	4	✓		✓		33.4	41.2	48.3	35.3	45.1	54.2	19.7	25.8	30.9	24.8	38.3	52.2

TABLE 10: Ablation study of model components with recall and mean recall metrics. For Dynamic Scene Graph Generation ( DSGG ) and Scene Graph Forecasting ( SGF ), the task is SGDET, while for Scene Graph Anticipation ( SGA ), the task is AGS. For SGA and SGF , the observed fraction

\mathcal{F}=0.5

. Notations for the model components: S - Static SGG Module; T - Temporal Aggregation Module; TDM - Triplet Dynamics Model; LDM - Location Dynamics Model. Best values are highlighted in boldface .

Task	LDM	Recall (With Constraint) $\uparrow$
Task	LDM	R@10	R@20	R@50
DSGG	IM	35.3	42.8	49.9
DSGG	MLP	35.4	43.0	50.0
SGDET	Decoder	35.3	43.2	50.4
	NeuralODE	34.8	42.4	49.4
	NeuralSDE	34.9	42.6	49.3

TABLE 11: Comparison of different realizations for a Location Dynamics Model (LDM). IM denotes the Identity Mapping.

Task	Method	Recall (With Constraint) $\uparrow$
Task	Method	10	20	50
DSGG	(a)	35.0	42.4	49.3
	(b)	34.7	42.0	49.0
	Ours	35.3	42.9	49.8
SGA	(a)	25.9	32.6	39.8
SGA	(b)	28.1	35.7	43.4

TABLE 12: Comparison between different temporal aggregation strategies. (a) aggregating forecasts before observations; (b) aggregating both information in a single cross-attention layer. Best values are highlighted in boldface .

F	Method	Recall (R@K) $\uparrow$
		With Constraint			No Constraint
		10	20	50	10	20	50
0.3	FAR + OED	8.1	10.5	12.7	10.0	12.8	15.5
0.3	FDSG (ours)	9.2	11.4	13.1	10.2	13.1	16.2
0.5	FAR + OED	10.5	13.2	16.4	11.6	15.7	20.0
0.5	FDSG (ours)	10.6	13.3	15.5	12.0	15.5	18.8

TABLE 13: Performance comparison between FDSG and video forecasting pipeline FAR+OED on SGF with Action Genome. Best values are highlighted in boldface .

Fig. 4 : Visualization of Dynamic Scene Graph Generation ( DSGG ) and Scene Graph Forecasting ( SGF ) outputs from our FDSG. The first two rows show frames with ground truth entity bounding boxes and ground truth scene graphs, while the last two rows show generated scene graphs from DSGG (left) and SGF (right). In the scene graphs, incorrect predicates are highlighted with text in red, while entities/predicates with missing labels in the ground truth are highlighted in orange.

Fig. 5 : Visualization of triplet status transitions on Action Genome (left) and VidOR (right). Forecasted triplets that match to ground truth are the foreground, while the other triplets are the background. On both datasets the triplets are observed to predict new predicates/subjects/objects.

Fig. 6 : Visualization of sampling points of the Deformable Attention in the last Cross-Attention layer of FDSG’s entity decoder for the DSGG task. A brighter color indicates a larger attention weight associated with the sampling point. We also label the detected triplets and the bounding boxes.

Scene Graph Generation

Scene graph generation turns images or videos into structured graphs of objects, their attributes, and their spatial or semantic relations.

See all 48 papers

A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models
Mar 20, 2025Xuanming Cui, Jaiminkumar Ashokbhai Bhoi, Chionh Wei Peng +2Scene Graph Generation Video SGG
University of Central Florida · DSO National Laboratories
SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX
OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation
Aug 14, 2026John Helsby, Yi Yang, Bodo Rosenhahn +1Scene Graph Generation Video SGG
University of Bath, UK · Leibniz Universität Hannover, Germany

Mar 20, 2025J/K move · Enter open · S save

A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models

Xuanming Cui, Jaiminkumar Ashokbhai Bhoi, Chionh Wei Peng, Adriel Kuek+1

University of Central Florida · DSO National Laboratories

Scene Graph Generation Video SGG Fine-Tuning

Open paper AskPDF arXiv

Dynamic Scene Graph Generation (DSGG) aims to capture objects and their evolving relations in videos. Despite recent progress, the practicality and quality of generated scene graphs remain limited compared to the rapid advances in Multimodal Large Language Models (MLLMs). In this work, we revisit DSGG from two fundamental perspectives: task setup and model design. From the task setup perspective, we identify two key limitations of the current recall-oriented evaluation protocol: (i) a severe precision-recall trade-off, and (ii) uninformative and redundant relation generation. To better assess practical usefulness, we introduce five additional metrics that provide a more comprehensive evaluation of the quality of generated dynamic scene graphs. From the model design perspective, we explore directly using MLLMs for scene graph generation and establish a strong MLLM-based DSGG baseline through three design changes. First, we replace the conventional bottom-up pipeline with a top-down reason-then-locate strategy. Second, we reformulate frame-wise dynamic graphs as Temporal Relation Set (TRS) prediction, improving both efficiency and performance. Third, we introduce Importance-Aware Finetuning (IAF) to encourage more relevant and diverse relation generation. Extensive experiments on Action Genome, VidVRD, and PVSG show that our approach consistently achieves state-of-the-art performance.

Abstract

Scene Graph Generation

Explore similar work

A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation