Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos

Rohith Peddi; Saurabh; Shravan Shanmugam; Likhitha Pallapothula; Yu Xiang; Parag Singla; Vibhav Gogate

cs.CVMar 13, 2026

Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos

Authors: Rohith Peddi, Saurabh, Shravan Shanmugam, Likhitha Pallapothula, Yu Xiang, Parag Singla, Vibhav Gogate

Organizations: The University of Texas at Dallas, Richardson, TX 75080, USA · Indian Institute of Technology Delhi, New Delhi, India

Abstract

Spatio-temporal scene graphs provide a principled representation for modeling evolving object interactions, yet existing methods remain fundamentally frame-centric: they reason only about currently visible objects, discard entities upon occlusion, and operate in 2D. To address this, we first introduce ActionGenome4D, a dataset that upgrades Action Genome videos into 4D scenes via feed-forward 3D reconstruction, world-frame oriented bounding boxes for every object involved in actions, and dense relationship annotations including for objects that are temporarily unobserved due to occlusion or camera motion. Building on this data, we formalize World Scene Graph Generation (WSGG), the task of constructing a world scene graph at each timestamp that encompasses all interacting objects in the scene, both observed and unobserved. We then propose three complementary methods, each exploring a different inductive bias for reasoning about unobserved objects: PWG (Persistent World Graph), which implements object permanence via a zero-order feature buffer; MWAE (Masked World Auto-Encoder), which reframes unobserved-object reasoning as masked completion with cross-view associative retrieval; and 4DST (4D Scene Transformer), which replaces the static buffer with differentiable per-object temporal attention enriched by 3D motion and camera-pose features. We further design and evaluate the performance of strong open-source Vision-Language Models on the WSGG task via a suite of Graph RAG-based approaches, establishing baselines for unlocalized relationship prediction. WSGG thus advances video scene understanding toward world-centric, temporally persistent, and interpretable scene reasoning.

Figures & tables

Figure 1: World Scene Graph Generation (WSGG). Unlike standard Video Scene Graph Generation (left) that is constrained to the instantaneous camera view and discards objects once they exit the frame or get occluded, our proposed task (right) grounds scene understanding in a global 3D world coordinate frame. Models trained on WSGG output a comprehensive world scene graph at each timestamp containing all the objects in the environment. As shown at $t=30$ s, out-of-view objects (e.g., bed, laptop) remain perfectly localized in 3D, enabling global, view-independent interpretable scene reasoning. Blue curves drawn from person to objects (right) represent relationships.

Task	Input	Spatial	Temporal	Localization	Object Scope	Coord.	Unobs.	Rel.
		Dim.				Frame	Objects	Persistence
Image SGG	Image	2D	✗	2D BBox	Detected	Image	✗	Per-frame
Video SGG	Video	2D	✓	2D BBox	Detected / frame	Image	✗	Cross-frame
3D SGG	3D Scan	3D	✗	3D BBox	All in scan	Scene	✗	Per-scan
4D SGG	Video + 3D	3D	✓	3D BBox	Detected / frame	Scene	✗	Cross-frame
Panoptic SGG	Image	2D	✗	2D Mask	Things + Stuff	Image	✗	Per-frame

Table 1: SGG Tasks. Bold entries highlight the unique capabilities of WSGG.

Figure 2: WorldSGG Dataset & 3D OBB construction pipeline. Our offline framework generates persistent 3D annotations through three sequential steps: (a) Scene Construction recovers the global scene point cloud and camera poses using $\pi^{3}$ and bundle adjustment; (b) Floor Determination establishes a canonical ground plane by aligning 3D human meshes extracted via PromptHMR; and (c) 3D OBB Construction refines raw object geometries to fit robust 3D bounding boxes in the world frame.

Figure 3: WorldSGG Methods. Multi-modal inputs including DINO visual features, monocular 3D geometries, and camera extrinsics are processed into unified structural, motion, and ego-pose tokens. (Right & Bottom) Unobserved Object Processing: To maintain strict object permanence for entities outside the current camera frustum, we propose and evaluate three structural variants: (1) PWG (Persistent World Graph) utilizes a Last-Known-State (LKS) buffer to explicitly track and propagate historical features based on temporal staleness ( $\Delta_{n}$ ). (2) MWAE (Masked World Auto-Encoder) frames natural occlusion as a masked modeling task; it masks unobserved visual streams and uses an Associative Retriever with asymmetric cross-attention (querying all tokens against only visible ones) to reconstruct missing features of unobserved objects. (3) 4DST (4D Scene Transformer) fuses multi-modal tokens at a Fusion Node and applies unmasked bidirectional temporal self-attention followed by Spatial GNNs to output dense, globally-aware spatiotemporal representations ( $\mathbf{H}^{(t)}$ ).

		PredCls						SGDet
		With Constraint			No Constraint			With Constraint			No Constraint
Method	Backbone	R@10	R@20	R@50	R@10	R@20	R@50	R@10	R@20	R@50	R@10	R@20	R@50
PWG	DINOv2-L	65.07	67.99	68.00	80.06	94.39	99.59	41.69	57.14	69.63	28.62	40.69	56.66
	DINOv3-L	65.58	68.57	68.58	81.01	94.73	99.76	39.96	57.11	70.93	26.84	37.93	52.57
MWAE	DINOv2-L	65.33	68.30	68.31	80.96	94.51	99.65	41.69	57.06	69.50	28.91	40.85	56.96
	DINOv3-L	65.57	68.58	68.59	81.50	95.11	99.78	39.67	57.06	70.90	26.89	38.17	52.86

Table 2: Recall (R@K) comparison across methods and feature backbones for PredCls and SGDet tasks on ActionGenome4D. Best results per column are in bold .

		PredCls						SGDet
		With Constraint			No Constraint			With Constraint			No Constraint
Method	Backbone	mR@10	mR@20	mR@50	mR@10	mR@20	mR@50	mR@10	mR@20	mR@50	mR@10	mR@20	mR@50
PWG	DINOv2-L	34.87	38.79	38.80	52.83	72.56	93.54	11.09	22.46	36.86	24.58	36.41	54.98
	DINOv3-L	36.90	41.28	41.29	54.27	74.73	94.27	9.56	22.61	39.61	23.25	35.60	55.51
MWAE	DINOv2-L	35.03	39.46	39.47	53.31	73.90	95.04	10.95	22.19	36.33	22.98	37.60	55.47
	DINOv3-L	36.41	40.98	40.99	55.09	75.15	95.02	9.75	22.97	39.74	23.16	35.74	54.08

Table 3: Mean Recall (mR@K) comparison across methods and feature backbones for PredCls and SGDet tasks on ActionGenome4D. Best results per column in bold .

Figure 4: The Graph RAG inference pipeline. (Left) Sequential video segments are processed by a VLM into local captions to construct a global Coarse Event Graph, establishing a high-level temporal narrative. (Right) To generate the fine-grained world scene graph, our Graph RAG module retrieves overarching spatiotemporal context from the coarse graph. This global prior is injected into both the visual stream (Vision Encoder) and the textual stream, where it enriches targeted Object and Relationship queries. An LLM then deduces the final, offline World Scene Graph.

Pipeline	Model	Mode	Attention			Contacting			Spatial			Micro	Macro
Pipeline	Model	Mode	P	R	F1	P	R	F1	P	R	F1	F1	F1
Graph RAG	InternVL 2.5	PredCls	53.8	53.7	53.8	45.4	40.3	42.7	30.8	24.4	27.2	40.8	23.4
	InternVL 2.5	SGDet	49.6	22.3	30.8	43.5	18.1	25.5	35.3	12.9	18.9	24.8	15.4
	Qwen 2.5-VL	PredCls	61.4	61.3	61.4	60.7	53.6	56.9	47.6	38.5	42.5	53.3	26.6
	Qwen 2.5-VL	SGDet	59.7	34.8	44.0	62.0	32.3	42.5	48.7	23.1	31.3	39.0	20.2
Subtitle-Only	InternVL 2.5	PredCls	50.6	50.6	50.6	44.3	38.5	41.2	23.4	17.7	20.2	37.0	21.7

Table 4: VLM relationship prediction on corrected ActionGenome4D annotations (micro-averaged). P = Precision, R = Recall, F1 = F1-score. Micro F1 : micro-average across all three predicate axes. Macro F1 : mean of per-class F1 scores.

Pipeline	Model	Mode	Attention			Contacting			Spatial			Overall
Pipeline	Model	Mode	P	R	F1	P	R	F1	P	R	F1	M-F1
Graph RAG	InternVL 2.5	PredCls	38.6	41.5	36.4	30.1	31.1	25.7	37.3	16.5	12.1	23.4
	InternVL 2.5	SGDet	39.7	19.3	21.9	31.4	15.4	17.5	34.4	6.7	7.5	15.4
	Qwen 2.5-VL	PredCls	38.3	39.6	38.4	31.0	27.1	26.0	40.2	23.6	23.1	26.6
	Qwen 2.5-VL	SGDet	39.3	23.3	28.3	29.9	17.4	19.9	39.0	14.0	17.5	20.2
Subtitle-Only	InternVL 2.5	PredCls	37.6	40.1	34.5	31.7	35.8	24.6	34.8	14.6	8.8	21.7

Table 5: VLM relationship prediction on corrected ActionGenome4D annotations (macro-averaged). Same setting as Table 4 ; P, R, and F1 are computed per class then averaged equally. M-F1 : overall macro-averaged F1 across all three axes.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: World Scene Graph Generation (WSGG). Left: Standard Video Scene Graph Generation (VidSGG) is anchored to the instantaneous camera view. Objects are localized as 2D bounding boxes in the image plane, and relationships are predicted only for currently detected entities. When an object exits the field of view or becomes occluded, it is silently dropped from the graph along with all of its relationships, resulting in an incomplete and temporally fragmented scene understanding. Right: Our proposed WSGG task grounds scene understanding in a persistent, global 3D world coordinate frame. Every object in the environment is represented by a 3D oriented bounding box (OBB) that persists across frames, regardless of whether the object is currently visible in the camera view. At each timestamp, the model outputs a complete world scene graph containing all known objects; observed ( $\mathcal{O}^{t}$ ) and unobserved ( $\mathcal{U}^{t}$ ); together with their pairwise semantic relationships (attention, spatial, and contacting predicates). As illustrated at $t{=}30$ s, objects that have left the camera’s field of view (e.g., bed, laptop) remain precisely localized in 3D world coordinates, enabling the model to continue predicting meaningful relationships for them. Blue curves drawn from person to objects denote predicted relationships. This view-independent, temporally persistent representation supports downstream tasks such as embodied navigation, robotic manipulation, and long-horizon activity understanding that require reasoning about the full state of the world, not just its currently visible slice.

Task	Input	Spatial	Temporal	Localization	Object Scope	Coordinate	Unobserved	Relationship
		Dimension				Frame	Objects	Persistence
Image SGG	Image	2D	✗	2D BBox	Detected	Image	✗	Per-frame
Video SGG	Video	2D	✓	2D BBox	Detected / frame	Image	✗	Cross-frame
3D SGG	3D Scan	3D	✗	3D BBox	All in scan	Scene	✗	Per-scan
4D SGG	Video + 3D	3D	✓	3D BBox	Detected / frame	Scene	✗	Cross-frame
Panoptic SGG	Image	2D	✗	2D Mask	Things + Stuff	Image	✗	Per-frame

Appendix

Table 6: SGG Tasks. Bold entries highlight the unique capabilities of WSGG.

Symbol / Notation	Description
Task Formalization
$V_{1}^{T}=\{I^{1},\dots,I^{T}\}$	Video consisting of $T$ frames
$I^{t}$	Video frame at timestamp $t$
$\mathcal{W}^{t}$	World state at timestamp $t$ (all objects in the scene)
$\mathcal{O}^{t}$	Set of observed objects visible in frame $I^{t}$
$\mathcal{U}^{t}$	Set of unobserved objects not visible in frame $I^{t}$

Appendix

Table 7: Notation reference.

Figure 6: Pipeline for constructing a 4D scene representation from Action Genome videos. The method first selects informative frames through adaptive sampling, then predicts static and dynamic world point clouds, decomposes the scene into background and per-frame dynamic components, and finally aligns dynamic point clouds to the static scene using trimmed ICP before mask-aware merging into the final 4D output.

Quantity	Shape	Type	Description
Local points	$(N,H,W,3)$	float32	Per-pixel 3D coordinates in camera frame
World points	$(N,H,W,3)$	float32	Per-pixel 3D coordinates in world frame
Confidence	$(N,H,W,1)$	float32	Per-pixel reliability score $\in[0,1]$
Camera poses	$(N,4,4)$	float32	Camera-to-world SE(3) matrices

Appendix

Table 8: Quantities estimated by the

\pi^{3}

model for

N

input views at resolution

H\times W

Parameter	Value	Description
Backbone	DINOv2-L (ViT-L/14)	Frozen encoder with registers
Decoder depth	36 layers	Alternating intra-/cross-view attn
Positional encoding	RoPE2D ( $f{=}100$ )	2D rotary position embeddings
Precision	bfloat16 / float16	Mixed-precision inference
Frame sampling	10	Temporal stride for selection
$\tau_{\text{static}}$	0.1	Confidence thr. (static scene)

Appendix

Table 9: Scene construction pipeline parameters.

Panel of Figure 7 — Figure 7: 3D bounding box pipeline overview. Objects are detected via Grounding DINO fused with GT annotations, segmented via SAM2 in dual mode, projected into 3D via masked point extraction from the $\pi^{3}$ reconstruction, aligned to the floor plane, and enclosed with oriented bounding boxes (OBBs).

Parameter	Value	Description
Detection
Detector model	Grounding DINO (base)	Zero-shot open-vocabulary
Box threshold	0.25	Detection confidence
Text threshold	0.30	Text-box alignment
NMS IoU threshold	0.50	Per-class NMS
GT pseudo-score	1.001	Ensures GT survives NMS

Appendix

Table 10: 3D bounding box pipeline parameters.

Figure 8: Overview of the annotation interface used for manual floor correction. The figure illustrates the sequential adjustment process across four steps, culminating in the final corrected transform for accurate 3D scene alignment.

Figure 9: Interface for manual relationship correction in WorldSGG. The tool presents object-centric frame sequences and allows annotators to refine attention, contact, and spatial predicates across time for accurate scene graph construction.

Scope
Property	Action Genome	ActionGenome4D
Object scope per frame	$\mathcal{O}^{t}$ only	$\mathcal{W}^{t}=\mathcal{O}^{t}\cup\mathcal{U}^{t}$
Relationship coverage	Observed pairs	All world-state pairs
Coordinate frame	2D image	3D world
Object localization	2D BBox	2D BBox + 3D OBB
Camera poses	✗	✓ (SE(3) per frame)

Appendix

Table 11: AG vs. ActionGenome4D. AG4D extends AG along geometric and semantic dimensions.

\mathcal{O}^{t}

: observed objects;

\mathcal{U}^{t}

: unobserved objects;

\mathcal{W}^{t}

: full world state.

Statistic	Train	Test
Videos	7,516	1,734
Annotated frames	175,751	56,352
Object instances (observed)	549,315	201,683
Object instances (unobserved)	61,149	22,624
Object instances (total)	610,464	224,307
Relationship triplets (observed)	373,564	145,331

Appendix

Table 12: ActionGenome4D statistics. The train set uses VLM pseudo-annotations for unobserved object relationships; the test set uses manually corrected annotations.

Key	Model	Hidden dim	Params	Patch size
v2	DINOv2 ViT-B	768	86M	14
v2s	DINOv2 ViT-S	384	22M	14
v2l	DINOv2 ViT-L	1024	304M	14
v3l	DINOv3 ViT-L	1024	304M	16

Appendix

Table 13: Supported backbone variants.

Parameter	Default	Description
model	v2	Backbone variant (Table 13 )
lr	$10^{-4}$	Peak learning rate
weight_decay	$10^{-3}$	AdamW weight decay
batch_size	128	Per-GPU batch size
epochs	70	Total training epochs
max_grad_norm	1.0	Gradient clipping threshold

Appendix

Table 14: Key training configuration parameters.

Backbone	Training	mAP	mAP 50	mAP 75
DINOv2-B	Separate	0.0988	0.2333	0.0692
DINOv2-L	Separate	0.1103	0.2627	0.0739
DINOv3-L	Separate	0.1762	0.3667	0.1461
DINOv2-B	Joint	0.0998	0.2377	0.0672
DINOv2-L	Joint	0.1086	0.2610	0.0712
DINOv3-L	Joint	0.1799	0.3660	0.1552

Appendix

Table 15: Comparison of DINO backbones under joint and separate training setups.

Backbone	Training	Mean IoU 3D	[email protected]
DINOv2-B	Separate	0.1446	0.0137
DINOv2-L	Separate	0.1377	0.0117
DINOv3-L	Separate	0.1295	0.0111
DINOv2-B	Joint	0.1411	0.0120
DINOv2-L	Joint	0.1390	0.0104
DINOv3-L	Joint	0.1285	0.0128

Appendix

Table 16: 3D localization performance of different DINO backbones under joint and separate training setups. Mean IoU 3D measures the average 3D overlap, while [email protected] denotes the fraction of predictions exceeding the IoU 3D threshold.

Figure 10: Pipeline of the proposed WorldSGG framework. The model extracts DINO-based object features, 3D OBB corners, and camera pose cues, encodes them into structural, motion, and camera tokens, and fuses them with buffered memory features to handle occlusions and missing observations. The resulting unified tokens are processed by a spatial GNN and relationship decoders to predict node categories and scene graph relations over time.

Property	Subtitle-Only (Method 1)	RAG-Based (Method 2)
Query scope	All objects $\times$ all frames	All objects $\times$ all frames
Uses precomputed graph?	No (subtitles only)	Yes
Deduplication	N/A (no retrieval steps)	Yes (unique prompts for steps 1–3)
VLM calls	1 batched + 1 verify	4 batched + 1 verify
Per-frame clips	Annotation-driven (fallback: full video)	Annotation-driven (fallback: RAG clip)
Grounding signal	Subtitles + video	Graph context + subtitles + video

Appendix

Table 17: Comparison of the two pseudo-annotation generation methods for World Scene Graph Generation.

Scene Graphs

See all 17 papers

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX
OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation
Aug 14, 2026John Helsby, Yi Yang, Bodo Rosenhahn +1Scene Graph Generation Video SGG
University of Bath, UK · Leibniz Universität Hannover, Germany
Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos
Feb 26, 2026Ziqi Gao, Jieyu Zhang, Wisdom Oluchi Ikezogwo +10Video SGG Scene Graph Generation
Allen Institute for AI · University of Washington · Microsoft +1

May 13, 2026J/K move · Enter open · S save

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Vladislav Makarov, Mark Gizetdinov, Dmitry Yudin

MIRAI · MIREA · MIRAI, AXXX

Scene Graph Generation Vision-Language Models Video SGG Scene Graphs

Open paper AskPDF arXiv

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

Abstract

Scene Graphs

Explore similar work

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation

Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos