Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos

Ziqi Gao; Jieyu Zhang; Wisdom Oluchi Ikezogwo; Jae Sung Park; Tario G. You; Daniel Ogbu; Chenhao Zheng; Weikai Huang

cs.CVFeb 26, 2026

Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos

Authors: Ziqi Gao, Jieyu Zhang, Wisdom Oluchi Ikezogwo, Jae Sung Park, Tario G. You, Daniel Ogbu, Chenhao Zheng, Weikai Huang,

Organizations: Allen Institute for AI · University of Washington · Microsoft · Woven by Toyota

Abstract

We introduce Synthetic Visual Genome 2 (SVG2), a large-scale panoptic video scene graph dataset. SVG2 contains over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, providing an order-of-magnitude increase in scale and diversity over prior spatio-temporal scene graph datasets. To create SVG2, we design a fully automated pipeline that combines multi-scale panoptic segmentation, online-offline trajectory tracking with automatic new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference. Human verification of SVG2 annotation accuracy confirms its reliability (objects: 93.8%, attributes: 88.3%, relations: 85.4%). Building on this resource, we train TRaSER, a video scene graph generation model. TRaSER augments VLMs with a trajectory-aligned token arrangement mechanism and new modules: an object-trajectory resampler and a temporal-window resampler to convert raw videos and panoptic trajectories into compact spatio-temporal scene graphs in a single forward pass. The temporal-window resampler binds visual tokens to short trajectory segments to preserve local motion and temporal semantics, while the object-trajectory resampler aggregates entire trajectories to maintain global context for objects. On the PVSG, VIPSeg, VidOR and SVG2 test datasets, TRaSER improves relation detection by +15 to 20%, object prediction by +30 to 40% over the strongest open-source baselines and by +13% over GPT-5, and attribute prediction by +15%. When TRaSER's generated scene graphs are sent to a VLM for video question answering, it delivers a +1.5 to 4.6% absolute accuracy gain over using video only or video augmented with Qwen2.5-VL's generated scene graphs, demonstrating the utility of explicit spatio-temporal scene graphs as an intermediate representation.

Figures & tables

Figure 1: Synthetic Visual Genome 2 ( SVG2 ) , a large-scale synthetic panoptic video scene graph dataset. SVG2 provides dense panoptic trajectories, fine-grained object categories and attributes, and temporally grounded spatialtemporal relations across over 636K videos, which is an order-of-magnitude increase in scale and diversity over prior datasets.

Figure 2: Overview of SVG2 synthesis pipeline. Phase 1 : panoptic trajectory generation with online–offline object tracking mechanism that discovers new objects and preserves identity consistency. Phase 2 : per-trajectory description and semantic parsing. Phase 3 : GPT-5–based spatiotemporal relation inference to produce the final video scene graph.

Data	#Vid	Ann.	Type	Frame/ Vid	#Obj/ Traj	Obj Cls	#Rel	Rel Cls	#Attr
SA-V [ 49 ]	50.9K	SAM-2 + Human	Seg S	330	0.6M	-	-	-	-
VIPSeg [ 37 ]	2.8K	Human	Seg S	23	38.2K	124	-	-	-
VidVRD [ 53 ]	0.8K	Human	Box S	304	2.4K	35	25.9K	132	-
AG [ 25 ]	7.8K	Human	Box S	808	26.2K	35	1.4M*	26	-
VidOR [ 52 ]	7.0K	Human	Box D	1K	34.6K	80	0.3M	50	-
PVSG [ 62 ]	338	Human	Seg S	375	6.3K	125	3.6K	62	-

Table 1: Comparison with related benchmarks. Subscripts

{}_{S}/_{D}

denote sparse (sampled) vs. dense (per-frame) annotations. ‘Cls’ indicates the number of categories. (*) AG reports total frame-level relation instances, whereas others report unique trajectory-level instances.

Figure 3: TraSeR architecture. TraSeR first performs trajectory-aligned token arrangement , grounding ViT tokens to instance trajectories to form identity-preserving token streams. It then applies an object-trajectory resampler to aggregate global semantics over each full trajectory, and a temporal-window resampler to preserve fine-grained motion and temporal cues. The resulting tokens are decoded by the language model into a structured video scene graph.

	Tripet			Relation			Object				Attribute
Model	PVSG	VidOR	SVG2 test	PVSG	VidOR	SVG2 test	VIPSeg	PVSG	VidOR	SVG2 test	SVG2 test	AvgRank
API call only
GPT-4.1 [ 1 ]	6.0	10.8	6.4	7.3	11.9	7.5	59.9	51.4	86.6	58.5	15.8	3
Gemini-2.5 PRO [ 11 ]	7.4	9.8	8.7	8.8	11.0	9.9	56.7	31.7	82.9	49.8	13.6	4
GPT-5 [ 43 ]	16.6	19.7	17.9	18.3	21.7	19.4	68.1	54.2	88.5	65.5	24.1	2
Open weights only

Table 2: We evaluate VLMs on four VidSGG benchmarks using panoptic object trajectories. We report performance across four metrics: triplet recall, relation recall, object accuracy, and attribute recall. The results in this table are reported under the lenient semantic criterion, with a temporal IoU threshold of 0.5 for relations and triplets. TraSeR outperforms all open-source baselines and surpasses GPT-5 in object and attribute prediction.

	Object		Relation
	Id+Syn	Lenient	Id+Syn	Lenient
Human vs. GPT-4o-mini	0.901	0.877	0.842	0.791
Gemini 3 Flash vs. GPT-4o-mini	0.846	0.724	0.750	0.765

Table 3: Validation of the LLM semantic aligner. We report Cohen’s

\kappa

agreement scores between GPT-4o-mini, Gemini 3 Flash and human annotators. The high agreement confirms the reliability of the automated aligner. “Id+Syn” denotes Identical or Synonym matches; “Lenient” includes all valid semantic categories.

Training	Triplet			Relation			Object				Attr.	Rank
	PVSG	VidOR	SVG2 t	PVSG	VidOR	SVG2 t	VIPSeg	PVSG	VidOR	SVG2 t	SVG2 t
Acad	1.35	13.61	2.68	12.55	28.66	18.74	32.71	21.50	61.09	25.68	–	4
SA-V	8.29	15.84	13.58	9.86	17.12	16.39	81.61	72.92	90.52	77.64	25.96	3
Acad+SA-V	10.70	19.23	13.00	11.12	21.06	14.69	83.10	73.46	90.96	78.14	22.72	2
Acad+SA-V+PVD	16.91	22.87	16.63	16.13	25.01	18.74	86.53	72.74	91.42	79.01	27.10	1

Table 4: Training-data ablation study. Incorporating SVG2 improves triplet and object prediction, and boosts object accuracy and attribute recall over academic-only training. Training with the full configuration achieves the best overall performance.

Input	AGQA 2.0	Perception-Test
video-only	25.9	66.8
Video+ QWen2.5-VL-3B’s VSG	24.8	68.6
Video+ TraSeR ’s VSG	26.3	71.4

Table 5: GPT-4.1 VQA performance on AGQA 2.0 and Perception-Test. Using TraSeR ’s video scene graphs consistently improves performance over baselines.

Variants			Triplet		Relation		Object
Temporal-window resampler	Object-trajectory resampler	Supervision	PVSG*	VidOR	PVSG*	VidOR	PVSG*	VidOR
✗	✗	Seg.	7.03	13.56	7.15	14.58	73.34	90.79
✗	✓	Seg.	2.44	8.23	2.68	9.04	77.10	89.62
✓	✗	Seg.	8.52	15.00	8.63	16.78	77.84	89.76
✓	✓	Bbox	8.69	15.01	9.39	15.19	73.05	89.76
✓	✓	Seg.	10.14	15.02	11.38	16.30	78.21	90.15

Table 6: Model architecture ablation study. Combining the object-trajectory and temporal-window resamplers yields the best VSG generation performance. Segmentation-aligned token arrangement further provides superior grounding compared to bbox-based signal. (*) Variants without resamplers cannot support long-video inference; therefore, PVSG results are reported on the PVSG VidOR subset only.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
point_grids	$\{32{\times}32,\ 16{\times}16,\ 4{\times}4\}$
points_per_batch	64
pred_iou_thresh	0.75
stability_score_thresh	0.85
stability_score_offset	1.0
box_nms_thresh	0.7

Appendix

Table 7: Configuration of the SAM2AutomaticMaskGenerator employed in Phase 1 to produce dense per-frame segmentation proposals.

Dataset	Duration (s)	FPS
SVG2-SAV	$13.8\pm 3.3$	$24.0\pm 0.0$
SVG2-PVD	$16.6\pm 9.7$	$29.6\pm 8.1$
SVG2	$16.4\pm 9.4$	$29.2\pm 8.0$
$SVG2_{test}$	$13.6\pm 6.3$	$11.8\pm 8.4$

Appendix

Table 8: Video duration and frame rate (FPS) statistics.

Figure 4: Distribution of object categories in SVG2. The dataset covers a diverse range of semantic classes including persons, vehicles, animals, furniture, and various everyday objects.

Figure 5: Distribution of attribute annotations in SVG2. The attributes span visual properties such as color, material, state, and other fine-grained descriptors.

Figure 6: Overview of relationship categories in SVG2, showing the distribution across spatial, motion, functional, stateful, social, attentional, and event-level relation types.

Panel of Figure 7 — Figure 7: Detailed distribution of relationship predicates across all seven categories in SVG2.

Figure 8: Example 1: Visualizations from the SVG2 dataset, illustrating panoptic trajectories, objects, attributes, and diverse inter-object relations.

Figure 9: Example 2: Visualizations from the SVG2 dataset, illustrating panoptic trajectories, objects, attributes, and diverse inter-object relations.

Figure 10: Example 3: Visualizations from the SVG2 dataset, illustrating panoptic trajectories, objects, attributes, and diverse inter-object relations.

Figure 11: Rearranged object-level token sequence example. We introduce two new special tokens, <obj_traj_start> and <obj_traj_end> , to delimit individual trajectories. Within each trajectory, we explicitly mark the object identity using the text “Object i” , and we continue to use the existing special tokens from the Qwen2.5-VL-3B vocabulary to separate visual tokens.

	Triplet (IoU 0.1)			Relation (IoU 0.1)			Object (Strict)
Model	PVSG	VidOR	SVG2 test	PVSG	VidOR	SVG2 test	VIPSeg	PVSG	VidOR	SVG2 test
API call only
GPT-4.1 [ 1 ]	10.3	15.4	6.8	12.3	17.2	8.1	30.0	15.0	26.9	24.9
Gemini-2.5 PRO [ 11 ]	12.3	12.8	9.3	13.9	14.3	10.6	28.0	9.2	20.7	23.3
GPT-5 [ 43 ]	23.8	25.6	18.9	24.2	28.5	20.5	38.2	20.1	22.2	33.4
Open weights only

Appendix

Table 9: Comparison results under different evaluation criteria. We evaluate models using a relaxed temporal grounding threshold for relations and triplets (IoU 0.1). For object evaluation, we report a rule-based ’Strict’ score, which requires an exact string match between the predicted and ground-truth labels.

Metric / Data scale	25%	50%	75%	100%
Triplet
PVSG	10.1	12.2	13.7	16.1
VidOR	20.7	20.5	22.4	22.9
SVG2 test	17.5	18.9	15.7	16.7
Relation
PVSG	10.8	13.1	15.4	16.9

Appendix

Table 10: Data scale ablation showing consistent average performance gains as training data increases.

Variants		Triplet		Relation		Object
Depth	# query of TWR	PVSG*	VidOR	PVSG*	VidOR	PVSG*	VidOR
2	16	12.3	15.1	13.5	16.5	81.3	89.9
3	16	12.8	14.2	11.7	15.4	80.1	90.2
2	32	12.0	15.6	14.7	17.5	80.9	90.0
3	32	13.6	15.0	15.6	16.3	79.7	90.2

Appendix

Table 11: Ablation study on resampler depth and query count. We evaluate the impact of varying the number of layers (Depth) and the number of learnable latent queries (#query) within the Temporal-Window Resampler (TWR). *PVSG results are evaluated on the VidOR subset.

	Triplet			Relation			Object				Attribute
Input Trajectory	PVSG †	VidOR	SVG2 test	PVSG †	VidOR	SVG2 test	VIPSeg	PVSG †	VidOR	SVG2 test	SVG2 test
pipeline-predicted	10.0	4.3	10.6	13.4	6.1	16.2	72.2	63.4	62.9	64.1	17.6
ground-truth	19.2	22.9	16.7	20.1	25.0	18.7	86.5	81.9	91.4	79.0	27.1

Appendix

Table 12: Sensitivity to upstream trajectory quality. We compare TraSeR using ground-truth panoptic segmentation masks (used in all main experiments) versus pipeline-predicted masks matched at IoU

\geq

0.3. Even with noisy and multi-granularity predicted trajectories, TraSeR maintains highly competitive performance. † PVSG is evaluated on its VidOR subset only.

	Triplet			Relation			Object
Model	VidOR (48s)	Epic-Kitchens (120s)	Ego4D (166s)	VidOR (48s)	Epic-Kitchens (120s)	Ego4D (166s)	VidOR (48s)	Epic-Kitchens (120s)	Ego4D (166s)
TraSeR	19.2	6.5	5.1	20.1	7.4	7.7	81.7	63.2	66.0

Appendix

Table 13: Zero-shot temporal generalization across three PVSG subsets spanning durations from 48s (VidOR) to 166s (Ego4D). Object prediction remains highly robust across all lengths, while relation and triplet performance decreases for longer videos, motivating future long-video training strategies.

Panel of Figure 12 — Figure 12: Crowdworkers editing and annotating objects and attributes from segmentation tracking.

Video SGG

See all 11 papers

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX
OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation
Aug 14, 2026John Helsby, Yi Yang, Bodo Rosenhahn +1Scene Graph Generation Video SGG
University of Bath, UK · Leibniz Universität Hannover, Germany
Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding
Jun 10, 2026Hyomin Kim, Junghye Kim, Joanie Hayoun Chung +4Reward Modeling T2V Generation
Department of Artificial Intelligence, Korea University · Department of Statistics, Korea University

May 13, 2026J/K move · Enter open · S save

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Vladislav Makarov, Mark Gizetdinov, Dmitry Yudin

MIRAI · MIREA · MIRAI, AXXX

Scene Graph Generation Vision-Language Models Video SGG Scene Graphs

Open paper AskPDF arXiv

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

Abstract

Video SGG

Explore similar work

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation

Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding