We introduce Synthetic Visual Genome 2 (SVG2), a large-scale panoptic video scene graph dataset. SVG2 contains over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, providing an order-of-magnitude increase in scale and diversity over prior spatio-temporal scene graph datasets. To create SVG2, we design a fully automated pipeline that combines multi-scale panoptic segmentation, online-offline trajectory tracking with automatic new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference. Human verification of SVG2 annotation accuracy confirms its reliability (objects: 93.8%, attributes: 88.3%, relations: 85.4%). Building on this resource, we train TRaSER, a video scene graph generation model. TRaSER augments VLMs with a trajectory-aligned token arrangement mechanism and new modules: an object-trajectory resampler and a temporal-window resampler to convert raw videos and panoptic trajectories into compact spatio-temporal scene graphs in a single forward pass. The temporal-window resampler binds visual tokens to short trajectory segments to preserve local motion and temporal semantics, while the object-trajectory resampler aggregates entire trajectories to maintain global context for objects. On the PVSG, VIPSeg, VidOR and SVG2 test datasets, TRaSER improves relation detection by +15 to 20%, object prediction by +30 to 40% over the strongest open-source baselines and by +13% over GPT-5, and attribute prediction by +15%. When TRaSER's generated scene graphs are sent to a VLM for video question answering, it delivers a +1.5 to 4.6% absolute accuracy gain over using video only or video augmented with Qwen2.5-VL's generated scene graphs, demonstrating the utility of explicit spatio-temporal scene graphs as an intermediate representation.
Figures & tables
Figure 1: Synthetic Visual Genome 2 ( SVG2 ) , a large-scale synthetic panoptic video scene graph dataset. SVG2 provides dense panoptic trajectories, fine-grained object categories and attributes, and temporally grounded spatialtemporal relations across over 636K videos, which is an order-of-magnitude increase in scale and diversity over prior datasets.
Figure 2: Overview of SVG2 synthesis pipeline. Phase 1 : panoptic trajectory generation with online–offline object tracking mechanism that discovers new objects and preserves identity consistency. Phase 2 : per-trajectory description and semantic parsing. Phase 3 : GPT-5–based spatiotemporal relation inference to produce the final video scene graph.
Data
#Vid
Ann.
Type
Frame/ Vid
#Obj/ Traj
Obj Cls
#Rel
Rel Cls
#Attr
SA-V [ 49 ]
50.9K
SAM-2 + Human
Seg S
330
0.6M
-
-
-
-
VIPSeg [ 37 ]
2.8K
Human
Seg S
23
38.2K
124
-
-
-
VidVRD [ 53 ]
0.8K
Human
Box S
304
2.4K
35
25.9K
132
-
AG [ 25 ]
7.8K
Human
Box S
808
26.2K
35
1.4M*
26
-
VidOR [ 52 ]
7.0K
Human
Box D
1K
34.6K
80
0.3M
50
-
PVSG [ 62 ]
338
Human
Seg S
375
6.3K
125
3.6K
62
-
Table 1: Comparison with related benchmarks. Subscripts S/D denote sparse (sampled) vs. dense (per-frame) annotations. ‘Cls’ indicates the number of categories. (*) AG reports total frame-level relation instances, whereas others report unique trajectory-level instances.
Figure 3: TraSeR architecture. TraSeR first performs trajectory-aligned token arrangement , grounding ViT tokens to instance trajectories to form identity-preserving token streams. It then applies an object-trajectory resampler to aggregate global semantics over each full trajectory, and a temporal-window resampler to preserve fine-grained motion and temporal cues. The resulting tokens are decoded by the language model into a structured video scene graph.
Tripet
Relation
Object
Attribute
Model
PVSG
VidOR
SVG2 test
PVSG
VidOR
SVG2 test
VIPSeg
PVSG
VidOR
SVG2 test
SVG2 test
AvgRank
API call only
GPT-4.1 [ 1 ]
6.0
10.8
6.4
7.3
11.9
7.5
59.9
51.4
86.6
58.5
15.8
3
Gemini-2.5 PRO [ 11 ]
7.4
9.8
8.7
8.8
11.0
9.9
56.7
31.7
82.9
49.8
13.6
4
GPT-5 [ 43 ]
16.6
19.7
17.9
18.3
21.7
19.4
68.1
54.2
88.5
65.5
24.1
2
Open weights only
Table 2: We evaluate VLMs on four VidSGG benchmarks using panoptic object trajectories. We report performance across four metrics: triplet recall, relation recall, object accuracy, and attribute recall. The results in this table are reported under the lenient semantic criterion, with a temporal IoU threshold of 0.5 for relations and triplets. TraSeR outperforms all open-source baselines and surpasses GPT-5 in object and attribute prediction.
Object
Relation
Id+Syn
Lenient
Id+Syn
Lenient
Human vs. GPT-4o-mini
0.901
0.877
0.842
0.791
Gemini 3 Flash vs. GPT-4o-mini
0.846
0.724
0.750
0.765
Table 3: Validation of the LLM semantic aligner. We report Cohen’s κ agreement scores between GPT-4o-mini, Gemini 3 Flash and human annotators. The high agreement confirms the reliability of the automated aligner. “Id+Syn” denotes Identical or Synonym matches; “Lenient” includes all valid semantic categories.
Training
Triplet
Relation
Object
Attr.
Rank
PVSG
VidOR
SVG2 t
PVSG
VidOR
SVG2 t
VIPSeg
PVSG
VidOR
SVG2 t
SVG2 t
Acad
1.35
13.61
2.68
12.55
28.66
18.74
32.71
21.50
61.09
25.68
–
4
SA-V
8.29
15.84
13.58
9.86
17.12
16.39
81.61
72.92
90.52
77.64
25.96
3
Acad+SA-V
10.70
19.23
13.00
11.12
21.06
14.69
83.10
73.46
90.96
78.14
22.72
2
Acad+SA-V+PVD
16.91
22.87
16.63
16.13
25.01
18.74
86.53
72.74
91.42
79.01
27.10
1
Table 4: Training-data ablation study. Incorporating SVG2 improves triplet and object prediction, and boosts object accuracy and attribute recall over academic-only training. Training with the full configuration achieves the best overall performance.
Input
AGQA 2.0
Perception-Test
video-only
25.9
66.8
Video+ QWen2.5-VL-3B’s VSG
24.8
68.6
Video+ TraSeR ’s VSG
26.3
71.4
Table 5: GPT-4.1 VQA performance on AGQA 2.0 and Perception-Test. Using TraSeR ’s video scene graphs consistently improves performance over baselines.
Variants
Triplet
Relation
Object
Temporal-window resampler
Object-trajectory resampler
Supervision
PVSG*
VidOR
PVSG*
VidOR
PVSG*
VidOR
✗
✗
Seg.
7.03
13.56
7.15
14.58
73.34
90.79
✗
✓
Seg.
2.44
8.23
2.68
9.04
77.10
89.62
✓
✗
Seg.
8.52
15.00
8.63
16.78
77.84
89.76
✓
✓
Bbox
8.69
15.01
9.39
15.19
73.05
89.76
✓
✓
Seg.
10.14
15.02
11.38
16.30
78.21
90.15
Table 6: Model architecture ablation study. Combining the object-trajectory and temporal-window resamplers yields the best VSG generation performance. Segmentation-aligned token arrangement further provides superior grounding compared to bbox-based signal. (*) Variants without resamplers cannot support long-video inference; therefore, PVSG results are reported on the PVSG VidOR subset only.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
point_grids
{32×32,16×16,4×4}
points_per_batch
64
pred_iou_thresh
0.75
stability_score_thresh
0.85
stability_score_offset
1.0
box_nms_thresh
0.7
Appendix
Table 7: Configuration of the SAM2AutomaticMaskGenerator employed in Phase 1 to produce dense per-frame segmentation proposals.
Dataset
Duration (s)
FPS
SVG2-SAV
13.8±3.3
24.0±0.0
SVG2-PVD
16.6±9.7
29.6±8.1
SVG2
16.4±9.4
29.2±8.0
SVG2test
13.6±6.3
11.8±8.4
Appendix
Table 8: Video duration and frame rate (FPS) statistics.
Figure 4: Distribution of object categories in SVG2. The dataset covers a diverse range of semantic classes including persons, vehicles, animals, furniture, and various everyday objects.
Figure 5: Distribution of attribute annotations in SVG2. The attributes span visual properties such as color, material, state, and other fine-grained descriptors.
Figure 6: Overview of relationship categories in SVG2, showing the distribution across spatial, motion, functional, stateful, social, attentional, and event-level relation types.
Figure 7: Detailed distribution of relationship predicates across all seven categories in SVG2.
Figure 8: Example 1: Visualizations from the SVG2 dataset, illustrating panoptic trajectories, objects, attributes, and diverse inter-object relations.
Figure 9: Example 2: Visualizations from the SVG2 dataset, illustrating panoptic trajectories, objects, attributes, and diverse inter-object relations.
Figure 10: Example 3: Visualizations from the SVG2 dataset, illustrating panoptic trajectories, objects, attributes, and diverse inter-object relations.
Figure 11: Rearranged object-level token sequence example. We introduce two new special tokens, <obj_traj_start> and <obj_traj_end> , to delimit individual trajectories. Within each trajectory, we explicitly mark the object identity using the text “Object i” , and we continue to use the existing special tokens from the Qwen2.5-VL-3B vocabulary to separate visual tokens.
Triplet (IoU 0.1)
Relation (IoU 0.1)
Object (Strict)
Model
PVSG
VidOR
SVG2 test
PVSG
VidOR
SVG2 test
VIPSeg
PVSG
VidOR
SVG2 test
API call only
GPT-4.1 [ 1 ]
10.3
15.4
6.8
12.3
17.2
8.1
30.0
15.0
26.9
24.9
Gemini-2.5 PRO [ 11 ]
12.3
12.8
9.3
13.9
14.3
10.6
28.0
9.2
20.7
23.3
GPT-5 [ 43 ]
23.8
25.6
18.9
24.2
28.5
20.5
38.2
20.1
22.2
33.4
Open weights only
Appendix
Table 9: Comparison results under different evaluation criteria. We evaluate models using a relaxed temporal grounding threshold for relations and triplets (IoU 0.1). For object evaluation, we report a rule-based ’Strict’ score, which requires an exact string match between the predicted and ground-truth labels.
Metric / Data scale
25%
50%
75%
100%
Triplet
PVSG
10.1
12.2
13.7
16.1
VidOR
20.7
20.5
22.4
22.9
SVG2 test
17.5
18.9
15.7
16.7
Relation
PVSG
10.8
13.1
15.4
16.9
Appendix
Table 10: Data scale ablation showing consistent average performance gains as training data increases.
Variants
Triplet
Relation
Object
Depth
# query of TWR
PVSG*
VidOR
PVSG*
VidOR
PVSG*
VidOR
2
16
12.3
15.1
13.5
16.5
81.3
89.9
3
16
12.8
14.2
11.7
15.4
80.1
90.2
2
32
12.0
15.6
14.7
17.5
80.9
90.0
3
32
13.6
15.0
15.6
16.3
79.7
90.2
Appendix
Table 11: Ablation study on resampler depth and query count. We evaluate the impact of varying the number of layers (Depth) and the number of learnable latent queries (#query) within the Temporal-Window Resampler (TWR). *PVSG results are evaluated on the VidOR subset.
Triplet
Relation
Object
Attribute
Input Trajectory
PVSG †
VidOR
SVG2 test
PVSG †
VidOR
SVG2 test
VIPSeg
PVSG †
VidOR
SVG2 test
SVG2 test
pipeline-predicted
10.0
4.3
10.6
13.4
6.1
16.2
72.2
63.4
62.9
64.1
17.6
ground-truth
19.2
22.9
16.7
20.1
25.0
18.7
86.5
81.9
91.4
79.0
27.1
Appendix
Table 12: Sensitivity to upstream trajectory quality. We compare TraSeR using ground-truth panoptic segmentation masks (used in all main experiments) versus pipeline-predicted masks matched at IoU ≥ 0.3. Even with noisy and multi-granularity predicted trajectories, TraSeR maintains highly competitive performance. † PVSG is evaluated on its VidOR subset only.
Triplet
Relation
Object
Model
VidOR (48s)
Epic-Kitchens (120s)
Ego4D (166s)
VidOR (48s)
Epic-Kitchens (120s)
Ego4D (166s)
VidOR (48s)
Epic-Kitchens (120s)
Ego4D (166s)
TraSeR
19.2
6.5
5.1
20.1
7.4
7.7
81.7
63.2
66.0
Appendix
Table 13: Zero-shot temporal generalization across three PVSG subsets spanning durations from 48s (VidOR) to 166s (Ego4D). Object prediction remains highly robust across all lengths, while relation and triplet performance decreases for longer videos, motivating future long-video training strategies.
Figure 12: Crowdworkers editing and annotating objects and attributes from segmentation tracking.
Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.
Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as ⟨subject, predicate, object⟩ triplets, and underpin downstream tasks such as video captioning, video question answering, and action analysis. However, end-to-end dynamic scene graph generation (DSGG) methods are closed-set: they recognize only objects and predicates from a fixed training vocabulary and struggle with the long-tailed distribution of rare concepts, severely limiting their real-world applicability. Existing open-vocabulary models typically inherit pretrained large language models, resulting in multi-stage training and inference with substantial cost. We introduce OvDSGG, the first end-to-end framework for open-vocabulary DSGG. OvDSGG builds on top of an open-vocabulary Spatial Backbone and a Temporal Backbone; we further propose a Triplet Feature Extraction Module that bridges them, and a Visual-Language Alignment Module that preserves open-vocabulary recognition by learning an adaptive decision boundary in the joint visual-language feature space, without expensive knowledge distillation in existing methods. We further introduce a rigorous open-vocabulary DSGG benchmark adapted from Action Genome, with disjoint Base/Novel splits for both objects and predicates. OvDSGG significantly outperforms open-vocabulary baselines across all metrics, with zero-shot Recall@K scores 10.0--20.4 percentage point higher than the next-best baseline, while on closed-set DSGG remaining competitive with state-of-the-art models. Code and benchmark are publicly available at https://github.com/jhelsby/OvDSGG/.
John Helsby, Yi Yang, Bodo Rosenhahn +1
University of Bath, UK · Leibniz Universität Hannover, Germany
Reward models for text-to-video (T2V) generation guide post-training but often fail at fine-grained semantic alignment. We trace this to two structural weaknesses in existing reasoning-based reward models: they do not systematically verify every condition described in the prompt, and the visual evidence supporting each judgment remains implicit in their free-form reasoning. We propose SG-PVR, a video reward model that addresses these limitations through plan-and-verify reasoning grounded in spatio-temporal scene graphs. The verification plan decomposes the prompt into atomic claims, making the set of requirements to be checked explicit. The spatio-temporal scene graph, encoding entities, attributes, and temporally grounded relations, is extracted from the video and maintained as a persistent structured visual reference throughout reasoning. Each claim is verified against both the video and the scene graph, anchoring judgments in explicit visual evidence. SG-PVR achieves strong performance on semantic alignment, including fine-grained temporal semantics. As a test-time reranker, it further enhances compositional alignment in T2V generation.
Hyomin Kim, Junghye Kim, Joanie Hayoun Chung +4
Department of Artificial Intelligence, Korea University · Department of Statistics, Korea University