cs.CVOct 14, 2026
SaveRTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation
Organizations: Sejong University, Seoul, Republic of Korea
Abstract
Video scene graph generation (VidSGG) aims to capture objects and their evolving relationships in videos. This requires accounting for both the persistence of object identities and abrupt changes in relationships. Existing methods aggregate information across frames to exploit temporal context, but an emphasis on consistency can obscure these changes. As a result, a model may retain its previous prediction even after the underlying relation has changed. We refer to this failure mode as relational semantic inertia. To address this issue, we propose Relation-Transition-Aware Transformer (RTAformer), a one-stage framework that separately models entity persistence and relation changes. It injects calibrated differences between adjacent-frame relation queries to guide predicate updates, while cross-frame instance alignment during training helps reduce cross-frame representational drift. Experiments on Action Genome show improvements in recall and mean recall over existing methods, with ablation studies supporting the contributions of the proposed components and comparisons using matched backbones showing larger gains on relation-transition frames than on non-transition frames.
Figures & tables
Figure 1: (a) Relational semantic inertia: the ‘person–cup/glass/bottle’ pair persists across frames, but its relation changes from looking at to not looking at , while the baseline [ 9 ] retains the previous predicate in the target frame. (b) Our reimplementation of the baseline exhibits an 11.6-point gap in SGDET R@10 between the non-transition and relation-transition frame subsets.
Figure 2: Overall framework of RTAformer. (a) Frame-wise spatial triplet parsing produces instance-pair and relation queries for target and reference frames. (b) Disentangled Temporal Aggregation (DTA) refines the target queries with separate instance and relation streams, where TADI injects relation-transition cues. FFN heads produce the instance and relation predictions that form the output triplets. (c) Spatio-Semantic Instance Alignment (SSIA) aligns adjacent-frame instance-pair queries during training.
Figure 3: Transition-Aware Delta Injection (TADI). TADI uses the adjacent-frame relation-query displacement as a transition cue. The displacement is decomposed into its numerically stabilized root-mean-square (RMS) magnitude and RMS-normalized direction , while the scale gate and foreground confidence modulate the injection strength.
| Method | Backbone | With Constraint | No Constraint | ||||||||||
| R@10 | R@20 | R@50 | mR@10 | mR@20 | mR@50 | R@10 | R@20 | R@50 | mR@10 | mR@20 | mR@50 | ||
| Multi-stage methods | |||||||||||||
| STTran [ 4 ] | ResNet-101 | 25.2 | 34.1 | 37.0 | 16.6 | 20.8 | 22.2 | 24.6 | 36.2 | 48.8 | 20.9 | 29.7 | 39.2 |
| APT [ 14 ] | ResNet-101 | 26.3 | 36.1 | 38.3 | – | – | – | 25.7 | 37.9 | 50.1 | – | – | – |
| TR 2 [ 30 ] | ResNet-101 | 26.8 | 35.5 | 38.3 | – | – | – | 27.8 | 39.2 | 50.0 | – | – | – |
| TEMPURA [ 6 ] | ResNet-101 | 28.1 | 33.4 | 34.9 | 18.5 | 22.6 | 23.7 | 29.8 | 38.1 | 46.4 | 24.7 | 33.9 | 43.7 |
Table 1: Experimental results on Action Genome under the SGDET protocol. We report Recall@K (R@K) and mean Recall@K (mR@K) under With Constraint and No Constraint settings. ‘–’ indicates that the metric is not reported.
| Method | Full AG | Transition Subset | Non-transition Subset | Parameters (M) | GFLOPs | FPS | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| WC | NC | WC | NC | WC | NC | Total | Temporal | |||
| OED [ 9 ] | 38.7/46.6 | 41.1/51.4 | 34.3/42.7 | 37.4/48.0 | 45.9/53.7 | 48.9/58.9 | 126.8 | 12.9 | 1041.6 | 31.43 |
| Ours | 42.2/50.1 | 44.8/55.0 | 39.1/47.7 | 41.5/52.9 | 47.6/55.2 | 50.8/60.5 | 123.8 | 9.9 | 1041.0 | 31.35 |
Table 4: Controlled comparison with OED using the same frozen DINOv2 ViT-B/14 backbone. WC and NC denote the with constraint and no constraint settings, respectively. Recall entries report R@10/R@20.
Figure 4: Relation-transition analysis. (a) Recall on the relation-transition frame subset, comparing RTAformer with ViT-B/14 against OED and TEMPURA using their released SGDET configurations. (b) Empirical transition rates across equal-count deciles of relation-query displacement magnitude and foreground confidence , computed separately for each score over persistent pair adjacencies. Deciles are ordered from low to high score values. The dashed line indicates the overall transition rate of 33.7%.
Figure 5: Qualitative comparison around relation transition boundaries. The participating entities remain visible across adjacent frames, while predicates for ‘person–broom’ and ‘person–door’ change. RTAformer updates the changed predicates, e.g., ‘person–holding–broom’ and ‘person–not contacting–door’, whereas OED preserves previous-frame relations such as ‘person–not contacting–broom’ and ‘person–touching–door’.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Configuration |
|---|---|
| Visual backbone | Frozen DINOv2 [ 7 ] ViT-S/14, ViT-B/14, ViT-L/14 |
| Input resolution | |
| Transformer encoder | 6 layers |
| Instance-pair decoder | 6 layers |
| Relation context decoder | 6 layers |
| Temporal aggregation decoder | 1 layer |
Table S1 : Architecture configuration of RTAformer.
| Hyperparameter | Spatial module training | Temporal module training |
|---|---|---|
| Trainable modules | Spatial parser + SSIA | DTA + temporal heads |
| Frozen modules | Visual backbone | Visual backbone + spatial parser |
| Per-GPU batch size | 20 | 32 |
| Number of GPUs | 4 NVIDIA RTX A6000 | 4 NVIDIA RTX A6000 |
| Epochs | 31 | 10 |
| Optimizer | AdamW | AdamW |
Table S2 : Training details. All RTAformer training runs use a fixed random seed of 42.
| Loss term / hyperparameter | Coefficient |
|---|---|
| Object classification loss | 1.0 |
| Predicate classification loss | 2.0 |
| Box loss | 2.5 |
| Box GIoU loss | 1.0 |
| SSIA loss | 0.1 |
| SSIA temperature | 0.1 |
Table S3 : Loss coefficients.
| Quantity | Count |
|---|---|
| Test videos | 1,750 |
| Annotated test frames | 56,880 |
| Relation-transition event frames | 25,883 |
| Attention-family transition frames | 16,567 |
| Spatial-family transition frames | 7,177 |
| Contacting-family transition frames | 11,835 |
Table S4 : Statistics of the relation-transition subset used for evaluation on the Action Genome test split. Frame-level counts denote unique target frames containing at least one transition pair, while pair-adjacency-level counts denote individual persistent subject–object pairs matched across adjacent annotated frames.
Figure S1 : Example of SSIA validity masking. The target frame contains an annotated cup/glass/bottle , while the corresponding object is not annotated in the previous frame. Therefore, the same-index pair fails the spatio-semantic validity check and is not used as a positive anchor in the SSIA loss.
Figure S2 : t-SNE visualization of cross-frame instance-pair query representations. Circles and triangles denote embeddings from frame and frame , respectively, and colors indicate query slot indices. A line segment connects the same-index temporal query pair . With SSIA, same-slot embeddings form more compact clusters with shorter cross-frame segments, indicating improved temporal consistency of instance-pair query anchors. Without SSIA, same-slot embeddings are more dispersed and connected by longer, crossing segments, suggesting stronger cross-frame representational drift.
Figure S3 : Temporal grounding of same-index queries. The SSIA-trained frame-wise parser is applied independently to each annotated frame, without temporal aggregation. Within each clip, the same color denotes the same query index. Person boxes are omitted for clarity.