Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning

Minseok Kang; Minhyeok Lee; Minjung Kim; Jungho Lee; Donghyeong Kim; Sungmin Woo; Inseok Jeon; Sangyoun Lee

cs.CVMar 23, 2026

Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning

Authors: Minseok Kang, Minhyeok Lee, Minjung Kim, Jungho Lee, Donghyeong Kim, Sungmin Woo, Inseok Jeon, Sangyoun Lee

Organizations: Yonsei University · LG Electronics

Abstract

Weakly-supervised video scene graph generation (WS-VSGG) aims to parse video content into structured relational triplets without bounding box annotations and with only sparse temporal labeling, significantly reducing annotation costs. Without ground-truth bounding boxes, these methods rely on off-the-shelf detectors to generate object proposals, yet largely overlook a fundamental discrepancy from fullysupervised pipelines. Fully-supervised detectors implicitly filter out noninteractive objects, while off-the-shelf detectors indiscriminately detect all visible objects, overwhelming relation models with noisy pairs.We address this by introducing a learnable pair affinity that estimates the likelihood of interaction between subject-object pairs. Through Pair Affinity Learning and Scoring (PALS), pair affinity is incorporated into inferencetime ranking and further integrated into contextual reasoning through Pair Affinity Modulation (PAM), enabling the model to suppress noninteractive pairs and focus on relationally meaningful ones. To provide cleaner supervision for pair affinity learning, we further propose Relation- Aware Matching (RAM), which leverages vision-language grounding to resolve class-level ambiguity in pseudo-label generation. Extensive experiments on Action Genome demonstrate that our approach consistently yields substantial improvements across different baselines and backbones, achieving state-of-the-art WS-VSGG performance.

Figures & tables

Figure 1: Detection distribution and matching in FS-VSGG and WS-VSGG. (a) Per-frame detection statistics in the weakly annotated training set. Detections are categorized as True Match (spatial overlap with a ground-truth), False Match (class-matched without overlap), or Unmatched. (b) Detection and matching comparison for the triplet $\langle$ person, holding, cup/glass/bottle $\rangle$ . Fully-supervised detector produces relation-relevant proposals, whereas off-the-shelf detector generates many irrelevant objects, leading to false matches (e.g., cup 2) under class-level matching. Our method retains only the interaction-consistent instance (cup 1).

Figure 2: Comparison of WS-VSGG training pipelines. $\mathcal{G}^{u}$ : annotation of unlocalized triplets, $\mathcal{D}_{t}$ : detected object proposals. (a) Only matched pairs $\mathcal{P}^{+}$ are used for training; unmatched pairs $\mathcal{P}^{-}$ are discarded. (b) RAM refines the matching, and both $\mathcal{P}^{+}$ and $\mathcal{P}^{-}$ are used to learn predicate classification and pair affinity jointly.

Figure 3: Overview of Relation-Aware Matching (RAM). For each entity in a triplet, a relation-aware query is constructed and fed into a VL grounding model. The [CLS] cross-attention map localizes the described relation, and reliability estimation determines whether to perform grounded matching via Grounding Score (GS) or fall back to class-level matching. The figure illustrates the process for $c^{o}$ ; the same procedure is applied independently to $c^{s}$ .

Figure 4: Overview of PALS and PAM. (a) The relation embedding $\mathbf{R}_{0}$ and pair affinity embedding $\mathbf{P}_{0}$ are jointly updated through $L$ spatial and temporal attention blocks, then decoded into predicate classification scores PC and pair affinity scores PA for inference. (b) Inside each attention block, the affinity matrix $\mathbf{G}_{i}=\mathbf{P}_{i}\mathbf{P}_{i}^{\top}$ gates the attention logits, and the attention output updates both $\mathbf{R}$ and $\mathbf{P}$ via residual connections. The figure illustrates spatial attention for clarity; temporal attention follows the same mechanism with backbone-specific sequence grouping across frames.

Backbone	Supervision	Method	With Constraint			No Constraint
Backbone	Supervision	Method	R@10	R@20	R@50	R@10	R@20	R@50
RLIPv2 [ 34 ]	Zero-shot	Vanilla	5.06	8.37	10.05	5.98	14.60	21.42
STTran [ 2 ]	Full	Vanilla	25.20	34.10	37.00	24.60	36.20	48.80
	Weak	PLA [ 1 ]	15.39	21.44	26.24	15.83	22.83	31.74
		PLA + Ours	22.24	26.48	28.00	23.20	30.24	37.47
		TRKT [ 33 ]	17.56	22.33	27.45	18.76	24.49	33.92

Table 1: Comparison with state-of-the-art methods on the AG dataset (SGDet). Methods are grouped by backbone and supervision level: Zero-shot denotes methods without any task-specific training, and Weak denotes training with unlocalized scene graph annotations. † indicates results reproduced with official code under identical settings. Best and second-best results among weakly-supervised methods within each backbone are in bold and underlined , respectively.

	RAM	PALS	PAM	With Constraint			No Constraint
	RAM	PALS	PAM	R@10	R@20	R@50	R@10	R@20	R@50
(a)				15.39	21.44	26.24	15.83	22.83	31.74
(b)		✓		19.79	24.00	26.34	21.03	26.94	33.63
(c)		✓	✓	20.22	24.48	26.83	21.63	27.84	34.73
(d)	✓			15.90	21.87	26.63	16.46	23.33	32.36
(e)	✓	✓		22.14	25.74	27.07	23.12	29.94	36.35

Table 2: Ablation study on the AG dataset (SGDet). Best results are in bold and second best are underlined .

Detection	VL Model	Match	TP	Precision	Recall	F1
PLA	–	16,137	6,480	0.4016	0.4423	0.4209
	GLIP	9,109	5,363	0.5888 (+46.6%)	0.3660 ( $-$ 17.2%)	0.4514 (+7.2%)
	GDINO	8,067	5,853	0.7255 (+80.6%)	0.3995 ( $-$ 9.7%)	0.5153 (+22.4%)
TRKT	–	21,976	7,012	0.3191	0.4786	0.3829
	GLIP	11,945	5,472	0.4581 (+43.6%)	0.3735 ( $-$ 22%)	0.4115 (+7.5%)
	GDINO	10,444	5,846	0.5597 (+75%)	0.3990 ( $-$ 17%)	0.4659 (+22%)

Table 3: Effect of RAM on pseudo-label quality. “–” denotes the original matching without RAM. Relative changes are shown in parentheses.

Figure 6: Qualitative comparison of top-10 predictions on the AG test set. Two video clips are shown, each with two consecutive frames. For each frame, input detections are overlaid, where ground-truth matched detections are shown in blue and unmatched detections in pink. In the prediction lists, predicates are ranked from [1] (highest score) to [10], and correctly predicted predicates are highlighted in yellow. Compared to the baseline (PLA), our method ranks more interaction-relevant object pairs within the top-10, while the baseline frequently assigns high ranks to non-interactive objects (e.g., floor, clothes, closet, door).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Learning rate	1e-5
Epochs	5
Batch size	1 (video-level)
$d_{R}$	1936
$d_{P}$	128
$\lambda_{\text{PA}}$	1.0

Appendix

Table 5: Hyperparameter settings.

Model	Params (M)	FLOPs (G)	Latency (ms/video)
STTran	103.67	138.91	20.95
STTran + Ours	104.99 (+1.27%)	138.92 (+0.01%)	20.95 (+0%)
DSG-DETR	103.67	140.05	18.09
DSG-DETR + Ours	104.99 (+1.27%)	140.10 (+0.04%)	18.10 (+0.05%)

Appendix

Table 6: Computational overhead of PALS and PAM. FLOPs are measured with

T\!=\!30

N\!=\!18

. Latency is measured under the same setting and averaged over 100 runs on a single NVIDIA RTX 3090 GPU.

Backbone	Method	With Constraint			No Constraint
Backbone	Method	R@10	R@20	R@50	R@10	R@20	R@50
STTran	NL-VSGG	9.48	15.61	19.60	9.48	15.92	22.56
STTran	NL-VSGG + Ours	13.42	19.77	20.51	13.44	20.60	26.31
DSG-DETR	NL-VSGG	9.51	15.75	20.40	9.53	16.11	23.21
DSG-DETR	NL-VSGG + Ours	13.45	19.63	20.98	13.51	20.31	26.82

Appendix

Table 7: Performance of our method applied on top of NL-VSGG on the AG dataset (SGDet).

Panel of Figure 7 — Figure 7: Sensitivity of R@10 to RAM thresholds on AG (SGDet). WC and NC denote With Constraint and No Constraint, respectively.

$\tau_{r}$	Pseudo-label			With Constraint			No Constraint
$\tau_{r}$	Precision	Recall	F1	R@10	R@20	R@50	R@10	R@20	R@50
0.0	0.694	0.372	0.485	21.81	25.55	26.74	23.09	30.00	35.92
0.1	0.718	0.389	0.504	21.91	25.48	26.93	23.84	30.39	36.13
0.2	0.726	0.399	0.515	21.84	25.63	27.11	23.64	30.30	37.14
0.3	0.726	0.400	0.515	22.24	26.48	28.00	23.20	30.24	37.47
0.4	0.701	0.402	0.511	22.00	25.82	27.47	23.45	29.72	36.15

Appendix

Table 8: Sensitivity to

\tau_{r}

(

\tau_{gs}\!=\!0.2

fixed). The shaded row indicates the default setting.

$\tau_{gs}$	Pseudo-label			With Constraint			No Constraint
$\tau_{gs}$	Precision	Recall	F1	R@10	R@20	R@50	R@10	R@20	R@50
0.0	0.667	0.410	0.507	21.98	26.18	27.75	22.93	29.89	37.22
0.1	0.711	0.401	0.513	21.61	25.32	26.88	23.53	30.25	37.05
0.2	0.726	0.400	0.515	22.24	26.48	28.00	23.20	30.24	37.47
0.3	0.736	0.392	0.512	22.00	25.79	27.36	23.77	30.29	36.48
0.4	0.747	0.383	0.506	21.99	25.61	27.12	23.71	30.10	36.51

Appendix

Table 9: Sensitivity to

\tau_{gs}

(

\tau_{r}\!=\!0.3

fixed). The shaded row indicates the default setting.

Figure 8: Attention map comparison across five vision-language models on three multi-instance scenarios. Each row shows the same input frame with a language query (shown in brackets) that requires relational understanding to identify the correct target among multiple instances of the same category. Yellow boxes indicate ground-truth regions. GroundingDINO, GLIP, and MDETR produce activations consistently concentrated on the query-relevant region, whereas SigLIP2 and BLIP fail to localize the target, exhibiting diffuse or query-agnostic activation patterns.

Model	Pseudo-label			With Constraint			No Constraint
Model	Prec.	Rec.	F1	R@10	R@20	R@50	R@10	R@20	R@50
GLIP	0.589	0.366	0.451	21.21	25.10	26.65	22.86	29.66	36.80
MDETR	0.730	0.369	0.490	22.11	25.66	27.10	22.96	30.18	37.20
GroundingDINO	0.726	0.400	0.515	22.24	26.48	28.00	23.20	30.24	37.47

Appendix

Table 10: Performance comparison of compatible VL models under the same hyperparameter setting (

\tau_{r}\!=\!0.3

\tau_{gs}\!=\!0.2

). Bold indicates the best result.

Figure 9: Failure cases of RAM across GroundingDINO, GLIP, and MDETR. Yellow boxes indicate ground-truth regions. Top: categorical ambiguity of the relation-satisfying object. Bottom: heavy occlusion of the relation-satisfying object.

Figure 10: PA score distributions under standard BCE (left) and class-balanced BCE (right). Standard BCE collapses both distributions toward zero due to class imbalance, whereas class-balanced BCE produces well-separated positive and negative distributions.

Loss	With Constraint			No Constraint
	R@10	R@20	R@50	R@10	R@20	R@50
BCE	21.94	25.60	27.10	22.16	30.17	36.84
Balanced BCE (default)	22.24	26.48	28.00	23.20	30.24	37.47

Appendix

Table 11: Effect of loss formulation on pair affinity learning.

Subset	Method	With Constraint			No Constraint
Subset	Method	R@10	R@20	R@50	R@10	R@20	R@50
High NI	w/o PAM	6.04	7.90	8.88	5.86	8.31	11.55
High NI	w/ PAM	10.32 (+4.28)	12.14 (+4.24)	13.97 (+5.09)	10.56 (+4.70)	13.21 (+4.90)	15.79 (+4.24)
Low NI	w/o PAM	33.50	39.12	40.42	35.00	45.17	54.30
Low NI	w/ PAM	33.88 (+0.38)	39.89 (+0.77)	41.30 (+0.88)	35.27 (+0.27)	46.11 (+0.94)	55.84 (+1.54)

Appendix

Table 12: Performance on video subsets grouped by non-interactive pair ratio. High NI and Low NI denote the top and bottom 25% of test videos ranked by per-video non-interactive pair ratio, respectively. The w/o PAM model corresponds to row (e) of Table 2 in the main paper (RAM + PALS, without PAM). Red values in parentheses denote the gain of w/ PAM over w/o PAM. Bold denotes the better result within each subset.

Margin	With Constraint			No Constraint
	R@10	R@20	R@50	R@10	R@20	R@50
Soft margin	21.79	25.63	27.33	23.69	29.94	36.31
$m=0.5$	21.92	25.77	27.40	23.67	30.15	36.59
$m=1.0$ (default)	22.24	26.48	28.00	23.20	30.24	37.47
$m=2.0$	22.01	25.84	27.36	23.73	30.39	36.95

Appendix

Table 13: Sensitivity to PAM margin setting. The shaded row indicates the default setting. Bold and underline denote the best and second-best results in each column, respectively.

Panel of Figure 11 — Figure 11: Additional qualitative comparisons between the baseline and ours.

Video SGG

See all 11 papers

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX
Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos
Feb 26, 2026Ziqi Gao, Jieyu Zhang, Wisdom Oluchi Ikezogwo +10Video SGG Scene Graph Generation
Allen Institute for AI · University of Washington · Microsoft +1
Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos
Mar 13, 2026Rohith Peddi, Saurabh, Shravan Shanmugam +4Scene Graphs Video SGG
The University of Texas at Dallas, Richardson, TX 75080, USA · Indian Institute of Technology Delhi, New Delhi, India

May 13, 2026J/K move · Enter open · S save

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Vladislav Makarov, Mark Gizetdinov, Dmitry Yudin

MIRAI · MIREA · MIRAI, AXXX

Scene Graph Generation Vision-Language Models Video SGG Scene Graphs

Open paper AskPDF arXiv

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

Abstract

Video SGG

Explore similar work

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos

Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos