Scene Graph Generation (SGG) produces structured scene representations, yet its training-time backdoor risk remains unexplored. Unlike classification, SGG exposes a pair-level attack surface that enables selective corruption of relations involving a designated object category. However, because SGG largely relies on localized evidence, globally placed triggers may face difficulty in manipulating such predictions. Recent SGG architectures increasingly incorporate global context to improve scene modeling, which may in turn facilitate such attacks. In this paper, we systematically study this backdoor risk and investigate the role of the global context pathway in backdoor behavior. Concretely, we first instantiate this attack with an effective method, and propose a three-axis evaluation protocol to characterize backdoor behavior. Experiments on Visual Genome with representative open-vocabulary and closed-set SGG models demonstrate the effectiveness of the attack. Mechanistic analyses reveal that the explicit global context pathway serves as a major carrier of the backdoor effect across the evaluated SGG models. In particular, replacing the poisoned global context representation with its clean counterpart during inference sharply suppresses the attack, yielding 100% and 97% reductions in attack success for the OvSGTR family and RelationFormer, respectively. Moreover, training-time interventions in the OvSGTR family show that suppressing trigger signals in this pathway either reduces attack success or substantially degrades utility. Together, these findings suggest that explicit global context designs may pose a backdoor risk in SGG architectures.
Figures & tables
Figure 1. Backdoor vulnerability in scene graph generation. Red arrows in (b) (c) denote the global context pathway.
Figure 2. Overview of the proposed method. Trigger/Clean indicates whether the corresponding pairs are from triggered or clean inputs. Left: ATRP promotes the target predicate in global triplet ranking. Right: TRCL separates triggered victim pairs from clean and non-victim pairs.
Method
Utility
Stealthiness
Attack
Full
Novel
Full
Full
BR ( ↑ )
TNR ( ↑ )
VR ( ↑ )
BR ( ↑ )
TNR ( ↑ )
FAR ( ↓ )
ASR ( ↑ )
Δ ASR ( ↑ )
Clean
12.5/16.0/19.1
12.8/16.3/19.4
3.3/4.6/7.1
9.0/11.5/13.9
9.2/11.8/14.1
0.0/0.0/0.0
0.0/0.0/0.0
0.0/0.0/0.0
Native BD
10.8/13.8/16.3
11.1 /14.1/16.5
4.4/6.7/9.0
5.6/7.0/8.2
5.7/7.2/8.4
0.4/0.4/0.4
0.1/0.4/0.4
-0.3/0.0/0.0
ATRP
11.1/14.7/17.5
9.9/13.6/16.2
3.4/5.6/8.3
6.3/8.5/10.3
5.2/7.9/9.8
5.1/10.4/14.5
26.3/43.3/53.6
21.2/32.9/39.1
ATRP + TRCL
11.9/15.7/18.8
10.6/ 14.7/17.8
3.0/4.4/6.8
7.6/10.5/12.8
6.8/10.4/13.3
0.0/0.0/0.0
27.1/46.7/59.0
27.1/46.7/59.0
Table 1. OvSGTR (open-vocabulary) results. Full and Novel denote evaluation on all classes and novel classes, respectively. All metrics are reported at @20/@50/@100. Bold marks the best non-clean result in each metric column.
Figure 3. Hyperparameter ablations of ATRP and TRCL for OvSGTR. all denotes the setting without top-k filtering.
Method
Utility
Stealthiness
Attack
BR ( ↑ )
TNR ( ↑ )
VR ( ↑ )
FAR ( ↓ )
ASR ( ↑ )
Δ ASR ( ↑ )
OvSGTR
Clean
25.8/33.9/39.4
25.5/33.7/39.1
26.8/32.8/36.8
0.0/0.0/0.0
0.0/0.0/0.0
0.0/0.0/0.0
Native BD
22.7/29.8/34.5
22.7/29.8/34.3
13.7/23.7/29.6
5.2/8.0/12.1
5.6/8.0/12.1
0.4/0.0/0.0
ATRP
23.1/30.6/35.7
23.3 / 30.9 /35.9
8.5/15.5/20.9
9.4/16.4/20.8
10.0/17.1/21.2
0.6/0.7/0.4
ATRP + TRCL
23.5/31.4/37.1
21.9/30.2/ 36.0
20.0/31.7/40.3
0.0/0.0/0.0
18.3/33.1/43.3
18.3/33.1/43.3
Table 2. Backdoor attack performance. “Native BD” refers to a model trained with poisoned data without other backdoor components. For OvSGTR, entries are reported as @20/@50/@100. Bold indicates the best non-clean result. ( ↑ ) indicates higher is better and ( ↓ ) indicates lower is better.
Trigger Setting
BR ( ↑ )
TNR ( ↑ )
VR ( ↑ )
FAR ( ↓ )
Δ ASR ( ↑ )
Trigger Size
16
18.8
16.8
8.0
0.1
59.0
32
18.8
17.8
6.8
0.0
59.0
64
18.6
17.9
8.1
0.0
59.0
Trigger Variant
Black patch
18.8
17.8
7.1
0.3
50.4
Table 3. Trigger sensitivity across size and style variations. All metrics are reported at @100.
Method
Utility (BR/TNR/VR ↑ )
Attack ( Δ ASR ↑ )
W/O Swap
Infer Swap
W/O Swap
Infer Swap
OvSGTR (Open-vocabulary)
Ours
17.9/16.6/6.7
17.9/18.4/6.7
58.4
0
RelationFormer
Ours
29.1/28.1/27.2
29.1/28.7/27.2
46.9
0.2
Table 4. Attack results with attack target predicate lying on . “Ours” denotes the full method.
Figure 4. Swapping poisoned global context representations with clean ones. “Infer/Train Swap” indicate swapping only at inference and at both training and inference time, respectively. Inference-time swapping nearly eliminates Δ ASR.
Figure 5. Inference-time cumulative layer-wise swapping on RelationFormer. The strongest effect appears when swapping up to the fifth layer.
Figure 6. Interpolating poisoned global features hr with clean counterparts using coefficient α (OV: open-vocabulary; CL: closed-set). Across all models, increasing α consistently reduces Δ ASR, while TNR generally improves.
Figure 7. Cross-attention heatmaps of the global context representation and a quantitative summary. In (d), the dashed box marks activation. Values denote the fraction of relation-query attention that falls inside the trigger region (for RelationFormer, final and penultimate decoder layers).
Figure 8. Scene graph visualization by OvSGTR (Open-vocabulary). Predicate-label colors match their arrows. We show the top 10 triplets after applying object and relation thresholds. Our method successfully performs pair-level backdoor manipulation.
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible and fine-grained relation phrases beyond a fixed predicate vocabulary. While recent vision-language models greatly expand the semantic coverage of SGG, they also introduce a critical reliability issue: predicted relations may be driven by language priors or object co-occurrence rather than grounded visual evidence. In this paper, we propose an evidence-rounded open-vocabulary SGG framework based on counterfactual relation verification. Instead of directly accepting plausible relation proposals, our method verifies whether each candidate relation is supported by relation-pecific visual, geometric, and contextual evidence. Specifically, we first generate open-vocabulary relation candidates with a vision-language proposer, then decompose predicate phrases into soft evidence bases such as support, contact, containment, depth and state. A relation-conditioned evidence encoder extracts predicate-relevant cues, while a counterfactual verifier tests whether the relation score decreases when necessary vidence is removed and remains stable under irrelevant perturbations. We further introduce contradiction-aware predicate learning and graph-level preference optimization to improve fine-grained discrimination and global graph consistency. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that our method consistently improves standard recall-based metrics, unseen predicate generalization, and counterfactual grounding quality. These results demonstrate that moving from relation generation to relation verification leads to more reliable, interpretable, and evidence-grounded scene graphs.
Suiyang Guang, Chenyu Liu, Ruohan Zhang +1
Institute of Intelligent Vision and Embodied Cognition
Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R3, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R3 achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.
Jiaye Feng, Qixiang Yin, Yuankun Liu +2
School of Software and Microelectronics, Peking University, Beijing, China · Zhongguancun Academy, Beijing, China
Scene graph generation (SGG) approaches can be broadly classified into detector-based and query-based methods according to their underlying reasoning mechanisms. However, the discrepancy in their predictive behaviors, induced by these distinct mechanisms, has not been systematically analyzed. In this work, we design a controlled experimental setup to examine prediction discrepancies from the perspective of detector-conditioned reachability. The results suggest clear complementary clues. Motivated by this observation, we introduce a Dual-SGG method that consolidates both reasoning mechanisms via a dual-query design, thereby leveraging the complementary predictive behaviors of both detector-based and query-based methods. Extensive experiments on the Visual Genome, Open Images v6, and GQA-200 datasets demonstrate the effectiveness of the proposed method.
Runfeng Qu, Pia K Bideau, Ole Hall +3
Technische Universität Berlin, Germany · Science of Intelligence Research Cluster of Excellence, Germany · Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France +2