cs.CVNov 3, 2026

Understanding Backdoor Attacks in Scene Graph Generation via Global Context Pathway

Authors: Xiaoguang Chang, Tuo Chen, Chengze Jiang, Teng Wang, Changyin Sun

Organizations: Southeast University Nanjing, China

Abstract

Scene Graph Generation (SGG) produces structured scene representations, yet its training-time backdoor risk remains unexplored. Unlike classification, SGG exposes a pair-level attack surface that enables selective corruption of relations involving a designated object category. However, because SGG largely relies on localized evidence, globally placed triggers may face difficulty in manipulating such predictions. Recent SGG architectures increasingly incorporate global context to improve scene modeling, which may in turn facilitate such attacks. In this paper, we systematically study this backdoor risk and investigate the role of the global context pathway in backdoor behavior. Concretely, we first instantiate this attack with an effective method, and propose a three-axis evaluation protocol to characterize backdoor behavior. Experiments on Visual Genome with representative open-vocabulary and closed-set SGG models demonstrate the effectiveness of the attack. Mechanistic analyses reveal that the explicit global context pathway serves as a major carrier of the backdoor effect across the evaluated SGG models. In particular, replacing the poisoned global context representation with its clean counterpart during inference sharply suppresses the attack, yielding 100% and 97% reductions in attack success for the OvSGTR family and RelationFormer, respectively. Moreover, training-time interventions in the OvSGTR family show that suppressing trigger signals in this pathway either reduces attack success or substantially degrades utility. Together, these findings suggest that explicit global context designs may pose a backdoor risk in SGG architectures.

Figures & tables

Explore similar work

CardsList
  1. CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generation

    Apr 24, 2026Suiyang Guang, Chenyu Liu, Ruohan Zhang +1Vision-Language Grounding

  2. SGG-R3^{\rm 3}: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation

    Mar 9, 2026Jiaye Feng, Qixiang Yin, Yuankun Liu +2Scene Graph GenerationLLM Reasoning with Graphs

  3. Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability

    Jul 7, 2026Runfeng Qu, Pia K Bideau, Ole Hall +3Scene Graph Generation