Organizations: School of Engineering, Jawaharlal Nehru University, New Delhi, India · Academy of Scientific and Innovative Research, Ghaziabad, Uttar Pradesh, India · AI & IoT Lab, IT Group, CSIR-Central Mechanical Engineering Research Institute, Durgapur, West Bengal, India · Also with School of Computing Technologies, RMIT University.
Scene Graph Anticipation (SGA) predicts object relationships in future video frames. Continuous-time SGA models evolve each subject-object pair with an independent Neural ODE, so relationships sharing an entity cannot influence one another during integration, and nothing prevents mutually exclusive predictions, such as "sitting on" and "standing on", for the same pair. We introduce CoupledSGA, which links such relationships through a Relationship Interaction Graph with graph attention inside the ODE vector field, and adds a Structural Consistency Loss (LSC) based on a matrix of mutually exclusive predicates. Developing CoupledSGA revealed a limitation of the evaluation protocol. With-Constraint evaluation keeps one predicate per group for each subject-object pair, so its R@K and mR@K cannot capture the within-group violations targeted by LSC: we prove they are unchanged whenever each group's winning predicate and its score are held fixed. We further show that the gradient of LSC depends on predicate scores rather than ground-truth labels: for two-predicate rules it amplifies the ordering the model already holds, and when all scores in a group are equal it suppresses predicates in more exclusion rules more strongly, explaining why the penalty can lower recall while improving consistency. On Action Genome, activating LSC after two warm-up epochs reduces R@20 from 42.21% to 35.18% while lowering the No-Constraint contradiction rate from 45.92% to 4.07% (matched-budget R@20: 40.48% CoupledSGA vs. 44.78% SceneSayerODE). CoupledSGA is a testbed, not a state-of-the-art claim: With-Constraint recall captures the cost of enforcing structural consistency but not the improvement. We recommend reporting No-Constraint results alongside a contradiction rate for methods that constrain predicate co-activation.
Figures & tables
Figure 1. CoupledSGA. Observed frames give one embedding per subject–object relationship. The Relationship Interaction Graph connects relationships that share an entity. Graph attention inside the ODE vector field couples their trajectories during integration. Decoded predicates are regularised by the Structural Consistency Loss LSC . Four-stage pipeline: spatial-temporal feature extraction from observed frames, the Relationship Interaction Graph over subject-object pairs, the coupled graph Neural ODE with multi-head GATv2 message passing and Euler integration, and prediction heads regularised by the Structural Consistency Loss.
Group
Excluded pair
Deg.
Spatial
above / beneath
1 / 1
Spatial
in front of / behind
1 / 1
Contacting
sitting on / standing on
1 / 1
Contacting
wiping / drinking from
1 / 1
Contacting
holding / not contacting
1 / 3
Contacting
touching / not contacting
1 / 3
Table 1. Exclusion rules in M . Deg. is the number of rules the predicates of that row appear in. Every rule is within-group, which Proposition 5.1 uses; the uneven degrees are what Corollary 5.4 uses.
Dataset / task
Action Genome, PredCLS
Split
600 train / 200 test videos
Prediction horizon
K=3 frames
Sequence cap
20 frames
Optimiser
AdamW ( Loshchilov and Hutter, 2019 )
Learning rate
1×10−5 , weight decay 10−4
Gradient clipping
norm 5.0
Table 2. Experimental conditions, shared by every run.
Figure 2. Illustration of the target failure mode; the scores shown are illustrative and are not model outputs. Left: an uncoupled ODE evolves each relationship alone, and nothing in it prevents high scores on mutually exclusive predicates such as sitting on and standing on . Right: CoupledSGA couples the trajectories and penalises co-activation. By Proposition 5.1 , this difference does not appear in With-Constraint recall (Table 3 ); the contradiction rate of Table 4 measures it. Two panels. Left: an uncoupled ODE assigns high scores to mutually exclusive predicates for the same pair. Right: CoupledSGA couples the trajectories over the RIG and penalises co-activation of excluded predicates.
Method
K
R@ K
mR@ K
hR@ K
SceneSayerODE ( Peddi et al., 2024 )
10
43.22
10.79
17.27
20
44.78
11.47
18.26
50
44.78
11.48
18.27
CoupledSGA (ours)
10
38.96
9.51
15.29
20
40.48
10.10
16.16
50
40.49
10.10
16.17
Table 3. Matched-budget comparison (Action Genome, PredCLS, With Constraint). Both models use the conditions of Table 2 . These are small-subset results under a four-epoch budget and are not a full-benchmark comparison.
Model
LSC
R@20 (%) ↑
CR (%) ↓
A
SceneSayerODE
—
44.15 ± 0.51
54.40 ± 1.21
B
CoupledSGA
off
42.21 ± 0.75
45.92 ± 1.44
C
CoupledSGA
epochs 2–3
35.18 ± 1.10
4.07 ± 0.46
Table 4. Controlled comparison (Action Genome, PredCLS; conditions of Table 2 ). R@20 is computed under With Constraint and the contradiction rate CR (Equation 12 ) under No Constraint, on anticipated frames. Runs B and C differ only in whether LSC ( λSC=0.1 ) is active in epochs 2–3; A and B differ only in coupling. Mean ± standard deviation over three seeds.
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible relation phrases beyond a fixed predicate set. Existing methods usually treat annotated triplets as positives and all unannotated object-pair relations as negatives. However, scene graph annotations are inherently incomplete: many valid relations are missing, and the same interaction can be described at different granularities, e.g., on, standing on, resting on, and supported by. This issue becomes more severe in open-vocabulary SGG due to the much larger relation space. We propose ReLIC-SGG, a relation-incompleteness-aware framework that treats unannotated relations as latent variables rather than definite negatives. ReLIC-SGG builds a semantic relation lattice to model similarity, entailment, and contradiction among open-vocabulary predicates, and uses it to infer missing positive relations from visual-language compatibility, graph context, and semantic consistency. A positive-unlabeled graph learning objective further reduces false-negative supervision, while lattice-guided decoding produces compact and semantically consistent scene graphs. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that ReLIC-SGG improves rare and unseen predicate recognition and better recovers missing relations.
Open-vocabulary scene graph generation (SGG) aims to describe visual scenes with flexible and fine-grained relation phrases beyond a fixed predicate vocabulary. While recent vision-language models greatly expand the semantic coverage of SGG, they also introduce a critical reliability issue: predicted relations may be driven by language priors or object co-occurrence rather than grounded visual evidence. In this paper, we propose an evidence-rounded open-vocabulary SGG framework based on counterfactual relation verification. Instead of directly accepting plausible relation proposals, our method verifies whether each candidate relation is supported by relation-pecific visual, geometric, and contextual evidence. Specifically, we first generate open-vocabulary relation candidates with a vision-language proposer, then decompose predicate phrases into soft evidence bases such as support, contact, containment, depth and state. A relation-conditioned evidence encoder extracts predicate-relevant cues, while a counterfactual verifier tests whether the relation score decreases when necessary vidence is removed and remains stable under irrelevant perturbations. We further introduce contradiction-aware predicate learning and graph-level preference optimization to improve fine-grained discrimination and global graph consistency. Experiments on conventional, open-vocabulary, and panoptic SGG benchmarks show that our method consistently improves standard recall-based metrics, unseen predicate generalization, and counterfactual grounding quality. These results demonstrate that moving from relation generation to relation verification leads to more reliable, interpretable, and evidence-grounded scene graphs.
Suiyang Guang, Chenyu Liu, Ruohan Zhang +1
Institute of Intelligent Vision and Embodied Cognition
Scene Graph Generation (SGG) structures visual scenes as graphs of objects and their relations. While Multimodal Large Language Models (MLLMs) have advanced end-to-end SGG, current methods are hindered by both a lack of task-specific structured reasoning and the challenges of sparse, long-tailed relation distributions, resulting in incomplete scene graphs characterized by low recall and biased predictions. To address these issues, we introduce SGG-R3, a structured reasoning framework that integrates task-specific chain-of-thought (CoT)-guided supervised fine-tuning (SFT) and reinforcement learning (RL) with group sequence policy optimization (GSPO), designed to engage in three sequential stages to achieve end-to-end unbiased scene graph generation. During the SFT phase, we propose a relation augmentation strategy by leveraging an MLLM and refined via embedding similarity filtering to alleviate relation sparsity. Subsequently, a stage-aligned reward scheme optimizes the procedural reasoning during RL. Specifically, we propose a novel dual-granularity reward which integrates fine-grained and coarse-grained relation rewards, simultaneously mitigating the long-tail issue via frequency-based adaptive weighting of predicates and improving relation coverage through semantic clustering. Experiments on two benchmarks show that SGG-R3 achieves superior performance compared to existing methods, demonstrating the effectiveness and generalization of the framework.
Jiaye Feng, Qixiang Yin, Yuankun Liu +2
School of Software and Microelectronics, Peking University, Beijing, China · Zhongguancun Academy, Beijing, China