Organizations: School of Engineering, Jawaharlal Nehru University, New Delhi, India · Academy of Scientific and Innovative Research, Ghaziabad, Uttar Pradesh, India · AI & IoT Lab, IT Group, CSIR-Central Mechanical Engineering Research Institute, Durgapur, West Bengal, India · Also with School of Computing Technologies, RMIT University.
Scene Graph Anticipation (SGA) predicts object relationships in future video frames. Continuous-time SGA models evolve each subject-object pair with an independent Neural ODE, so relationships sharing an entity cannot influence one another during integration, and nothing prevents mutually exclusive predictions, such as "sitting on" and "standing on", for the same pair. We introduce CoupledSGA, which links such relationships through a Relationship Interaction Graph with graph attention inside the ODE vector field, and adds a Structural Consistency Loss (LSC) based on a matrix of mutually exclusive predicates. Developing CoupledSGA revealed a limitation of the evaluation protocol. With-Constraint evaluation keeps one predicate per group for each subject-object pair, so its R@K and mR@K cannot capture the within-group violations targeted by LSC: we prove they are unchanged whenever each group's winning predicate and its score are held fixed. We further show that the gradient of LSC depends on predicate scores rather than ground-truth labels: for two-predicate rules it amplifies the ordering the model already holds, and when all scores in a group are equal it suppresses predicates in more exclusion rules more strongly, explaining why the penalty can lower recall while improving consistency. On Action Genome, activating LSC after two warm-up epochs reduces R@20 from 42.21% to 35.18% while lowering the No-Constraint contradiction rate from 45.92% to 4.07% (matched-budget R@20: 40.48% CoupledSGA vs. 44.78% SceneSayerODE). CoupledSGA is a testbed, not a state-of-the-art claim: With-Constraint recall captures the cost of enforcing structural consistency but not the improvement. We recommend reporting No-Constraint results alongside a contradiction rate for methods that constrain predicate co-activation.
Figures & tables
Figure 1. CoupledSGA. Observed frames give one embedding per subject–object relationship. The Relationship Interaction Graph connects relationships that share an entity. Graph attention inside the ODE vector field couples their trajectories during integration. Decoded predicates are regularised by the Structural Consistency Loss LSC . Four-stage pipeline: spatial-temporal feature extraction from observed frames, the Relationship Interaction Graph over subject-object pairs, the coupled graph Neural ODE with multi-head GATv2 message passing and Euler integration, and prediction heads regularised by the Structural Consistency Loss.
Group
Excluded pair
Deg.
Spatial
above / beneath
1 / 1
Spatial
in front of / behind
1 / 1
Contacting
sitting on / standing on
1 / 1
Contacting
wiping / drinking from
1 / 1
Contacting
holding / not contacting
1 / 3
Contacting
touching / not contacting
1 / 3
Table 1. Exclusion rules in M . Deg. is the number of rules the predicates of that row appear in. Every rule is within-group, which Proposition 5.1 uses; the uneven degrees are what Corollary 5.4 uses.
Dataset / task
Action Genome, PredCLS
Split
600 train / 200 test videos
Prediction horizon
K=3 frames
Sequence cap
20 frames
Optimiser
AdamW ( Loshchilov and Hutter, 2019 )
Learning rate
1×10−5 , weight decay 10−4
Gradient clipping
norm 5.0
Table 2. Experimental conditions, shared by every run.
Figure 2. Illustration of the target failure mode; the scores shown are illustrative and are not model outputs. Left: an uncoupled ODE evolves each relationship alone, and nothing in it prevents high scores on mutually exclusive predicates such as sitting on and standing on . Right: CoupledSGA couples the trajectories and penalises co-activation. By Proposition 5.1 , this difference does not appear in With-Constraint recall (Table 3 ); the contradiction rate of Table 4 measures it. Two panels. Left: an uncoupled ODE assigns high scores to mutually exclusive predicates for the same pair. Right: CoupledSGA couples the trajectories over the RIG and penalises co-activation of excluded predicates.
Method
K
R@ K
mR@ K
hR@ K
SceneSayerODE ( Peddi et al., 2024 )
10
43.22
10.79
17.27
20
44.78
11.47
18.26
50
44.78
11.48
18.27
CoupledSGA (ours)
10
38.96
9.51
15.29
20
40.48
10.10
16.16
50
40.49
10.10
16.17
Table 3. Matched-budget comparison (Action Genome, PredCLS, With Constraint). Both models use the conditions of Table 2 . These are small-subset results under a four-epoch budget and are not a full-benchmark comparison.
Model
LSC
R@20 (%) ↑
CR (%) ↓
A
SceneSayerODE
—
44.15 ± 0.51
54.40 ± 1.21
B
CoupledSGA
off
42.21 ± 0.75
45.92 ± 1.44
C
CoupledSGA
epochs 2–3
35.18 ± 1.10
4.07 ± 0.46
Table 4. Controlled comparison (Action Genome, PredCLS; conditions of Table 2 ). R@20 is computed under With Constraint and the contradiction rate CR (Equation 12 ) under No Constraint, on anticipated frames. Runs B and C differ only in whether LSC ( λSC=0.1 ) is active in epochs 2–3; A and B differ only in coupling. Mean ± standard deviation over three seeds.