Existing methods for graph out-of-distribution (OOD) generalization primarily rely on empirical studies on synthetic datasets. Such approaches tend to overemphasize the causal relationships between invariant sub-graphs and labels, thereby neglecting the non-negligible role of environment in real-world scenarios. In contrast to previous studies that impose rigid independence assumptions on environments and invariant sub-graphs, this paper presents the theorems of environment-label dependency and mutable rationale invariance, where the former characterizes the usefulness of environments in determining graph labels while the latter refers to the mutable importance of graph rationales. Based on analytic investigations, a novel variational inference based method named ``Probability Dependency on Environments and Rationales for OOD Graphs on Real-world Data'' (DEROG) is introduced. To alleviate the adverse effect of unknown prior knowledge on environments and rationales, DEROG utilizes generalized Bayesian inference. Further, DEROG employs an EM-based algorithm for optimization. Finally, extensive experiments on real-world datasets under different distribution shifts are conducted to show the superiority of DEROG. Our code is publicly available at https://github.com/LEOXC1571/DEROG.
Figures & tables
Figure 1. The process of drugs to take effect through binding with receptors. This figure demonstrates that there are a number of factors other than pharmacophore that determine validity and properties of drugs in real-world scenarios. Diagram of barbiturate drugs progressing through absorption and distribution, receptor binding, metabolism, excretion, and the resulting drug validity.
Dataset
HIV
LBAP-core-IC50
Twitter
Domain
Scaffold
Size
Assay
Length
Shift
Covariate
Concept
Covariate
Concept
Min.
0.7068
0.8392
0.9042
0.9031
Max.
0.7068
0.8874
0.9768
0.9957
Mean
0.7068
0.8443
0.9434
0.9822
Table 1. Statistics of Edge Mask Weights in LECI
Dataset
HIV
Twitter
Domain
Scaffold
Size
Length
Shift
Covariate
Concept
Covariate
loss(cout,y)
0.1035
0.2218
2.5274
loss(sout,y)
0.1035
0.2220
2.4446
Table 2. Preliminary studies on cout and sout from CIGA
Figure 2. The overall framework of the proposed method DEROG. Framework diagram of DEROG, showing its probabilistic model, latent environment and rationale inference, classifier optimization, and contrastive learning components.
Task
Molecule Learning
Sentiment Analysis
Dataset
GOODHIV
DrugOOD-LBAP-core-IC50
GOODTwitter
GOODSST2
Domain
Scaffold
Size
Scaffold
Size
Assay
Length
Length
Shift
Covariate
Concept
Covariate
Concept
Covariate
Covariate
Covariate
Covariate
Concept
Covariate
Concept
ERM
72.01(1.02)
67.23(2.41)
63.05(2.80)
61.88(4.27)
70.03(0.59)
67.02(0.21)
69.93(0.40)
57.77(0.75)
45.44(1.85)
78.94(0.55)
71.35(0.56)
IRM
71.05(1.25)
64.05(2.30)
67.94(1.07)
70.92(7.60)
67.78(0.51)
65.37(0.51)
71.33(0.73)
55.89(1.41)
49.68(3.13)
80.21(1.13)
72.17(1.47)
VREx
67.78(1.80)
65.68(3.42)
64.29(1.98)
57.85(1.45)
68.52(0.46)
65.09(1.50)
71.06(0.93)
57.01(0.08)
49.93(3.13)
78.62(1.83)
71.59(0.20)
Table 3. Performance comparisons in real-world scenarios. The evaluation metric employed for the two molecule datasets is ROC-AUC, whereas classification accuracy is adopted for NLP datasets. The best result is emphasized in bold while the second-best one is underscored.
Dataset
HIV
Twitter
SST2
Domain
Scaffold
Size
Length
Length
Shift
Covariate
Concept
Covariate
Concept
ERM
72.01(1.02)
61.88(4.27)
57.77(0.75)
71.35(0.56)
LECI
72.50(0.85)
55.91(2.76)
59.64(0.60)
72.97(1.62)
DEROG-v1
75.51(0.66)
76.33(1.99)
60.15(0.58)
73.58(0.56)
DEROG-v2
76.87(0.80)
80.65(8.35)
60.54(0.43)
75.48(1.17)
Table 4. Ablation study on DEROG
Figure 3. Efficiency study Line plots used for the efficiency study.
Figure 4. Performance of DEROG under various hyperparameter settings. Two plots showing DEROG performance under different hyperparameter settings.
Out-of-distribution (OoD) generalization occurs when representation learning encounters a distribution shift. This occurs frequently in practice when training and testing data come from different environments. Covariate shift is a type of distribution shift that occurs only in the input data, while the concept distribution stays invariant. We propose RIA - Regularization for Invariance with Adversarial training, a new method for OoD generalization under convariate shift. Motivated by an analogy to Q-learning, it performs an adversarial exploration for counterfactual data environments. These new environments are induced by adversarial label invariant data augmentations that prevent a collapse to an in-distribution trained learner. It works with many existing OoD generalization methods for covariate shift that can be formulated as constrained optimization problems. We develop an alternating gradient descent-ascent algorithm to solve the problem in the context of causally generated graph data, and perform extensive experiments on OoD graph classification for various kinds of synthetic and natural distribution shifts. We demonstrate that our method can achieve high accuracy compared with OoD baselines.
Simon Zhang, Ryan P. DeMilt, Kun Jin +1
Department of Computer Science, Purdue University, West Lafayette, USA · Department of Computer Science and Engineering, The Ohio State University, Columbus Ohio, USA · Department of Industrial and Systems Engineering, The Ohio State University, Columbus Ohio, USA
Graph-level out-of-distribution (OOD) generalization is critical for deploying graph neural networks in real-world applications where distribution shifts are inevitable. Existing approaches typically require class labels during representation learning to decide which substructures are predictive, limiting their applicability. We propose SCARCE, a label-free representation-learning framework that yields distribution-shift-robust graph representations from an environment partition of the training graphs alone, whether that partition comes from dataset metadata or, when none is available, from clustering a self-supervised embedding. Our method operates in two stages: (1) a variational graph autoencoder is warm-started for graph reconstruction; and (2) an adversarial confusion mechanism forces the encoder to produce representations from which a discriminator cannot distinguish environments, while preserving graph-reconstruction fidelity. We provide theoretical analysis showing that environment confusion contracts environment-discriminative directions in the representation space, which, under regularized downstream classifiers, is equivalent to imposing a stronger directional penalty on environment-associated features. This feature-reweighting mechanism reduces reliance on spurious correlations without erasing information, offering a label-free, continuous relaxation of last-layer reweighting methods. Experiments on a controlled synthetic datasets and the DrugOOD datasets show that SCARCE achieves state-of-the-art OOD generalization among label-free methods, using no class labels during representation learning.
Tianjun Yao, Chen Xi, Xinyang Li +1
Shenzhen Loop Area Institute · Guangzhou University · North China Electric Power University +1
Although dynamic graph neural networks (DyGNNs) have demonstrated promising capabilities, most existing methods ignore out-of-distribution (OOD) shifts that commonly exist in dynamic graphs. Dynamic graph OOD generalization is non-trivial due to the following challenges: 1) Identifying invariant and variant patterns amid complex graph evolution, 2) Capturing the intrinsic evolution rationale from these patterns, and 3) Ensuring model generalization across diverse OOD shifts despite limited data distribution observations. Although several attempts have been made to tackle these challenges, none has successfully addressed all three simultaneously, and they face various limitations in complex OOD scenarios. To solve these issues, we propose a Dynamic graph Causal Invariant Learning (DyCIL) model for OOD generalization via exploiting invariant spatio-temporal patterns from a causal view. Specifically, we first develop a dynamic causal subgraph generator to identify causal dynamic subgraphs explicitly. Next, we design a causal-aware spatio-temporal attention module to extract the intrinsic evolution rationale behind invariant patterns. Finally, we further introduce an adaptive environment generator to capture the underlying dynamics of distributional shifts. Extensive experiments on both real-world and synthetic dynamic graph datasets demonstrate the superiority of our model over state-of-the-art baselines in handling OOD shifts.
Xinxun Zhang, Pengfei Jiao, Mengzhou Gao +2
School of Cyberspace, Hangzhou Dianzi University, China · State Key Laboratory of AI Safety, Beijing · School of Computer Science, Wuhan University, China +1