Beyond Fixed Features: Architecture-Dependent Sensitivity to Node Representations under Heterophily

Priyanath Maji; Sidharth Gaur; Rajavinoth Paul Durai

cs.LGSep 27, 2026

Beyond Fixed Features: Architecture-Dependent Sensitivity to Node Representations under Heterophily

Authors: Priyanath Maji, Sidharth Gaur, Rajavinoth Paul Durai

Organizations: Georgia Institute of Technology

Abstract

Graph Neural Networks (GNNs) perform well on homophilic graphs but struggle in heterophilic settings, where connected nodes often carry dissimilar labels. Existing evaluations typically compare architectures under a fixed node-feature representation, leaving unclear whether conclusions about heterophily robustness remain stable as the input representation changes. We address this question by constructing parallel feature variants of two large-scale heterophilic benchmarks, Roman-Empire and Amazon-Ratings, pairing each graph with representations ranging from static fastText vectors to contextual Transformer embeddings and evaluating seven GNN architectures across these representations. We find that the effect of representation varies across architectures: on Roman-Empire, the contextual gain ranges from 2.38 percentage points for GCN-sep to 13.67 points for GAT, with H2GCN gaining 8.77 points. On Amazon-Ratings, where node text is limited to short product titles, GAT improves by 6.78 points from fastText to MPNet, while GCN-sep changes by only 0.20 points. These results show that architectural performance is conditional on node representation: the same representation change can produce different magnitudes of performance gain across architectures, so architecture and representation cannot be treated as independent evaluation factors. A rank-correlation analysis on these two benchmarks further shows that the relative ordering of architectures remains highly stable across representations, isolating differential sensitivity, rather than ranking instability, as the primary effect.

Figures & tables

	Heterophilic Datasets					Homophilic
	Roman	Texas	Corn.	Wisc.	Amazon	PubMed	Cora
Nodes	22,600	183	183	251	24,492	19,717	2,708
Edges	32,684	279	277	450	93,050	44,324	5,278
Avg. Deg.	2.89	3.05	3.03	3.59	7.60	4.50	3.90
Features	256-768	1,703	1,703	1,703	384-768	500	1,433
Classes	18	5	5	5	5	3	7

Table 1: Structural and homophily statistics of the evaluated graph benchmarks.

Model	Custom	fastText	BERT	RoBERTa	Best Contextual Gain
GCN	$52.16\pm 1.00$	$65.07\pm 0.80$	$64.34\pm 0.88$	$68.62\pm 1.96$	$+3.55$
GCN-sep	67.39 $\pm$ 0.91	$83.41\pm 0.66$	$85.79\pm 0.65$	$85.68\pm 0.48$	$+2.38$
GAT	$28.00\pm 1.22$	$38.96\pm 1.23$	$48.69\pm 2.67$	$52.63\pm 3.40$	+13.67
GAT-sep	$58.81\pm 1.00$	83.52 $\pm$ 1.22	87.76 $\pm$ 0.60	87.00 $\pm$ 0.57	$+4.24$
GraphSAGE	$58.42\pm 0.91$	$78.06\pm 1.02$	$84.88\pm 0.88$	$84.43\pm 0.64$	$+6.82$
H2GCN	$58.77\pm 0.66$	$75.33\pm 0.65$	$84.10\pm 0.78$	$83.99\pm 0.53$	$+8.77$

Table 2: Macro F1 (%) on the Roman-Empire benchmark across node-feature representations. Values are mean

\pm

standard deviation across 10 cross-validation folds. Bold indicates the highest Macro F1 within each feature space and the largest contextual gain among GNN architectures. Best contextual gain is computed as the best Macro F1 among BERT and RoBERTa minus the fastText result; gain values should be interpreted together with the corresponding absolute performance and variance.

Model	fastText	SBERT	MPNet	Best Contextual Gain
GCN	$44.17\pm 0.94$	$44.89\pm 0.82$	$45.25\pm 0.73$	$+1.08$
GCN-sep	$48.14\pm 1.17$	$48.24\pm 0.56$	$48.34\pm 0.60$	$+0.20$
GAT	$35.15\pm 8.47$	$40.86\pm 1.27$	$41.93\pm 0.39$	+6.78
GAT-sep	$45.75\pm 1.68$	$49.09\pm 1.05$	49.65 $\pm$ 1.25	$+3.90$
GraphSAGE	48.88 $\pm$ 0.81	50.34 $\pm$ 0.80	$49.58\pm 1.03$	$+1.46$
H2GCN	$45.69\pm 0.73$	$47.83\pm 0.82$	$47.50\pm 1.11$	$+2.14$

Table 3: Macro F1 (%) on the Amazon-Ratings benchmark across node-feature representations. Values are mean

\pm

standard deviation across 10 cross-validation folds. Bold indicates the highest Macro F1 within each feature space and the largest contextual gain. Best contextual gain is computed as the best Macro F1 among SBERT and MPNet minus the fastText result.

Model	Heterophilic			Homophilic
Model	Corn.	Texas	Wisc.	Cora	Pubm.
GCN	31.71 $\pm$ 10.99	28.46 $\pm$ 7.44	32.09 $\pm$ 9.33	85.90 $\pm$ 1.88	87.60 $\pm$ 0.71
GCN-sep	41.60 $\pm$ 8.79	50.11 $\pm$ 7.78	49.90 $\pm$ 8.90	86.08 $\pm$ 1.26	88.76 $\pm$ 0.40
GAT	31.77 $\pm$ 5.58	25.26 $\pm$ 5.99	34.06 $\pm$ 10.34	86.94 $\pm$ 1.39	87.20 $\pm$ 0.42
GAT-sep	53.04 $\pm$ 8.34	43.75 $\pm$ 8.54	51.91 $\pm$ 7.32	86.93 $\pm$ 1.44	88.91 $\pm$ 0.40
GraphSAGE	50.72 $\pm$ 6.20	52.56 $\pm$ 9.19	54.65 $\pm$ 7.16	87.35 $\pm$ 1.28	87.80 $\pm$ 0.46

Table 4: Macro F1 (%) on small-scale WebKB heterophily benchmarks and homophilic citation-network controls, using each dataset’s native node features.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Adj. Hom.	Naive GAT	GAT-sep	$\Delta$
Dataset	Score	Value	Value	( $p$ -value)
Roman-Empire (BERT)	$-0.055$	48.69	87.76	$+39.07$ ( $p<0.001$ )
Amazon-Ratings (fastText)	$-$ 0.141	35.15	45.75	$+10.60$ ( $p=0.008$ )
PubMed	$-$ 0.686	87.20	88.91	$+1.71$ ( $p<0.001$ )
Cora	$-$ 0.771	86.94	86.93	$-0.01$ ( $p=0.947$ )

Appendix

Table 5: Supplementary paired comparisons of GAT-sep and naive GAT across selected benchmarks.

\Delta

denotes the Macro F1 point difference. The reported

p

-values assess consistency of the paired differences across the corresponding cross-validation folds.

Dataset	Comparison	Spearman $\rho$	Kendall $\tau$	Inversions	Supported
Roman	Custom vs. fastText	0.9286 ( $p=.0025$ )	0.8095 ( $p=.0107$ )	2	0
	Custom vs. BERT	0.9286 ( $p=.0025$ )	0.8095 ( $p=.0107$ )	2	0
	Custom vs. RoBERTa	0.9286 ( $p=.0025$ )	0.8095 ( $p=.0107$ )	2	0
	fastText vs. BERT	1.0000 ( $p<.001$ )	1.0000 ( $p<.001$ )	0	0
	fastText vs. RoBERTa	1.0000 ( $p<.001$ )	1.0000 ( $p<.001$ )	0	0
	BERT vs. RoBERTa	1.0000 ( $p<.001$ )	1.0000 ( $p<.001$ )	0	0

Appendix

Table 6: Systematic evaluation of GNN architectural ranking stability across alternative node feature spaces. Spearman (

\rho

) and Kendall (

\tau

) coefficients quantify agreement between architecture rankings; their associated

p

-values explicitly test the null hypothesis of departure from zero association, rather than evaluating equivalence or stability itself. “Inversions” denotes the number of pairwise architecture-order reversals between the two representations. “Supported” denotes the number of nominal reversals satisfying both the 95% bootstrap confidence-interval criterion and the Holm-adjusted exact sign-flip significance criterion. Multi-comparison validation uses a Holm-adjusted exact sign-flip test on the corresponding interaction contrast together with a 95% bootstrap confidence interval to assess whether nominal rank reversals are statistically supported.

Feature	Architecture	Control Type	Layers	Hidden Dim.	Macro F1 (%)
BERT	GAT-naive	Original	1	64	48.69
BERT	GAT-naive	Depth/width matched	5	128	36.29
BERT	GAT-sep	Reference	5	128	87.76
RoBERTa	GAT-naive	Original	1	64	52.63
RoBERTa	GAT-naive	5L/128D control	5	128	39.13
RoBERTa	GAT-sep	Reference	3	64	87.00

Appendix

Table 7: Capacity and structural control experiments on Roman-Empire. The controls assess whether differences between naive and separation-based architectures can be explained primarily by depth, hidden dimension, or dominant hidden-layer parameter capacity.

Figure 1: Per-class performance and variability for the feature-only MLP baseline on the Roman-Empire. Performance varies substantially across syntactic categories, indicating that the node representation provides uneven task-relevant information across classes.

Figure 2: Per-class performance and variability for H2GCN on the Roman-Empire. Compared with the feature-only MLP, graph-based propagation changes performance across syntactic categories, with particularly large differences for selected classes.

Model	Metric	Custom	fastText	BERT	RoBERTa
Model	Metric	Value	Value	Value	Value
GCN	OA	59.55 $\pm$ 0.81	71.66 $\pm$ 0.64	70.93 $\pm$ 1.05	74.55 $\pm$ 1.93
GCN	BA	51.82 $\pm$ 0.92	65.27 $\pm$ 0.89	63.92 $\pm$ 1.08	68.20 $\pm$ 1.76
GCN-sep	OA	71.60 $\pm$ 0.78	86.26 $\pm$ 0.38	88.36 $\pm$ 0.37	88.35 $\pm$ 0.59
GCN-sep	BA	66.77 $\pm$ 1.18	83.17 $\pm$ 0.76	85.30 $\pm$ 0.91	85.52 $\pm$ 0.60
GAT	OA	35.70 $\pm$ 2.15	48.83 $\pm$ 1.10	54.86 $\pm$ 2.42	58.83 $\pm$ 3.08

Appendix

Table 8: Overall Accuracy (OA) and Balanced Accuracy (BA) (%) across evaluated feature spaces on the Roman-Empire benchmark.

Figure 3: Macro F1 scaling trends across restricted corpus, static lexical, and deep contextual node feature spaces on the Roman-Empire benchmark datasets.

Figure 4: Macro F1 scaling trends across static lexical, sentence, and base contextual node feature spaces on the Amazon-Ratings benchmark datasets.

Figure 5: Cross-validation performance footprints across WebKB and Planetoid benchmarks, illustrating mean Macro F1-score percentages and associated variance intervals across heterophilic and homophilic graph structures.

Figure 6: Per-class F1 scores across core network architecture transitions on the Roman-Empire benchmark datasets.

Figure 7: Detailed comparison of per-class F1 scores across feature spaces on the Roman-Empire benchmark, highlighting performance variations across different syntactic dependencies.

Model	Metric	fastText	SBERT	MPNet
Model	Metric	Value	Value	Value
GCN	OA	49.82 $\pm$ 0.86	50.55 $\pm$ 0.57	50.86 $\pm$ 0.65
GCN	BA	43.55 $\pm$ 0.88	44.27 $\pm$ 0.95	44.77 $\pm$ 0.64
GCN-sep	OA	53.53 $\pm$ 1.11	53.42 $\pm$ 0.42	53.84 $\pm$ 0.56
GCN-sep	BA	46.89 $\pm$ 0.81	47.70 $\pm$ 0.77	47.90 $\pm$ 0.78
GAT	OA	45.47 $\pm$ 4.85	49.50 $\pm$ 0.88	49.45 $\pm$ 0.63

Appendix

Table 9: Overall Accuracy (OA) and Balanced Accuracy (BA) (%) across evaluated feature spaces on the Amazon-Ratings benchmark.

Category	Dataset Space	Model Name	Layers ( $L$ )	Hidden Dim ( $d$ )	LR ( $\eta$ )	Dropout ( $p$ )
Roman-Empire	Custom	GCN	5	256	0.01	0.5
		GCN-sep	5	256	0.01	0.5
		GAT (8 heads)	1	64	0.01	0.2
		GAT-sep (8 heads)	3	64	0.005	0.2
		GraphSAGE	3	256	0.01	0.5
		H2GCN	2	256	0.005	0.2

Appendix

Table 10: Optimal hyperparameter configurations for large-scale Roman-Empire and Amazon-Ratings benchmarks across all feature spaces.

Category	Dataset Space	Model Name	Layers	Hidden Dim	Learning Rate	Dropout
Category	Dataset Space	Model Name	( $L$ )	( $d$ )	( $\eta$ )	( $p$ )
Heterophilic	Cornell	GCN	4	64	0.005	0.2
		GCN-sep	4	128	0.01	0.1
		GAT (8 heads)	1	64	0.01	0.2
		GAT-sep (8 heads)	3	64	0.01	0.2
		GraphSAGE	2	128	0.001	0.2

Appendix

Table 11: Optimal hyperparameter configurations for small-scale WebKB heterophilic benchmarks.

Category	Dataset Space	Model Name	Layers	Hidden Dim	Learning Rate	Dropout
Category	Dataset Space	Model Name	( $L$ )	( $d$ )	( $\eta$ )	( $p$ )
Homophilic	Cora	GCN	5	128	0.01	0.1
		GCN-sep	4	256	0.001	0.5
		GAT (8 heads)	2	128	0.01	0.2
		GAT-sep (8 heads)	3	128	0.005	0.1
		GraphSAGE	3	256	0.001	0.2

Appendix

Table 12: Optimal hyperparameter configurations for homophilic citation-network baseline controls.

Explore similar work

Revealing the Pitfalls and Re-Evaluating the Advancement of Heterophilic Graph Learning
Sep 9, 2024Sitao Luan, Qincheng Lu, Chenqing Hua +3Text-Attributed Graph Graph Neural Networks
Mila - Quebec Artiﬁcial Intelligence Institute · 1McGill University
Not All Nodes Are Created Equal: Homophily-Aware Stratification for Stable GNN Evaluation
Sep 16, 2026Naga Venkata Sai Jitin Jami, Thomas Altstidl, Sebastian Hoefler +4Graph Neural Networks Cross Validation
Chair for Machine Learning in Medicine, Universität Bayreuth · Department Artiﬁcial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg · Chair of AI-supported Therapy Decisions, Ludwig-Maximilians-Universität München +1
HeterSEED: Semantics-Structure Decoupling for Heterogeneous Graph Learning under Heterophily
May 6, 2026Xinyi Li, Ming Li, Lu Bai +5Text-Attributed Graph Heterogeneity
Zhejiang Normal University · Beijing Normal University · Central University of Finance and Economics +2

Sep 9, 2024, cs.LGJ/K move · Enter open · S save

Revealing the Pitfalls and Re-Evaluating the Advancement of Heterophilic Graph Learning

Sitao Luan, Qincheng Lu, Chenqing Hua, Xinyu Wang+2

Mila - Quebec Artiﬁcial Intelligence Institute · 1McGill University

Text-Attributed Graph Graph Neural Networks Heterogeneity Benchmark Datasets

Open paper AskPDF arXiv

Over the past decade, Graph Neural Networks (GNNs) have achieved great success on machine learning tasks with relational data. However, recent studies have found that heterophily can cause significant performance degradation of GNNs, especially on node-level tasks. Numerous heterophilic benchmark datasets have been put forward to validate the efficacy of heterophily-specific GNNs, and various homophily metrics have been designed to help recognize these challenging datasets. Nevertheless, there still exist multiple pitfalls that severely hinder the proper evaluation of new models and metrics: 1) lack of hyperparameter tuning; 2) insufficient evaluation on the truly challenging heterophilic datasets; 3) missing quantitative evaluation for homophily metrics on synthetic graphs. To overcome these challenges, we first train and fine-tune baseline models on

27

most widely used benchmark datasets, and categorize them into three distinct groups: malignant, benign and ambiguous heterophilic datasets. We identify malignant and ambiguous heterophily as the truly challenging subsets of tasks, and to our best knowledge, we are the first to propose such taxonomy. Then, we re-evaluate

11

state-of-the-arts (SOTA) GNNs, covering six popular methods, with fine-tuned hyperparameters on different groups of heterophilic datasets. Based on the model performance, we comprehensively reassess the effectiveness of different methods on heterophily. At last, we evaluate

11

popular homophily metrics on synthetic graphs with three different graph generation approaches. To overcome the unreliability of observation-based comparison and evaluation, we conduct the first quantitative evaluation and provide detailed analysis.

Abstract

Explore similar work

Revealing the Pitfalls and Re-Evaluating the Advancement of Heterophilic Graph Learning

Not All Nodes Are Created Equal: Homophily-Aware Stratification for Stable GNN Evaluation

HeterSEED: Semantics-Structure Decoupling for Heterogeneous Graph Learning under Heterophily