Why Does Graph Learning Fail to Fully Benefit from a Text Teacher?

Fumiaki Kimino; Ryoma Sato

cs.LGAug 26, 2026

Why Does Graph Learning Fail to Fully Benefit from a Text Teacher?

Authors: Fumiaki Kimino, Ryoma Sato

Organizations: SOKENDAI · National Institute of Informatics · National Institute of Informatics, SOKENDAI

Abstract

Graph neural networks (GNNs) are widely used to represent complex interactions and relationships among entities. We investigate a multimodal model that combines two complementary ideas: a self-supervised method that enables a GNN encoder pretrained on one dataset to operate directly on another dataset with a different node-feature dimensionality, without rebuilding the model or realigning the data; and an alternating optimization method that updates a language-model module in an E-step and a GNN module in an M-step, rather than jointly training a large language model and a GNN end to end on a large graph. Despite expectations, the combined model did not sufficiently improve predictive performance. We identify six factors: (1) an external anchor in the E-step has a strength-safety trade-off: a weak anchor has little effect, whereas an overly strong anchor can damage the graph representation; (2) the knowledge of the E-step teacher is not injected directly into the GCN embedding Z; (3) the representation space constructed in the M-step is not optimized for the same objective as the E-step teacher space, resulting in a compromise representation for target classification; (4) GCN propagation averages a node's own textual information with information from its neighbors; (5) cosine alignment does not guarantee axes that are discriminative for classification, so stronger geometric alignment with the E-step text anchor need not sufficiently improve the target decision boundary or classification performance; and (6) the force that preserves the source-side self-supervised geometry in the M-step conflicts with the force that moves the representation toward the E-step teacher. We support these observations through a staged set of experiments that varies the influence of the E-step.

Figures & tables

Figure 1: Overview of FUG+GLEM-ITT training. An independent TextHead that does not imitate the GCN output is first pretrained for 60 epochs using raw text-hash features, source labels, and dropout-based self-supervision. FUG is then warmed up for 60 epochs with its self-supervised objective, followed by six EM-like iterations. In the E-like step of each iteration, TextHead is refreshed for two epochs to produce an updated text anchor. In the M-step, the FUG encoder is updated for 18 epochs using the FUG self-supervised loss and auxiliary losses for the TextHead, MLP-only, raw text-hash, and external text anchors: $\mathcal{L}_{M}=0.80\mathcal{L}_{\mathrm{FUG}}+1.80\mathcal{L}_{\mathrm{text}}+1.10\mathcal{L}_{\mathrm{MLP}}+0.70\mathcal{L}_{\mathrm{raw}}+0.25\mathcal{L}_{\mathrm{ext}}$ . The updated FUG parameters are carried forward to the next iteration.

Experiment	Method	Full GCN ( $Z$ )	Raw Text Hash	Raw MPNet	Ensemble
Baseline	FUG-only	0.7459	0.7623	–	0.7702
FUG+GLEM	FUG+GLEM-ITT	0.7480	0.7623	–	0.7717

Table 1: Cross-domain transfer performance between the FUG baseline and FUG+GLEM-ITT.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Method	Full GCN ( $Z$ )	Raw Text Hash	Raw MPNet	Ensemble
Exp1	FUG-only	0.7459	0.7623	–	0.7702
Exp2	GCN pseudo-( $Z$ ) imitation TextHead	0.7445	0.7623	–	0.7695
Exp3	Raw text-hash external anchor	0.7437	0.7623	–	0.7717
Exp4	Frozen MPNet external teacher	0.7437	0.7623	0.7888	0.7724 / 0.7845 a

Appendix

Table 2: Comparison of cross-domain transfer performance across experimental settings.

Representation or difference	Exp2	Exp3	Exp4	FUG+GLEM-ITT
Test accuracy of each representation
Raw Text Hash	$0.7623$	$0.7623$	$0.7623$	$0.7623$
MLP-only (before GCN propagation)	$0.7380$	$0.7380$	$0.7373$	$0.7359$
Full GCN $Z$ (after GCN propagation)	$0.7445$	$0.7437$	$0.7437$	$0.7480$
Raw MPNet	–	–	$0.7888$	–
Ensemble (GCN $Z$ + Text Hash)	$0.7695$	$0.7717$	$0.7724$	$0.7717$

Appendix

Table 3: Test accuracy at different representation stages under the OpenAlex linear probe.

Method	Raw Text Hash	MLP-only	Full GCN ( $Z$ )	Ensemble
FUG-only	$0.7623$	$0.7480$	$0.7459$	$0.7702$
FUG+GLEM-ITT	$0.7623$	$0.7359$	$0.7480$	$0.7717$

Appendix

Table 4: Representation-level classification performance of FUG-only and FUG+GLEM-ITT on OpenAlex.

Setting	Metric	Direction	Initial	Final	Change	Full GCN $Z$ test accuracy	Difference from FUG-only
(a) Cosine similarity: larger values indicate stronger anchor alignment
Exp3	ood_cos	$\uparrow$	$0.0003$	$0.6918$	$+0.6915$	$0.7437$	$-0.0022$
Exp3	raw_cos	$\uparrow$	$-0.0084$	$0.2693$	$+0.2777$	–	–
Exp4	mpnet_cos	$\uparrow$	$0.0095$	$0.4341$	$+0.4246$	$0.7437$	$-0.0022$
(b) Cosine anchor loss: smaller values indicate stronger anchor alignment
FUG+GLEM-ITT	text_anc	$\downarrow$	$0.8032$	$0.3484$	$-0.4548$	$0.7480$	$+0.0021$

Appendix

Table 5: Changes in alignment metrics and Full GCN-

Z

classification performance on OpenAlex.

Candidate	valid_bacc	ood_cos	text_cos	Composite	Difference from Warmup ( valid_bacc )	Selected
Warmup	$0.4788$	$0.0003$	$-0.0166$	$0.1132$	$0.0000$	–
em1	$0.4707$	$0.2826$	$0.4404$	$0.3927$	$-0.0081$	–
em2	$0.4715$	$0.3967$	$0.5576$	$0.4797$	$-0.0073$	–
em3	$0.4765$	$0.4554$	$0.6097$	$0.5224$	$-0.0023$	–
em4	$0.4723$	$0.4742$	$0.6328$	$0.5372$	$-0.0065$	–
em5	$0.4746$	$0.4807$	$0.6444$	$0.5447$	$-0.0042$	–

Appendix

Table 6: Candidate-model metrics and the FUG+GLEM-ITT model-selection result.

Experiment	Candidate checkpoints	Selection score	Notes
FUG+GLEM-ITT	Warmup, EM1–EM6, SWA	$S=0.25Bal_{\mathrm{val}}+0.35Cos_{\mathrm{MLP}}+0.40Cos_{\mathrm{TextHead}}$	Warmup, the end of each EM iteration, and the SWA model obtained by averaging model states after the EM iterations are candidates. The final checkpoint maximizes a composite of source-domain validation balanced accuracy and the mean cosine similarities, computed over all source-graph nodes, between Full GCN $Z$ and the MLP-only and TextHead representations.
Exp1	After epochs $10,20,\ldots,160,168$	$S=Bal_{\mathrm{val}}$	Selects the FUG-only checkpoint with the highest source validation balanced accuracy.
Exp2	Warmup, EM1–EM6	$S=Bal_{\mathrm{val}}$	The cosine similarity between GCN $Z$ and the TextHead output is recorded but not used for selection.
Exp3	Warmup, EM1–EM6	$S_{\mathrm{impl}}=0.35Bal_{\mathrm{val}}+0.30Cos_{\mathrm{MLP}}$	Cosine similarity to the raw text-hash anchor is recorded, but it is not passed to the shared selection function as text_cos and is therefore absent from the selection score.
Exp4	Warmup, EM1–EM6	$S=0.35Bal_{\mathrm{val}}+0.30Cos_{\mathrm{MLP}}+0.35Cos_{\mathrm{MPNet}}$	The cosine similarity between GCN $Z$ and the frozen MPNet anchor is passed as text_cos and included in the selection score.

Appendix

Table 7: Checkpoint-selection procedures for FUG+GLEM-ITT and Exp1–Exp4.

Graph Representation Learning

Graph representation learning develops methods for encoding nodes, edges, and whole graphs for prediction, comparison, and transfer.

See all 305 papers

Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching
Jun 10, 2026Zhuoyi Peng, Hanlin Gu, Lixin Fan +1Graph Representation Learning Text-Attributed Graph Learning
1The Hong Kong University of Science and Technology · 2WeBank
ModTGCN: Modularity-aware Graph Neural Networks for Text Classification
Apr 29, 2026Rajarshi Misra, Aditya Sharma, Vinti Agarwal +1Graph Neural Networks Text Classification
BITS Pilani, India · Independent Researcher
Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings
May 19, 2025Adrien Lagesse, Marc LelargeGraph Neural Networks Synthetic Benchmark Generation
INRIA, École Normale Supérieure - PSL, Paris, France.

Jun 10, 2026J/K move · Enter open · S save

Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching

Zhuoyi Peng, Hanlin Gu, Lixin Fan, Yi Yang

1The Hong Kong University of Science and Technology · 2WeBank

Graph Representation Learning Text-Attributed Graph Learning Pseudo-Labeling LLMs for Graph Learning

Open paper AskPDF arXiv

Text-attributed graphs (TAGs) underlie real-world applications such as citation networks, social media, and e-commerce. Few-shot graph learning on TAGs is hard: with only a handful of labels per class and the rest of the graph unannotated, neither GNNs nor LLMs can learn well on their own. GNNs read topology and fail on cold nodes; LLMs read text and fail on text-ambiguous nodes. Existing LLM-GNN methods all follow the same recipe: designate one model as the golden teacher and use its outputs (e.g., features or pseudo-labels) to supervise the other. We argue this golden-teacher assumption breaks under sparse supervision: neither model is golden, and treating either as such transfers its blind spots into the student. We therefore ask: can we avoid designating either model as the golden teacher, and still perform effective graph learning? We answer with LLM-GNN Co-Teaching, a bidirectional co-teaching framework in which neither model is fixed as teacher. The GNN and LLM exchange their most confident pseudo-labels under an architecture-specific small-loss criterion, and both update every round. Supervision is then mined from the trajectory: whenever a node moves from cross-model contradiction at round t to cross-model agreement at round t+1, the LLM's two answers on the same input form a preference pair (old contradicting self < new peer-endorsed self) for DPO training. We call this Round-based Pseudo-Label Preference Optimization (RPL-PO). On six benchmarks, LLM-GNN Co-Teaching consistently outperforms GNN-as-Judge and all prior methods, with absolute 3-shot gains of 7.86% on Cora and 7.73% on ogbn-arxiv; improvements carry over to 5-shot and to zero-shot cross-dataset transfer. Error-structure analysis further shows that abandoning the golden-teacher assumption substantially improves the LLM's graph learning capability on challenging samples.

Abstract

Graph Representation Learning

Explore similar work

Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching

ModTGCN: Modularity-aware Graph Neural Networks for Text Classification

Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings