Organizations: Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany · University of Zurich, Zurich, Zurich, Switzerland · University of Michigan, Ann Arbor, Michigan, USA
Link prediction on text-attributed graphs (TAGs) is a fundamental problem with applications in knowledge discovery, recommendation and beyond. Recent progress has highlighted the promise of combining Graph Neural Networks (GNNs) with Language Models (LMs) in order to leverage the complementary structural and semantic signals. However, due to the distinct inductive biases of GNNs and LMs, the design of an optimal fusion strategy remains an open research question. In this work, we present Bridge-LP, an intuitive, simple, efficient and theoretically grounded framework that unifies graph and language representations. It is built around three theoretically-justified design components: (D1) jumping connection of textual proximity, which enables robust LM training; (D2) compressed embedding, which encodes neighborhood information long exploited by classical link prediction heuristics, thereby enriching structural signals in a learnable way; and (D3) cache embedding training strategy, which reduces computational complexity. Together, these mechanisms establish an end-to-end cross-modal framework that scales to large graphs while capturing richer graph-text interactions. We conduct an extensive empirical evaluation, comparing against 40 existing models across 9 diverse datasets. Experiments show that Bridge-LP consistently outperforms state-of-the-art approaches, delivering up to a 43.06% improvement in Mean Reciprocal Rank (MRR) and setting a new benchmark for language-graph integration.
Figures & tables
Figure 1. Bridge-LP maintains accuracy (x-axis) while improving efficiency (y-axis) by 5% over our most competitive baseline on GNN4LP and PLM-based baselines. § Sec. 4
Figure 2. Overview of Bridge-LP . The framework consists of three graph signals: S1 Structural proximity from the graph structure (common neighbors, Eq. 1 ) ; S2 Semantic proximity from PLM text embeddings ; S3 Propagated proximity from GCNs; and the final step concatenates S1-3 through an MLP to perform link prediction.
Works
ZS
FT
MLP
GCN
MLP
GCN
Graphformer ( Yang et al., 2021 )
✓
GAINT ( Chien et al., 2022 )
✓
✓
SimTeG ( Duan et al., 2023 )
✓
✓
TAPE ( He et al., 2024 )
✓
LEADING ( Xue et al., 2023 )
✓
Table 1. Comparison with existing methods. ✓ : public benchmark, ✓ : benchmarked model, Zero-Shot: ZS, Fine-Tune: FT
Figure 3. Ablation study of D1. Comparison of link prediction performance (AUC) with and without the proposed bypass connection. Across three benchmark datasets, the absence of D1 results in a performance degradation of 4%–6.8%, validating its role in facilitating effective gradient flow and preserving PLM semantic nuances.
Figure 4. Left: Overlapping rate between HC and ST ; Right: AUC gap between models with and without compression. The results show that HC and ST are approximately compressive and that the framework equipped with D2 converges to a similar performance level as the non-compressed variant.
Figure 5. (a) Sensitivity of Bridge-LP to the common-neighbor degree cutoff τ used in the TCN threshold, on Hits@50, MRR, and AUC. (b) Sensitivity of Bridge-LP to the structural-proximity mixing weight β used in D2, on Hits@50, MRR, and AUC. Both sweeps are on Cora, averaged over five seeds; shaded bands show standard deviation.
Figure 6. (a) The effect of graph encoder depth (number of GNN layers) on test AUC for Cora and PubMed datasets. The shaded areas represent the standard deviation, indicating the variance in performance across different trials. (b) The effect of LLM backbone choice on AUC for the Cora dataset. The figure shows three series: E5, BERT along with Sentence-Transformers.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Models
Cora
PubMed
Arxiv 2023
Hits@100
AUC
Hits@100
AUC
Hits@100
AUC
RotatE
56.60 ± 6.29
61.57 ± 3.88
38.58 ± 2.59
80.89 ± 0.40
34.57 ± 0.84
76.73 ± 0.44
TransE
57.00 ± 7.06
64.19 ± 3.83
50.04 ± 2.63
85.29 ± 0.97
30.22 ± 1.96
73.03 ± 1.04
ComplEx
73.59 ± 0.90
75.71 ± 1.09
53.87 ± 3.09
89.60 ± 0.51
32.17 ± 2.38
73.31 ± 0.61
DistMult
73.12 ± 0.28
76.53 ± 0.80
54.82 ± 2.95
89.91 ± 0.53
32.37 ± 1.62
73.25 ± 0.39
BERT-Inf
56.90 ± 3.26
65.09 ± 1.41
48.73 ± 1.43
88.96 ± 0.31
48.74 ± 1.15
86.37 ± 0.27
Appendix
Table 9. Benchmark result of local and global heuristic, graph embedding, vanilla GCNs, GCN2LPs, LLM-Inf, LLM-FT.
Models
Cora
PubMed
Arxiv 2023
Hits@50
MRR
Hits@50
MRR
Hits@50
MRR
MiniLM-GAT-CT
54.23 ± 4.08
13.74 ± 5.21
29.44 ± 2.84
4.26 ± 1.75
13.76 ± 2.22
2.62 ± 0.55
MiniLM-SAGE-CT
63.04 ± 9.23
15.09 ± 1.35
45.31 ± 3.83
6.70 ± 3.61
49.11 ± 4.22
11.48 ± 3.41
mpnet-GIN-CT
89.01 ± 5.54
29.06 ± 7.96
46.82 ± 4.22
11.86 ± 3.68
55.11 ± 3.70
18.88 ± 5.89
mpnet-GAT-CT
74.90 ± 9.22
23.16 ± 11.10
35.82 ± 3.81
5.36 ± 1.69
19.50 ± 1.91
4.49 ± 0.91
mpnet-SAGE-CT
82.01 ± 4.19
25.34 ± 8.06
57.58 ± 3.78
11.91 ± 3.58
52.97 ± 5.05
14.51 ± 3.37
Appendix
Table 10. Benchmark result of local and global heuristic, graph embedding, vanilla GCNs, GCN2LPs, ZO, FT.
Text-attributed graphs (TAGs), where each node carries a natural language description, require models to jointly reason over text and graph topology. Existing approaches often handle the two modalities separately: graph neural networks operate on shallow text features, while hybrids of LLMs and graphs use the language model mainly as a text encoder and delegate structure learning to a separate graph module. We propose method that unifies textual reasoning and graph message passing within a masked diffusion language model, a language model with bidirectional attention and generative decoding. For each graph instance, method linearises a sampled local neighbourhood into a token sequence and injects graph structure through a topology attention mask, which realises message passing over the graph. Because the diffusion language model can both interpret and generate text, the method adapts to different tasks simply by changing the prompt, supporting node classification, link prediction, and cross-dataset transfer with no target-specific fine-tuning. Experiments show that method outperforms graph neural networks, graph transformers, and LLM-based baselines on all three TAG benchmarks across two tasks, improving over the strongest baseline by up to 3.9 points.
Text-attributed graphs (TAGs) are widely used in many real-world domains, and learning on TAGs requires jointly modeling text semantics and graph structure. A standard approach for modeling TAGs is to combine a language model (LM) and a graph neural network (GNN), but joint training is computationally expensive and difficult to scale. Dataset distillation is a promising way to reduce training costs, but existing methods are not well suited to TAGs because they are typically designed for a single modality or still require repeatedly training expensive LM-GNN models on the full dataset during distillation. To address this, we propose TaLK, an effective dataset distillation method for TAGs that couples an LM with a graph-aware neural tangent kernel. This design enables efficient dataset distillation, avoiding repeated joint training on the full dataset while reflecting both textual and structural information for effective TAG learning. Experiments on multiple TAG benchmarks show that TaLK consistently outperforms existing baselines and achieves up to 97% of full-dataset performance with only 1% synthetic data.
Text-Attributed Graphs (TAGs) combine textual semantics with graph structure and are central to many graph learning tasks. However, existing fusion methods often treat text and structure as separate inputs in a shallow, one-way pipeline, which limits deep interaction between modalities and weakens performance under sparse connectivity or cross-graph generalisation. To address this issue, we propose PromptGNN-sim, a bi-directional structure-semantic fusion framework for collaborative GNN-LLM learning. PromptGNN-sim uses a Graph Attention Network (GAT) for semantically aware neighborhood selection by combining structural attention with textual similarity. The selected structural context is then used to generate structure-aware prompts for an LLM, including the target node summary, label categories, and representative keywords from similar neighbors. During training, bi-directional cross-modal contrastive learning and cross-attention are introduced to jointly optimize the GNN and LLM components. Experiments on six public datasets, including Cora, Pubmed, and WikiCS, evaluate accuracy, generalisation, and robustness under cross-task transfer, cross-dataset generalisation, and sparse perturbations. Results show that PromptGNN-sim outperforms classical GNNs, LLMs, and recent GNN-LLM fusion methods, demonstrating the effectiveness of interactive structure-semantic collaboration for text-attributed graph learning.