The rapid rise of large language models (LLMs) and their ability to capture semantic relationships has led to their adoption in a wide range of applications. Text-attributed graphs (TAGs) are a notable example where LLMs can be combined with Graph Neural Networks to improve the performance of node classification. In TAGs, each node is associated with textual content and such graphs are commonly seen in various domains such as social networks, citation graphs, recommendation systems, etc. Effectively learning from TAGs would enable better representations of both structural and textual representations of the graph and improve decision-making in relevant domains. We present GaLoRA, a parameter-efficient framework that integrates structural information into LLMs. GaLoRA demonstrates competitive performance on node classification tasks with TAGs, performing on par with state-of-the-art models with just 0.24% of the parameter count required by full LLM fine-tuning. We experiment with three real-world datasets to showcase GaLoRA's effectiveness in combining structural and semantical information on TAGs.
Figures & tables
Figure 1: GaLoRA Phase-1 training
Figure 2: GaLoRA Phase-2 training
Dataset
# Nodes
# Edges
# Tokens
Split (%)
# Classes
Metric
ArXiv
46,198
78,548
35,920,710
54/18/28
40
Accuracy
Instagram
11,339
144,010
579,263
80/10/10
2
ROC-AUC
Reddit
33,434
198,448
6,748,436
80/10/10
2
Accuracy
Table 1: Summary of datasets used in experiments.
ArXiv
Instagram
Reddit
Model
RoBERTa
GPT-2
RoBERTa
GPT-2
RoBERTa
GPT-2
GNN (PLM) †
0.7129 (0.0013)
0.7174 (0.0019)
0.6123 (0.0063)
0.6019 (0.0124)
0.6191 (0.0043)
0.6282 (0.0036)
GraphAdapter †
0.7273 ( 0.0021 )
0.7325 (0.0022)
0.6292 (0.0033)
0.6276 (0.0034)
0.6379 (0.0061)
0.6441 (0.0022)
GaLoRA (Ours)
0.7234 (0.0014)
0.7550 ( 0.0048 )
0.6392 ( 0.0093 )
0.6420 ( 0.0046 )
0.6464 ( 0.0035 )
0.6611 ( 0.0049 )
Table 2: LLM-controlled comparison (same LLM across methods). Instagram uses ROC–AUC; Reddit/ArXiv use Accuracy. For GaLoRA, results are mean ± std over 5 seeds (0–4). Baseline results (GNN, GraphAdapter) are taken directly from their reported values.
Model
PLM
# PLM Params
# Trainable Params
Relative to own PLM
GLEM
DeBERTa-Large
435M
435M
100%
GraphAdapter
LLaMA 2–13B
13B
2M
0.015%
GaLoRA (Ours)
GPT-2
124M
0.295M
0.238%
Table 3: Comparison of PLMs and trainable parameters. Relative % is computed w.r.t. each method’s own PLM parameter count. Phase 1: GNN module; Phase 2: LoRA-injected LLM layers.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Pass-1 embedding
Pass-2 embedding
GPT-2 (12 layers)
5, 6, 7
9, 10, 11
RoBERTa (12 layers)
5, 6, 7
9, 10, 11
Appendix
Table 4: LoRA layer selection for GPT-2 and RoBERTa backbones.
Model
Adapted Weights
GPT-2
c_attn, c_proj
RoBERTa
q, k, v, dense
Appendix
Table 5: LoRA configuration for GPT-2 and RoBERTa.
Model
Total Params
Trainable Params
Relative (%)
GPT-2
124.6M
115.2K
0.09%
RoBERTa
125.0M
175.1K
0.14%
Appendix
Table 6: Trainable parameter footprint for GPT-2 and RoBERTa with rank = 4.
Hyperparameter
Value
Loss function
Cross-Entropy
Optimizer
AdamW
Learning rate
3×10−4
Weight decay
10−2
Batch size
32
Implementation
Manual (PyTorch, HuggingFace)
Appendix
Table 7: Training configuration for LoRA adapters (default)
Rank ( r )
ROC-AUC
2
0.6347
4
0.6420
8
0.6421
Appendix
Table 8: Effect of LoRA rank on Instagram performance.
Prompt
ROC-AUC
No prompt (raw text)
0.6283
Classify:
0.6305
Classify this instagram account bio:
0.6344
Classify instagram account is commercial or not:
0.6420
Appendix
Table 9: Effect of prompt engineering on Instagram dataset performance.
Model
Instagram
Reddit
ArXiv
GraphSAGE + GPT-2
0.6420
0.6611
0.7550
GraphSAGE + RoBERTa
0.6392
0.6464
0.7234
GAT + GPT-2
0.6616
0.6613
0.7569
Appendix
Table 10: Performance comparison of GaLoRA model variants across datasets.
Chungbuk National University Cheongju, South Korea · Chungnam National University Daejeon, South Korea · The Pennsylvania State University University Park, PA, USA