LLM-Guided Knowledge Distillation for Temporal Knowledge Graph Reasoning
Authors: Wang Xing, Wei Song, Siyu Lin, Chen Wu, Man Wang
Organizations: School of Computer Science and Technology, Xidian University · School of Computing and Artificial Intelligence, Southwest Jiaotong University · School of Information Science and Engineering, Chongqing Jiaotong University
Temporal knowledge graphs (TKGs) support reasoning over time-evolving facts, yet state-of-the-art models are often computationally heavy and costly to deploy. Existing compression and distillation techniques are largely designed for static graphs; directly applying them to temporal settings may overlook time-dependent interactions and lead to performance degradation. We propose an LLM-assisted distillation framework specifically designed for temporal knowledge graph reasoning. Beyond a conventional high-capacity temporal teacher, we incorporate a large language model as an auxiliary instructor to provide enriched supervision. The LLM supplies broad background knowledge and temporally informed signals, enabling a lightweight student to better model event dynamics without increasing inference-time complexity. Training is conducted by jointly optimizing supervised and distillation objectives, using a staged alignment strategy to progressively integrate guidance from both teachers. Extensive experiments on multiple public TKG benchmarks with diverse backbone architectures demonstrate that the proposed approach consistently improves link prediction performance over strong distillation baselines, while maintaining a compact and efficient student model. The results highlight the potential of large language models as effective teachers for transferring temporal reasoning capability to resource-efficient TKG systems.
Figures & tables
Figure 1: Comparison between different distillation methods.
Figure 2: The training and prediction on quadruple knowledge with the large language model.
Dataset
Entities
Relations
Train
Valid
Test
YAGO
10,623
10
161,540
19,523
20,026
WIKI
12,544
24
539,286
67,538
63,110
Table 1: Statistical data of two datasets
Model
Encoder
Decoder
TTransE
–
∥s+p−o+t∥2
TADistMult
pseq=LSTM(p:t)
(s∘o)pseq⊤
Table 2: Introduction of classic TKG models
YAGO
WIKI
Model
Method
MRR
MR
Hits@1
Hits@3
Hits@10
MRR
MR
Hits@1
Hits@3
Hits@10
TTransE
BKD
7.65
1410.12
3.50
7.83
15.61
7.94
2383.67
4.75
8.22
14.04
FitNet
7.59
1201.69
3.06
7.18
16.48
7.86
2148.86
3.93
7.78
14.67
RKD
7.01
1186.27
3.56
6.95
13.47
7.89
2052.37
4.72
7.49
12.85
Ours
7.69
1193.15
3.61
7.89
16.57
7.92
1985.63
4.86
8.36
14.94
TADistMult
BKD
61.90
973.89
58.51
64.13
67.59
45.89
3150.11
42.46
48.87
51.18
Table 3: Distillation results of different model method combinations on two datasets
Temporal knowledge graphs (TKGs) represent time-stamped relational facts and support a wide range of reasoning tasks over evolving events. However, existing methods produce entity representations that are static at the entity level, in that each representation is a function of learned parameters only and retains no trace of the interactions in which the entity has participated. In this paper, we depart from this static view and propose that each entity be modeled as an adaptive process whose representation is refined every time the entity participates in a fact. To this end, we propose AdaTKG, which maintains a per-entity memory that is updated with every observed interaction, with the memory accumulating online and predictions improving as more interactions arrive. Specifically, we instantiate the memory update as a learnable exponential moving average governed by a single shared scalar instead of using learnable parameters for each entity, enabling AdaTKG to handle entities unseen during training. Extensive experiments confirm consistent gains over TKG baselines, demonstrating the effectiveness of adaptive memory. Code is available at: https://github.com/seunghan96/AdaTKG
Temporal Knowledge Graph (TKG) reasoning under the extrapolation setting focuses on forecasting future time-stamped events (facts) from historical data in a temporal knowledge graph. Existing approaches, reinforcement learning (RL)-based multi-hop reasoning methods are prominent for TKG reasoning because they produce human-interpretable predictions via explicit multi-hop path tracing. However, during RL training, rewards are typically sparse, and exploration is highly inefficient due to the vast, time-evolving action space. These issues hinder efficient training and often limit overall performance. To address these challenges, we propose RAPTOR (Reachability-Aware Pretraining for Efficient Target-Oriented Path Exploration), a self-supervised pretraining method that injects a reachability-aware inductive bias to the agent. By learning to estimate the reachability of candidate actions to the target entity, RAPTOR reduces exploration over unpromising paths and provides a strong initialization for downstream RL fine-tuning. Experimental results on the ICEWS14, ICEWS05-15, and ICEWS18 datasets demonstrate that RAPTOR pretraining markedly improves the training efficiency and consistently outperforms conventional baselines, establishing it as an effective approach for enhancing RL-based multi-hop reasoning methods for TKG reasoning.
Chien-Liang Liu, Tsao-Lun Chen
Artificial Intelligence Research Center, Chang Gung University · 2Graduate Institute of Oral Biology, National Taiwan University · 3Electrical Engineering, National Taiwan University of Science and Technology
Temporal knowledge graph (TKG) reasoning aims to infer future facts at unseen timestamps from temporally evolving entities and relations. Despite recent progress, existing approaches still suffer from inherent limitations due to their inductive biases, as they predominantly rely on time-invariant or weakly time-dependent structures and overlook the evolutionary dynamics. To overcome this limitation, we propose a novel collaborative learning framework for TKGR (dubbed CID-TKG) that integrates evolutionary dynamics and historical invariance semantics as an effective inductive bias for reasoning. Specifically, CID-TKG constructs a historical invariance graph to capture long-term structural regularities and an evolutionary dynamics graph to model short-term temporal transitions. Dedicated encoders are then employed to learn representations from each structure. To alleviate semantic discrepancies across the two structures, we decompose relations into view-specific representations and align view-specific query representations via a contrastive objective, which promotes cross-view consistency while suppressing view-specific noise. Extensive experiments verify that our CID-TKG achieves state-of-the-art performance under extrapolation settings.
Shuai-Long Lei, Xiaobin Zhu, Jiarui Liang +3
University of Science and Technology Beijing, Haidian, Beijing, China