Rethinking Generalization in Knowledge Graph Embedding: An Empirical Study and Sparse Peak Regularization

Linyu Li; Zhi Jin; Yuanpeng He; Dongming Jin; Haoran Duan; Zhengwei Tao; Xuan Zhang; Jiandong Li

cs.LGMay 29, 2025

Rethinking Generalization in Knowledge Graph Embedding: An Empirical Study and Sparse Peak Regularization

Authors: Linyu Li, Zhi Jin, Yuanpeng He, Dongming Jin, Haoran Duan, Zhengwei Tao, Xuan Zhang, Jiandong Li,

Organizations: School of Computer Science, Peking University, Beijing, China · Wuhan University, Wuhan, China

Abstract

Knowledge graphs (KGs) provide structured representations of entities and relations, and underpin data management and Web applications such as search, recommendation, and question answering. Since Web-scale KGs such as Freebase and Wikidata are inherently incomplete, knowledge graph embedding (KGE) maps entities and relations into continuous vector spaces and is widely used to infer missing facts through link prediction. Despite extensive progress in KGE model design, the role of regularization has been less systematically examined, especially beyond tensor-decomposition models. The primary contribution of this paper is a comprehensive empirical study of regularization for KGE from the perspective of generalization. Across translation-based, tensor-decomposition-based, GNN-based, temporal, and multimodal KGE models, we show that overfitting is common across model families and that appropriate regularization can substantially improve both generalization and predictive performance. Motivated by these findings, we further propose Sparse Peak Regularization (SPR), a simple and model-agnostic regularizer that introduces rank-based selective sparsity into the regularization target. SPR concentrates the penalty on dominant high-energy embedding components, which contribute most to the embedding norm and overfitting risk, while leaving low-energy components unpenalized so that weak but useful signals are not unnecessarily suppressed. Experiments on eight benchmark datasets and diverse KGE models show that SPR achieves the best results in nearly all settings compared with existing regularizers such as N3, DURA, and ER. These results show that regularization deserves more systematic attention in KGE, and that selective regularization is an effective and broadly applicable strategy for improving the generalization of KGE models.

Figures & tables

Figure 1: Overview of regularization and generalization in KGE. (a) Regularized training combines the main prediction loss $\mathcal{L}_{p}$ with a weighted penalty $\lambda\Omega$ , encouraging better-separated embeddings and stronger downstream behavior. (b) Without regularization, KGE minimizes only $\mathcal{L}_{p}$ and is more susceptible to poorly separated embeddings and weaker downstream performance.

Figure 2: Comparison of the overfitting performance of the GIE [ 11 ] model without regularization and with regularization training on the FB15K-237 dataset: Visual comparison of the model Loss, MRR, MR, and Hits@1 metrics as they change with Epoch. The shaded region denotes the train-validation gap. By adding the regularization method, this generalization gap is substantially narrowed.

Dataset	Reg.	CP				ComplEx				GIE				RESCAL
Dataset	Reg.	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
WN18RR	No-Reg	0.438	0.416	0.465	0.485	0.457	0.426	0.471	0.518	0.464	0.434	0.479	0.529	0.444	0.419	0.453	0.491
	F2	0.449	0.421	0.474	0.506	0.468	0.431	0.479	0.535	0.470	0.439	0.489	0.553	0.451	0.424	0.462	0.511
	N3	0.469	0.432	0.485	0.541	0.488	0.445	0.505	0.570	0.491	0.452	0.507	0.576	0.475	0.437	0.479	0.556
	DURA	0.471	0.433	0.488	0.545	0.488	0.447	0.502	0.568	0.488	0.449	0.503	0.571	0.495	0.454	0.511	0.576
	ER	0.475	0.436	0.489	0.549	0.486	0.444	0.502	0.566	0.490	0.450	0.505	0.574	0.493	0.452	0.505	0.575

Table 1: Main link prediction results on WN18RR, FB15K-237, and YAGO3-10; all models were run on the same machine. Within each dataset block, bold and underlined values are the best and second-best in each column (ties are marked alike).

Figure 3: Training curves with and without regularization across static and temporal KGE models. Panels (a)–(i) report MRR for CP, ComplEx, and RESCAL (rows) on WN18RR, FB15K-237, and YAGO3-10 (columns); panels (j) and (k) report HGE on ICEWS14 (MRR) and ICEWS05-15 (Hits@1), respectively. Orange and red denote No Reg and Add Reg; solid and dashed curves denote training and validation, dotted horizontal lines mark the final validation performance after convergence, and shading marks the train–validation gap. Regularization consistently narrows the gap and improves validation performance.

Reg.	CP				ComplEx
	UMLS		Kinship		UMLS		Kinship
	MRR	H@1	MRR	H@1	MRR	H@1	MRR	H@1
No-Reg	0.868	0.780	0.872	0.784	0.897	0.822	0.875	0.783
N3	0.879	0.782	0.885	0.819	0.907	0.843	0.883	0.786
DURA	0.881	0.788	0.885	0.792	0.904	0.841	0.888	0.796
ER	0.883	0.791	0.888	0.820	0.910	0.844	0.887	0.795

Table 2: Results on UMLS and Kinship. Bold and underlined values are the best and second-best in each column.

Figure 4: Overfitting analysis of OTKGE on WN9-IMG. (a)–(c) Train (dashed) and test (solid) MRR under No-Reg, N3, and SPR; the shaded area is the train–test gap, whose final value is given in each panel title. (d) Test MR of the three settings.

Figure 5: Sensitivity of ComplEx to the SPR regularization rate on five benchmarks. Green and orange bars report MRR and Hits@1, respectively; red and black dashed lines mark the corresponding no-regularization baselines.

Figure 6: MRR under (a)–(c) different SPR thresholds

\delta

, with no-regularization baselines (dashed) and

\delta=0.4

(dotted), and (d)–(e) No-Reg, Dropout, and SPR for CompGCN-TransE (CompGCN-T) and CompGCN-ConvE (CompGCN-C); red arrows give the relative gains of SPR.

Panel of Figure 7 — Figure 7: ComplEx entity embeddings on WN18RR under (a) SPR, (b) DURA, (c) ER, and (d) no regularization, projected by t-SNE and colored by K-Means clusters ( $K{=}10$ ); colors are not matched across panels.

KG Embedding

Knowledge graph embedding (KGE) represents entities and relations as vectors or structured geometric objects for prediction and reasoning.

See all 55 papers

Sep 3, 2026

Pattern Over-Generalization of Knowledge Graph Embedding

Knowledge graph embedding (KGE) demonstrates its effectiveness for predicting missing links in knowledge graphs (KGs) by projecting entities and relations into a low-dimensional vector space. It is crucial for KGE models to effectively capture inference patterns (patterns) inherent in KGs, such as symmetry/antisymmetry, inversion and composition. Although recent KGE models exhibit strong capabilities in modeling such diverse patterns, they suffer from inherent limitations stemming from pattern over-generalization, where embeddings learned from only a single pattern instance inevitably generalize that pattern to all related instances, i.e., generalize the pattern universally. To address this issue, we propose PogRE (Pattern Over-Generalization Robust Embedding), a simple but effective method that utilizes dense linear transformations and compound operations for relation representation. Our theoretical analysis demonstrates that a dense linear transformation allows a pattern to become progressively universal as more triples are observed in the pattern. Furthermore, after observing d+1 linearly independent entities (d+1 denotes the dimension of entity), the linear transformation guarantees universal generalization of the pattern across all related instances. Experimental results on three standard benchmark datasets show that PogRE outperforms existing state-of-the-art KGE models in link prediction. Moreover, our empirical results indicate that PogRE effectively addresses the negative impact of over-generalization.

Junsik Kim, Kangil Kim

Gwangju Institute of Science and Technology

Link Prediction KG Embedding KG Completion

Ask PDF

Oct 7, 2026

KGATE : a Knowledge Graph Embedding Training Environment

Knowledge graph embedding (KGE) models encode the entities and relations of a knowledge graph into a low-dimensional latent space, enabling tasks such as classification or link prediction. Most KGE models follow an autoencoder architecture, in which an encoder projects the knowledge graph into the latent space and a decoder reconstruct it. Combining both encoder and decoder components is increasingly needed, yet existing libraries rarely support complete autoencoders, are often unmaintained, rely on undocumented default hyperparameters, and produce results that cannot be compared across libraries. Here we present KGATE (Knowledge Graph Autoencoder Training Environment), a modular Python library built on PyTorch Geometric and TorchKGE. KGATE lets users assemble initializers, encoders, decoders, losses, negative samplers, and evaluation metrics as building blocks, or plug in their own block. KGATE includes a preprocessing procedure that controls data leakage, a builtin training pipeline, and reproducibility by design. Benchmarks against six existing KGE libraries show that KGATE training time is comparable with the fastest libraries while offering a broader set of features.

Benjamin Loire, Galadriel Brière, Célia Brahimi +2

Aix Marseille Univ., INSERM, Marseille Medical Genetics, Systems Biomedicine Team, Marseille, France · Neurology Therapeutic Area, R&D Servier Paris-Saclay Institut, Gif-sur-Yvette, France · LIRMM, Univ. Montpellier, CNRS, Montpellier, France +1

KG Embedding Autoencoders

Ask PDF

Apr 21, 2026

Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding

Knowledge Graph Embeddings (KGEs) support a wide range of downstream tasks over Knowledge Graphs (KGs). In practice, KGs evolve as new entities and facts are added, motivating Continual Knowledge Graph Embedding (CKGE) methods that update embeddings over time. Current CKGE approaches address catastrophic forgetting (i.e., the performance degradation on previously learned tasks) primarily by limiting changes to existing embeddings. However, we show that this view is incomplete. When new entities are introduced, their embeddings can interfere with previously learned ones, causing the model to predict them in place of previously correct answers. This phenomenon, which we call entity interference, has been largely overlooked and is not accounted for in current CKGE evaluation protocols. As a result, the assessment of catastrophic forgetting becomes misleading, and CKGE methods performance is systematically overestimated. To address this issue, we introduce a corrected CKGE evaluation protocol that accounts for entity interference. Through experiments on multiple benchmarks, we show that ignoring this effect can lead to performance overestimation of up to 25%, particularly in scenarios with significant entity growth. We further analyze how different CKGE methods and KGE models are affected by the different sources of forgetting, and introduce a catastrophic forgetting metric tailored to CKGE.

Gerard Pons, Carlos Escolano, Besim Bilalli +1

Data-intensive Technologies and Knowledge Systems (DTAK), Department of Service and Information System Engineering, Universitat Politècnica de Catalunya, Barcelona, 08034, Catalonia, Spain · Intelligent Data Science and Artificial Intelligence Research Center (IDEAI), Department of Computer Science, Universitat Politècnica de Catalunya, Barcelona, 08034, Catalonia, Spain

Graph Representation Learning KG Embedding Graph Embedding KG Completion Catastrophic Forgetting

Ask PDF

Abstract

KG Embedding

Explore similar work

Pattern Over-Generalization of Knowledge Graph Embedding

KGATE : a Knowledge Graph Embedding Training Environment

Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding