Learning Chain Of Thoughts Prompts for Predicting Entities, Relations, and even Literals on Knowledge Graphs

Alkid Baci; Luke Friedrichs; Caglar Demir; N'Dah Jean Kouagou; Axel-Cyrille Ngonga Ngomo

cs.CLApr 14, 2026

Learning Chain Of Thoughts Prompts for Predicting Entities, Relations, and even Literals on Knowledge Graphs

Authors: Alkid Baci, Luke Friedrichs, Caglar Demir, N'Dah Jean Kouagou, Axel-Cyrille Ngonga Ngomo

Organizations: Department of Computer Science Paderborn University Warburger Str. 100, 33098 Paderborn, Germany

Abstract

Knowledge graph embedding (KGE) models perform well on link prediction but struggle with unseen entities, relations, and especially literals, limiting their use in dynamic, heterogeneous graphs. In contrast, pretrained large language models (LLMs) generalize effectively through prompting. We reformulate link prediction as a prompt learning problem and introduce RALP, which learns string-based chain-of-thought (CoT) prompts as scoring functions for triples. Using Bayesian Optimization through MIPRO algorithm, RALP identifies effective prompts from fewer than 30 training examples without gradient access. At inference, RALP predicts missing entities, relations or whole triples and assigns confidence scores based on the learned prompt. We evaluate on transductive, numerical, and OWL instance retrieval benchmarks. RALP improves state-of-the-art KGE models by over 5% MRR across datasets and enhances generalization via high-quality inferred triples. On OWL reasoning tasks with complex class expressions (e.g., $\exists hasChild.Female$ , $\geq 5 \; hasChild.Female$ ), it achieves over 88% Jaccard similarity. These results highlight prompt-based LLM reasoning as a flexible alternative to embedding-based methods. We release our implementation, training, and evaluation pipeline as open source: https://github.com/dice-group/RALP .

Figures & tables

Figure 1: Overview of the RALP framework in three different tasks within the link prediction problem. Notice that sampling of triples or MIPRO optimization can be performed in all three of them but is omitted because of space limitations.

Models	S1				S2				S3
	MRR	@1	@3	@10	MRR	@1	@3	@10	MRR	@1	@3	@10
$\textsc{RALP}^{\dagger}$ light	0.979	0.958	1.000	1.000	0.750	0.500	1.000	1.000	0.595	0.375	0.791	0.833
$\textsc{RALP}^{\dagger}$ medium	1.000	1.000	1.000	1.000	0.750	0.500	1.000	1.000	0.764	0.541	1.000	1.000

Table 1: Link prediction results of

\textsc{RALP}^{\dagger}

on Countries-S1, Countries-S2, and Countries-S3. RALP single Bold and underlined results indicate the best results and second best results.

Models	S1				S2				S3
	MRR	@1	@3	@10	MRR	@1	@3	@10	MRR	@1	@3	@10
DistMult	0.783	0.583	0.979	1.000	0.582	0.583	0.667	0.958	0.283	0.104	0.375	0.604
DistMult- RALP	0.990	0.979	1.000	1.000	0.753	0.458	0.938	0.958	0.670	0.542	0.771	0.896
Keci	0.716	0.479	0.917	1.000	0.627	0.501	0.667	0.896	0.292	0.188	0.333	0.583
Keci- RALP	1.000	1.000	1.000	1.000	0.596	0.521	0.696	0.950	0.667	0.500	0.771	0.917
DeCaL	0.712	0.500	0.896	0.979	0.481	0.271	0.708	0.812	0.312	0.188	0.354	0.583

Table 2: Link prediction results on Countries-S1, Countries-S2, and Countries-S3. Link prediction results on the training and testing datasets are denoted with train and test. Bold results indicate the best generalization performance of a respective model.

Data Property	$y_{\text{avg}}$	$\sigma$	$\hat{y}_{\text{avg}}$	ICR	IW
population	$60\,923\,964$	$\pm$ 3 $44\,077\,886$	$58\,504\,042$	0.90	$167\,280\,658$
age_of_majority_years_old	18.3	$\pm$ 1.1	18.1	0.90	3.175
total_fertility_rate	3.029	$\pm$ 1.435	2.756	0.80	2.709
nominal_gdp_per_capita_usd	$11\,510$	$\pm$ $19\,408$	$11\,034$	0.75	$15\,140$
human_development_index	0.677	$\pm$ 0.145	0.698	0.79	0.236
life_expectancy_second	$2\,220\,348\,629$	$\pm$ $236\,968\,936$	$2\,204\,513\,934$	0.81	$640\,296\,028$

Table 3: Numerical link prediction results in LitWD1K dataset. The shown data properties are selected randomly without replacement for the illustration purposes. We predict

\hat{y}

together with interval bounds

\hat{y}_{\text{max}}

and

\hat{y}_{\text{min}}

. For our evaluation we measure the mean value of

{y}

, standard deviation

\sigma

, the mean value of

\hat{y}

and for the interval prediction, we measure interval coverage rate (ICR) for calibration and interval width (IW) for sharpness.

Figure 2: RALP predictions for the human_development_index property (P1081) in LitWD1K dataset. Ground truth depicted in black dots, regression prediction in blue cross and interval range in teal lines. For this property we report a mean squared error of 0.006 and a mean absolute error of 0.059. Some countries are omitted from the plot because of the limited space.

Concept Types	#	Namespace		No Namespace
		M	DL	M	DL
Atomic	3	1.000	1.000	1.000	1.000
Negation	55	0.887	0.803	0.869	0.801
Conjunction	19	0.986	0.881	0.973	0.973
Disjunction	33	0.984	0.775	0.969	0.818
Existential	44	0.656	0.700	0.673	0.764

Table 4: Results of instance retrieval on Father datasets using RALP. We show the mean Jaccard similarity measured between the true set and the predicted set for each concept type where ’#’ denotes the number of concepts per concept type, ’Namespace’ denotes whether we included the namespace for triples in the graph or not, ’M’ means that the expression was set in Manchester syntax and ’DL’ means that the expression was set in Description Logics syntax. The best results for each concept type are marked in bold.

KG Completion

Knowledge graph completion (KGC) predicts missing entities, relations, or facts from structured graphs and supporting evidence.

See all 73 papers

RADAR: Reasoning as Discrimination with Aligned Representations for LLM-based Knowledge Graph Reasoning
Feb 25, 2026Bo Xue, Yuan Jin, Luoyi Fu +2Graph Reasoning KG Completion
Shanghai Jiao Tong University, Shanghai, China
Predicate Importance Estimation and Decoupled Rationale-Score Distillation for Entity Alignment
Jun 22, 2026Keunha Kim, Yoonjin Jang, Hyeon-gu Lee +2KG Embedding Knowledge Graphs
SungKyunKwan University · NAVER
Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs
Jun 2, 2026Weiwei Ding, Zixuan Li, Long Bai +7LLM Grounding LLM Reasoning with Graphs
1Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · 2Shandong University · 3Shandong University-Weihai Research Institute of Industrial Technology

Feb 25, 2026J/K move · Enter open · S save

RADAR: Reasoning as Discrimination with Aligned Representations for LLM-based Knowledge Graph Reasoning

Bo Xue, Yuan Jin, Luoyi Fu, Jiaxin Ding+1

Shanghai Jiao Tong University, Shanghai, China

Graph Reasoning KG Completion LLMs for Graph Learning Knowledge Graph Question Answering

Open paper AskPDF arXiv

Knowledge graph reasoning (KGR) infers missing facts, with recent advances increasingly harnessing the semantic priors and reasoning abilities of Large Language Models (LLMs). However, prevailing generative paradigms are prone to memorizing surface-level co-occurrences rather than learning genuine relational semantics, limiting out-of-distribution generalization. To address this, we propose RADAR, which reformulates KGR from generative pattern matching to discriminative relational reasoning. We recast KGR as discriminative entity selection, where reinforcement learning enforces relative entity separability beyond token-likelihood imitation. Leveraging this separability, inference operates directly in representation space, ensuring consistency with the discriminative optimization and bypassing generation-induced hallucinations. Across four benchmarks, RADAR achieves 5-6% relative gains on link prediction and triple classification over strong LLM baselines, while increasing task-relevant mutual information in intermediate representations by 62.9%, indicating more robust and transferable relational reasoning.

Abstract

KG Completion

Explore similar work

RADAR: Reasoning as Discrimination with Aligned Representations for LLM-based Knowledge Graph Reasoning

Predicate Importance Estimation and Decoupled Rationale-Score Distillation for Entity Alignment

Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs