Debate to Align: Reliable Entity Alignment through Two-Stage Multi-Agent Debate

Cunda Wang; Ziying Ma; Po Hu; Weihua Wang; Feilong Bao

cs.CLApr 15, 2026

Debate to Align: Reliable Entity Alignment through Two-Stage Multi-Agent Debate

Authors: Cunda Wang, Ziying Ma, Po Hu, Weihua Wang, Feilong Bao

Organizations: Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, Central China Normal University, Wuhan, China · School of Computer Science, Central China Normal University, Wuhan, China · National Language Resources Monitoring and Research Center for Network Media, Central China Normal University, Wuhan, China · College of Computer Science, Inner Mongolia University, Hohhot, China ·

Abstract

Entity alignment (EA) aims to identify entities referring to the same real-world object across different knowledge graphs (KGs). Recent approaches based on large language models (LLMs) typically obtain entity embeddings through knowledge representation learning and use embedding similarity to identify an alignment-uncertain entity set. For each uncertain entity, a candidate entity set (CES) is then retrieved based on embedding similarity to support subsequent alignment reasoning and decision making. However, the reliability of the CES and the reasoning capability of LLMs critically affect the effectiveness of subsequent alignment decisions. To address this issue, we propose AgentEA, a reliable EA framework based on multi-agent debate. AgentEA first improves embedding quality through entity representation preference optimization, and then introduces a two-stage multi-role debate mechanism consisting of lightweight debate verification and deep debate alignment to progressively enhance the reliability of alignment decisions while enabling more efficient debate-based reasoning. Extensive experiments on public benchmarks under cross-lingual, sparse, large-scale, and heterogeneous settings demonstrate the effectiveness of AgentEA.

Figures & tables

Figure 1: Overview of an LLM-based EA framework. Entities with a small Top-1/Top-2 similarity gap (e.g., $e_{1}$ and $e_{2}$ ) are forwarded to LLM-based reasoning, whereas those with a large gap are directly aligned (e.g., $e_{3}$ ).

Figure 2: The overview framework of our proposed AgentEA, which consists of two main components: (1) embedding-based candidate retrieval and (2) two-stage multi-role debate alignment.

Models	DBP15K ${}_{\text{ZH-EN}}$			DBP15K ${}_{\text{JA-EN}}$			DBP15K ${}_{\text{FR-EN}}$
Models	Hits@1	Hits@10	MRR	Hits@1	Hits@10	MRR	Hits@1	Hits@10	MRR
Knowledge Representation Learning-based Entity Alignment Methods
MTransE	0.308	0.614	0.364	0.279	0.575	0.349	0.247	0.577	0.360
GCN-Align	0.413	0.744	0.549	0.399	0.745	0.546	0.411	0.772	0.530
BootEA	0.629	0.848	0.703	0.622	0.854	0.701	0.653	0.874	0.731
RDGCN	0.708	0.846	0.746	0.767	0.895	0.812	0.873	0.950	0.901

Table 1: Main experiment results on DBP15K dataset. * denotes results reproduced using publicly available code.

Models	ICEWS-WIKI			ICEWS-YAGO			DWY ${}_{\text{DBP-WIKI}}$			DWY ${}_{\text{DBP-YAGO}}$
Models	H@1	H@10	MRR	H@1	H@10	MRR	H@1	H@10	MRR	H@1	H@10	MRR
Knowledge Representation Learning-based Entity Alignment Methods
MTransE	0.021	0.158	0.068	0.012	0.084	0.040	0.281	0.520	0.363	0.252	0.493	0.334
GCN-Align	0.046	0.184	0.093	0.017	0.085	0.038	0.506	0.772	0.600	0.597	0.838	0.682
BootEA	0.072	0.275	0.139	0.020	0.120	0.056	0.748	0.898	0.801	0.761	0.894	0.808
RDGCN	0.064	0.202	0.096	0.029	0.097	0.042	0.623	0.805	0.684	0.936	0.973	0.950

Table 2: Results on ICEWS-WIKI, ICEWS-YAGO, DWY

{}_{\text{DBP-WIKI}}

and DWY

{}_{\text{DBP-YAGO}}

datasets.

Settings	DBP ${}_{\text{FR-EN}}$		SRPRS ${}_{\text{EN-DE}}$
Settings	Hits@1	MRR	Hits@1	MRR
AgentEA	0.996	0.998	0.977	0.980
- only DPO	0.976	0.986	0.948	0.963
- w/o DPO	0.859	0.889	0.862	0.888
- w/o DN	0.938	0.955	0.928	0.945
- w/o NN	0.960	0.976	0.945	0.960

Table 3: Ablation experiments on DBP15K

{}_{\text{FR-EN}}

and SRPRS

{}_{\text{EN-DE}}

datasets during the candidate entity generation phase.

Panel of Figure 3 — Figure 3: Visualization of entity embedding on DBP15K ${}_{\text{FR-EN}}$ . Different colors represent different KGs.

Settings	DBP15K ${}_{\text{FR-EN}}$
Settings	Hits@1	Hits@10	MRR
AgentEA	0.996	1.000	0.998
w/o Debate	0.976	0.984	0.986
w/o LDV	0.995	0.997	0.996
w/o DDA	0.986	0.999	0.991
w/o Attribute Agent	0.991	0.998	0.994

Table 4: Ablation experiments during the debate phase on DBP15K

{}_{\text{FR-EN}}

dataset (w/o Debate: only DPO).

Panel of Figure 4 — Figure 4: Additional experiments.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Sample type	Count
Positive pairs	13,500
Neighborhood-aware negative pairs	13,500
Name-similarity-based negative pairs	3,425

Appendix

Table 5: Composition of DPO training samples

Settings	DBP ${}_{\text{FR-EN}}$
Settings	Hits@1	MRR
LLaMA3-8B-Instruct	0.859	0.889
LLaMA3-8B-Instruct+DPO	0.976	0.986
LLaMA3-8B-Instruct+SFT	0.846	0.875

Appendix

Table 6: Comparison between DPO and SFT for fine-tuning LLaMA3-8B on entity alignment tasks.

Item	Configuration
LoRA	$r=22$ , $\alpha=44$ , dropout $=0.1$
Modules	q_proj , v_proj
Training	lr $=1\mathrm{e}{-4}$ , epochs $=1$ , bf16

Appendix

Table 7: LoRA Training Configuration

Dataset	KG	Entities	Relations	Attributes	Rel. Triples	Attr. Triples
DBP15K ${}_{\text{ZH-EN}}$	ZH	19,388	1,701	8,113	70,414	379,684
DBP15K ${}_{\text{ZH-EN}}$	EN	19,572	1,323	7,173	95,142	567,755
DBP15K ${}_{\text{JA-EN}}$	JA	19,814	1,299	5,882	77,214	354,619
DBP15K ${}_{\text{JA-EN}}$	EN	19,780	1,153	6,066	93,484	497,230
DBP15K ${}_{\text{FR-EN}}$	FR	19,661	903	4,547	105,998	354,619
DBP15K ${}_{\text{FR-EN}}$	EN	19,993	1,208	6,422	115,722	497,230

Appendix

Table 8: Statistical data of DBP15K, SRPRS and DWY.

Dataset	Entities	Relations	Facts	Density	Anchors	Overlapping	Struc. Sim.
ICEWS-WIKI	11,047	272	3,527,881	319.352	5,058	45.79%	15.4%
ICEWS-WIKI	15,896	226	198,257	12.472	5,058	31.82%	15.4%
ICEWS-YAGO	26,863	272	4,192,555	156.072	18,824	70.07%	14.0%
ICEWS-YAGO	22,734	41	107,118	4.712	18,824	82.80%	14.0%

Appendix

Table 9: Statistical data of ICEWS.

Models	SRPRS ${}_{\text{EN-DE}}$			SRPRS ${}_{\text{EN-FR}}$			SRPRS ${}_{\text{DBP-YAGO}}$			SRPRS ${}_{\text{DBP-WIKI}}$
Models	H@1	H@10	MRR	H@1	H@10	MRR	H@1	H@10	MRR	H@1	H@10	MRR
MTransE	0.107	0.248	0.160	0.213	0.447	0.290	0.196	0.401	0.270	0.188	0.382	0.260
MuGNN	0.245	0.431	0.310	0.131	0.342	0.208	0.175	0.381	0.240	0.151	0.366	0.220
NAEA	0.307	0.535	0.390	0.177	0.416	0.260	0.195	0.451	0.280	0.182	0.429	0.260
GCN-Align	0.385	0.600	0.460	0.243	0.522	0.340	0.319	0.586	0.410	0.291	0.556	0.380
KECG	0.444	0.707	0.540	0.298	0.616	0.403	0.350	0.651	0.450	0.323	0.646	0.430

Appendix

Table 10: Main experiment results on SRPRS dataset.

Figure 5: Trade-off analysis under different debate thresholds.

Figure 6: Trade-off analysis under different expansion thresholds.

\mathcal{E}_{u}=\{e_{i}\mid s_{i,1}-s_{i,2}<\delta_{1}\}\;

Appendix

Algorithm 1 AgentEA: Multi-Agent Debate for Reliable Entity Alignment

	19488	15519	13352	16770	16053
	No Good Either Way	House of Harmony	Romantic Princess	The Life and Times of a Sentinel	Just Love II
Multi-Agent Debate
Embedding Sim.	0.6368	0.6357	0.6181	0.6147	0.6082
Alias Agent	score=0.0, align= $\times$ , reason: different names	score=0.0, align= $\times$ , reason: different names	score=0.0, align= $\times$ , reason: different names	score=0.0, align= $\times$ , reason: different names	score=0.0, align= $\times$ , reason: different names
Type Agent	score=1.0, align= $\checkmark$ , reason: same type (TV show)	score=1.0, align= $\checkmark$ , reason: same type (TV show)	score=1.0, align= $\checkmark$ , reason: same type (TV show)	score=1.0, align= $\checkmark$ , reason: same type (TV show)	score=1.0, align= $\checkmark$ , reason: same type (TV show)
Attribute Agent	score=0.0, align= $\times$ , reason: different firstAired	score=0.0, align= $\times$ , reason: different firstAired	score=0.0, align= $\times$ , reason: different firstAired	score=1.0, align= $\checkmark$ , reason: same episodes	score=0.0, align= $\times$ , reason: different firstAired

Appendix

Table 11: Case study illustrating how MAD refines alignment decisions through multi-agent debate. The source entity is

\langle 6270,\ \text{Zijin Jing Lei}\rangle

, and the correct alignment is 16770 .

Settings	DBP15K ${}_{\text{FR-EN}}$
Settings	Hits@1	Hits@10	MRR
AgentEA	0.996	1.000	0.998
AgentEA(Single-LLM)	0.981	0.993	0.982

Appendix

Table 12: Ablation experiments during the debate phase on DBP15K

{}_{\text{FR-EN}}

dataset.

Multi-Agent Debate

Multi-Agent Debate studies how language-model agents exchange arguments, critique answers, and combine judgments for collective reasoning.

See all 144 papers

Reliability of LLM Judges for Evaluating Entity Alignment
Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalKG Entity Alignment LLM-as-a-Judge
University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California
Predicate Importance Estimation and Decoupled Rationale-Score Distillation for Entity Alignment
Jun 22, 2026Keunha Kim, Yoonjin Jang, Hyeon-gu Lee +2KG Entity Alignment KG Embedding
SungKyunKwan University · NAVER
IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment
Jul 28, 2026Xinran Liu, Shengtao Li, Shouqian Shi +2KG Entity Alignment KG Embedding

Oct 7, 2026J/K move · Enter open · S save

Reliability of LLM Judges for Evaluating Entity Alignment

Vaibhava Lakshmi Ravideshik, Mayank Kejriwal

University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California

KG Entity Alignment LLM-as-a-Judge Bias in LLM-as-a-Judge LLM Reliability Knowledge Graphs

Open paper AskPDF arXiv

Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12-0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93-1.00) and significant recovery on biomedical pairs (0.93-0.95). Counterfactual experiments confirm causality (FSR 53-99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen's kappa=0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH-SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.

Abstract

Multi-Agent Debate

Explore similar work

Reliability of LLM Judges for Evaluating Entity Alignment

Predicate Importance Estimation and Decoupled Rationale-Score Distillation for Entity Alignment

IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment