OpenSanctions Pairs: Large-Scale Entity Matching with LLMs

Chandler Smith; Magnus Sesodia; Friedrich Lindenberg; Christian Schroeder de Witt

cs.IRFeb 24, 2026

OpenSanctions Pairs: Large-Scale Entity Matching with LLMs

Authors: Chandler Smith, Magnus Sesodia, Friedrich Lindenberg, Christian Schroeder de Witt

Organizations: University of Oxford · OpenSanctions

Abstract

We release OpenSanctions Pairs, the first large-scale public benchmark for entity matching on sanctions and OSINT data. The dataset includes 755,540 expert-labeled pairs over 1 million entities, aggregated from 293 source datasets across 45 jurisdictions. It captures real-world diversity in compliance data, spanning multiple languages and writing systems (e.g., Latin, Cyrillic, Arabic), inconsistent structure, and time-varying provenance, and is substantially more heterogeneous than prior entity matching benchmarks. As baselines, we evaluate the production rule-based matcher (nomenklatura RegressionV1) alongside open- and closed-source LLMs in both zero- and few-shot settings, each tested with and without MIPROv2 prompt optimization to control for prompt sensitivity. The rule-based baseline reaches 91.3% F1; GPT-4o achieves the best result at 99.0% F1, and a locally deployable open-source model (DeepSeek-R1-Distill-Qwen-14B) achieves 98.2% F1. The rule-based baseline and LLMs fail in complementary ways: rules over-match, while LLMs struggle with cross-script transliteration. These results suggest that pairwise matching performance is approaching a practical ceiling and shift attention toward pipeline components such as blocking, clustering, and uncertainty-aware review.

Figures & tables

Dataset	Entities	Matches	Sources	Unique Fields	Domain
Fodors-Zagat [ 21 , 22 ]	864	110	2	6	Restaurant
Abt-Buy [ 35 ]	2,173	1,097	2	4	E-commerce
Amazon-GoogleProducts [ 35 ]	4,589	1,300	2	4	E-commerce
DBLP-ACM [ 35 ]	4,910	2,220	2	4	Bibliographic
BeerAdvo-RateBeer [ 21 , 22 ]	7,345	68	2	4	Beverage
WDC Products [ 32 ]	11,715	2,162 ∗	3,259	5	E-commerce

Table 1: Comparison with existing entity matching benchmarks. Entities reports the total number of distinct entity records appearing in the labeled corpus (undeduplicated across sources for all rows). Matches is the count of ground-truth positive pairwise matches; * denotes deduplicated product clusters, not pairwise matches.

Analyst-judged matching subjects
Schema combination	Pairs	Share	Pos / Neg
Person, Person	284,808	37.7%	203K / 81K
Company / Organization / LegalEntity (any)	175,311	23.2%	89K / 86K
Position, Position	26,330	3.5%	24K / 3K
Vessel, Vessel	7,550	1.0%	5K / 2K
Person, LegalEntity, other cross-schema	7,299	1.0%	7K / 0.3K

Table 2: Pair-level schema combinations. Analyst-judged pairs (top block) are over genuine matching subjects and contain both positive and negative judgments. Auto-merge pairs (bottom block) are relational records (Occupancy, Succession, Ownership, etc.) that propagate when a parent entity match is recorded; they are 100% positive by construction.

Rule-Based Baseline
Model	Config	Acc. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$
nomenklatura/RegressionV1	—	85.45	91.33	84.46	99.42
Open-Source Models
Llama-3.1-8B	0-shot	90.40	94.05	89.84	98.67
Llama-3.1-8B	0-shot (opt)	93.88	95.94	97.98	93.97
Llama-3.1-8B	2-shot (opt)	93.00	95.46	94.85	96.08

Table 3: Main results. Configurations marked “(opt)” use DSPy MIPROv2 prompt optimization. Best results in Accuracy and F1 are shown in bold . See Section 5.2 for the sample sizes used.

Method	TP%	TN%	FP%	FN%
Nomenklatura	0.766	0.088	0.141	0.004
Llama-3.1-8B	0.759	0.145	0.086	0.010
DeepSeek-Qwen	0.758	0.208	0.030	0.005
GPT-3.5 Turbo	0.701	0.218	0.013	0.068
GPT-4o	0.762	0.222	0.009	0.007
GPT-5 Nano	0.761	0.163	0.068	0.008

Table 4: Per-method confusion matrix on the test split. DeepSeek-Qwen

=

DeepSeek-R1-Distill-Qwen-14B; all configurations are 0-shot, not optimized.

Figure 2: Unoptimized (0-shot) F1 versus model release date. The dashed gray line is the Nomenklatura rule baseline (91.3% F1); the blue dashed line is a linear fit across LLMs. The LLM-rule gap widened from

\sim

2 F1 points in 2023 to over 7 by late 2025.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Manual prompt (left, 170 words) vs. the MIPROv2-optimized prompt (right, 50 words). The optimizer discovered a more concise framing while preserving the conflict-detection principle.

Entity Resolution

Entity resolution links records that refer to the same real-world entity across datasets despite differences in names, attributes, formats, and languages.

See all 40 papers

Labeling Training Data for Entity Matching Using Large Language Models
Jun 27, 2026Aaron Steiner, Christian BizerLLM-Assisted Annotation Knowledge Distillation
Data and Web Science Group University of Mannheim Mannheim, Germany
IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law
Sep 18, 2026Genis Skura, Roland Bouffanais, Didier WernliNamed Entity Recognition Legal NLP
Department of Computer Science, Faculty of Science Global Studies Institute University of Geneva, Switzerland
Entity Matching using Decision Models
Oct 25, 2026Aaron Steiner, Ralph Peeters, Christian BizerLLM Decision-Making
Data and Web Science Group, University of Mannheim, Mannheim, Germany

Jun 27, 2026J/K move · Enter open · S save

Labeling Training Data for Entity Matching Using Large Language Models

Aaron Steiner, Christian Bizer

Data and Web Science Group University of Mannheim Mannheim, Germany

LLM-Assisted Annotation Knowledge Distillation

Open paper AskPDF arXiv

Large language models (LLMs) achieve strong entity matching performance without task-specific training data, but applying them to large sets of candidate pairs is slow and costly. Matchers built on pretrained language models (PLMs), such as BERT, offer faster inference but require training data. We systematically study knowledge-distillation workflows in which an LLM teacher labels training pairs for a smaller student matcher. We vary pair selection, labeling budget, teacher model, correspondence post-processing, and student model across eight benchmarks, including unseen entities and non-English data. We compare students trained on machine-labeled data with matchers trained on the original benchmark training sets. In most cases, PLM-based matchers trained on LLM-labeled data perform similarly to those trained on benchmark sets. Pair selection matters most for small labeling budgets, where active learning is often most effective. An open-weight teacher trains competitive students, so distillation requires no closed-weight models. Compact PLM-based students compete with much larger LLM students on most tasks while requiring 34 to 459 times less inference time than direct LLM matching. On the two benchmarks with high shares of unseen products, PLM-based students substantially underperform their teachers, as do students trained on benchmark data. Under GPT-5.2 pricing, LLM labeling costs per training set average $5.86 to $8.11. These findings support knowledge distillation as a practical approach to reduce the effort of labeling task-specific training data while enabling efficient inference.

Abstract

Entity Resolution

Explore similar work

Labeling Training Data for Entity Matching Using Large Language Models

IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law

Entity Matching using Decision Models