Historical people may appear under different languages, scripts, and transcription traditions, while distinct individuals may share highly similar or even identical names. This makes historical identity reconciliation more than a problem of string matching or transliteration. We introduce MHER, a provenance-controlled benchmark for pairwise reconciliation of person-name attestations from the Mongol world. MHER contains a balanced 396-pair Name-only core over 84 primary historical persons and a stricter 160-pair Source-grounded subset constructed from mention-by-source evidence, with entity-disjoint development and test splits. Across five generative systems, correctly Source-grounded evidence improves paired TEST accuracy by 12.96 to 94.44 percentage points relative to Name-only input. On five identical-surface different-person cases, all models fail under names alone (0/25 model-item decisions), whereas Source-grounded evidence yields 24/25 correct resolutions, with the remaining output an abstention. Context-only ablations show that historical descriptions often carry substantial identity information, while explicitly signaled misgrounding controls produce substantially lower performance. We also find that names are not uniformly beneficial: for Qwen3-8B, restoring surface forms converts ten otherwise correct Context-only distinctions into false identity merges. These results show that historical entity reconciliation depends not only on surface correspondence, but on whether identity judgments respond appropriately to provenance-controlled historical evidence. MHER therefore provides a controlled framework for studying evidence use, abstention, and failure modes in historical NLP.
Figures & tables
Figure 1: Overview of MHER benchmark construction and evaluation conditions. A source-linked scholarly registry defines canonical historical entities and their multilingual or multiscript mentions before evaluation pairs are generated. The balanced Name-only core is split by canonical entity. A stricter provenance-eligible subset is then constructed from mention × source-specific evidence, yielding paired Source-grounded evaluation. Context-only and deterministic shuffled-context controls manipulate the same paired items without altering the underlying gold identity relation. Expert-unresolved cases are retained separately from the binary benchmark core.
Component
DEV
TEST
Total
Primary entities
17
67
84
Name-only pairs
80
316
396
same
40
158
198
different
40
158
198
Source-grounded pairs
52
108
160
same
26
54
80
Table 1: Frozen MHER benchmark composition. Source-grounded, Context-only, and shuffled-context conditions operate on the same paired items. The expert- unresolved challenge is separate from the primary binary benchmark.
Evidence
Method
Accuracy
Macro-F1
Name
Exact normalized match
48.42
.326
Unicode Levenshtein
53.16
.453
Unidecode + Levenshtein
59.81
.598
Gemini Embedding 2
64.87
.625
Context
Token Jaccard
76.85
.758
Gemini Embedding 2
86.11
.861
Table 2: Frozen non-generative TEST baselines. Name-form methods are evaluated on the full 316-item Name-only TEST set; context methods use the paired 108-item TEST subset. All thresholds and fitted parameters were selected on DEV only.
Model
Acc.
Macro-F1
Abst.
Sel. Acc.
GPT-5.6 Terra
35.13
.495
63.61
96.52
GPT-5.6 Luna
6.01
.110
93.99
100.00
GPT-5.6 Sol
4.11
.079
95.89
100.00
Gemini 3.7 Flash
16.77
.270
83.23
100.00
Qwen3-8B
67.09
.671
0.00
67.09
Table 3: Name-only behavior on the full 316-item TEST set. Accuracy and abstention should be interpreted jointly: the four API systems frequently decline to resolve identities from names alone, whereas Qwen3-8B makes a binary decision for every item. Selective accuracy is accuracy conditional on a non-abstaining prediction.
Model
Name
Source
Δ
95% CI
W → C
C → W
Terra
32.41
100.00
+67.59
[58.33, 75.93]
73
0
Luna
6.48
99.07
+92.59
[87.04, 97.22]
100
0
Sol
4.63
99.07
+94.44
[89.81, 98.15]
102
0
Gemini
20.37
100.00
+79.63
[72.22, 87.04]
86
0
Qwen3-8B
75.00
87.96
+12.96
[2.78, 23.15]
23
9
Table 4: Primary paired TEST result on the same 108 historical identity questions. Name and Source report ordinary accuracy (%). Δ is the Source-grounded minus Name-only accuracy difference in percentage points; confidence intervals are paired item-bootstrap 95% intervals. W → C and C → W denote wrong-to-correct and correct-to-wrong transitions, respectively.
Figure 2: Accuracy on the paired 108-item TEST subset under Name-only and Source-grounded evidence. Panel A shows condition-specific accuracies. Panel B shows Source-grounded minus Name-only accuracy gains with paired item-bootstrap 95% confidence intervals. The four first-party API systems move from highly abstention-dominated Name-only behavior to near-ceiling Source-grounded resolution; Qwen3-8B shows a smaller but positive paired gain.
Model
Ctx.-only
Source
Shuffle range
ΔS−C
ΔS−Shuf
Terra
87.04
100.00
13.89–16.67
+12.96
+85.19
Luna
80.56
99.07
11.11–22.22
+18.52
+82.41
Sol
92.59
99.07
0.93–4.63
+6.48
+96.30
Gemini
94.44
100.00
49.07–50.00
+5.56
+50.31
Qwen3-8B
94.44
87.96
57.41–63.89
-6.48
+28.09
Table 5: Falsification and ablation results on the paired 108-item TEST set. ΔS−C is Source-grounded minus Context-only accuracy. ΔS−Shuf is Source-grounded minus the mean of the three deterministic shuffled-context accuracies. The three derangements are analyzed separately rather than treated as independent replications.
Figure 3: Falsification and ablation analysis on the paired 108-item TEST set. Horizontal ranges summarize the three deterministic shuffled-context controls; Context-only and correctly Source-grounded accuracy are shown separately. Correctly Source-grounded evidence substantially outperforms the three explicitly signaled misgrounding controls across all five systems. The annotation beside each model gives Source-grounded minus Context-only accuracy.
Model
Pair gain
Cluster boot.
LOO range
(pp)
mean [95% CI]
(pp)
Terra
+67.59
+70.37 [65.43, 75.64]
65.69–69.70
Luna
+92.59
+93.18 [90.48, 96.30]
91.92–94.17
Sol
+94.44
+93.87 [92.11, 96.05]
93.94–95.28
Gemini
+79.63
+77.01 [73.33, 80.77]
78.43–81.82
Qwen3-8B
+12.96
+12.08 [7.04, 16.67]
9.80–15.53
Table 6: Repeated-entity robustness for the primary Name-only to Source-grounded intervention. Cluster bootstrap resamples canonical entities rather than individual pairs. LOO reports the range of paired gains after removing each entity in turn. Additional entity-level sensitivity outputs are retained in the frozen project record.
Error category
Name-induced
Persistent
Total
Unsupported alias / transliteration
5
1
6
Decision–rationale inconsistency
4
0
4
Coarse role / chronology conflation
1
2
3
Total
10
3
13
Table 7: Qualitative taxonomy of the 13 Qwen3-8B Source-grounded errors. “Name-induced” denotes cases that are correct under Context-only evidence but become false same merges when the name surfaces are restored. “Persistent” denotes errors already present under Context-only evidence. Only aggregate coding is reported in this arXiv version.
Figure 4: Outcome composition on the paired 108-item TEST subset under Name-only (left) and Source-grounded evidence (right). The first-party API systems are strongly abstention-dominated when names are presented without historical evidence, whereas Qwen3-8B makes a binary decision on every Name-only item. Correctly grounded evidence largely removes abstention for the API systems and sharply compresses cross-model behavioral differences.
Entity resolution links records that refer to the same real-world entity across datasets despite differences in names, attributes, formats, and languages.
University of Zurich, Switzerland · 2 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Swiss Federal Institute of Technology Zurich (ETHZ), Switzerland