cs.CLOct 25, 2026
SaveEntity Matching using Decision Models
Organizations: Data and Web Science Group, University of Mannheim, Mannheim, Germany
Abstract
Entity matching is the task of deciding whether two records describe the same real-world entity. Current entity matching methods rely on pre-trained language models (PLMs) which require task specific fine-tuning or generative large language models (LLMs) which do not require task-specific fine-tuning but are slow and expensive at matching large record sets. Decision models are a new class of LLM-based models that return a probability for each predefined answer option rather than generate text. This paper investigates decision models as an alternative to generative LLMs and fine-tuned PLMs for entity matching. We evaluate six decision models in a zero-shot setting across eight benchmarks and compare their performance with that of the generative LLM GPT-6 Luna and the PLM-based matcher Ditto. Our experiments show that, without task-specific training data, the four strongest decision models achieve mean F1 scores close to that of GPT-6 Luna, with the best model slightly surpassing it. This model also outperforms Ditto on six of the eight benchmarks. All decision models are faster than GPT-6 Luna, and most are less expensive. For three of the four strongest decision models, the predicted match probabilities are well calibrated without requiring any task-specific recalibration. A model cascade that routes the least confident record pairs to an ensemble of generative LLMs improves the mean F1 scores of the four strongest decision models by 1.9-4.0 percentage points. Using the best decision model and its best-performing prompt for each benchmark, the cascade surpasses the highest published F1 scores from literature on four of the eight benchmarks.
Figures & tables
Figure 1. A decision model answers a yes/no question about a record pair by returning a probability (field noul ). We classify the pair as Match if Match if and as No match otherwise. Two product records with the titles DYMO D1 - Roll (1.9cm x 7m) and DYMO D1 Tape 12mm x 7m form a request with the question whether both refer to the same real-world entity and the rule that different tape widths mean different products. A decision model returns the match probability as its raw output "noul": 0.02, shown on a scale from 0 to 1 with a mark at 0.5. Because the probability is below 0.5, the decision is no match.
| Validation | Test | |||
| Benchmark | #Pos | #Neg | #Pos | #Neg |
| Abt-Buy | 205 | 1,696 | 206 | 1,710 |
| Walmart-Amazon | 193 | 1,854 | 193 | 1,856 |
| WDC Products | 500 | 4,000 | 500 | 4,000 |
| DBLP-ACM | 442 | 2,007 | 444 | 2,029 |
| DBLP-Scholar | 1,070 | 4,635 | 1,070 | 4,672 |
Table 1. Matching (#Pos) and non-matching (#Neg) pairs in the validation and test sets.
| Model | Identifier | Weights | Base model |
|---|---|---|---|
| Jev 1.13 | typesafe/jev-1.13-20260917 | closed | n/d |
| Clef | cloudflare/clef | open | Qwen3.8-27B |
| Clef-flash | cloudflare/clef-flash | open | Qwen3.5-9B |
| Luna Dec. | openai/gpt-6-luna-decisions-20261006 | closed | GPT-6 Luna |
| PPLX v1.2 | perplexity/pplx-decider-v1.2-20261016 | open | Qwen3.8-27B |
| Kev | jaredpalmer/kev-4b-20260924 | open | Qwen3.5-4B-Base |
Table 2. Model versions as returned by the OpenRouter API (Luna Gen.: OpenRouter snapshot), weights, and base models (n/d: not disclosed).
| Decision model | Generative LLM | |
|---|---|---|
| Input | State: the serialized record pair | User message: the serialized record pair |
| Question | Yes/no question with the prompt as its text | System message: the prompt |
| Answer options | Declared in the request: yes ( true ) and no ( false ), each with a one-sentence description | Fixed by a JSON schema with a single Boolean field match |
| Output | Probability of yes, the match probability , no text | Generated text: {"match": true} or {"match": false} |
| Decision | Match if | Match if match is true |
Table 3. Comparison of the input, output, and decision rule of decision models and generative LLMs.
| Benchmark | Jev 1.13 | PPLX v1.2 | Clef | Clef-flash | Luna Dec. | Kev | Luna Gen. | Qwen3.8-27B | Ditto |
|---|---|---|---|---|---|---|---|---|---|
| Abt-Buy | 90.91 0.15 | 93.81 0.16 | 90.87 0.00 | 83.33 0.00 | 91.03 0.00 | 87.56 0.00 | 94.40 0.47 | 75.08 2.43 | 88.34 1.54 |
| Walmart-Amazon | 87.51 0.37 | 90.47 0.03 | 88.02 0.00 | 65.59 0.00 | 84.58 0.00 | 74.37 0.16 | 90.56 0.62 | 70.61 1.57 | 85.66 1.04 |
| WDC Products | 85.30 0.41 | 87.01 0.13 | 85.48 0.00 | 72.17 0.00 | 86.54 0.00 | 76.38 0.02 | 88.22 0.16 | 65.60 1.19 | 71.23 0.83 |
| DBLP-ACM | 96.97 0.16 | 98.11 0.00 | 97.24 0.00 | 92.05 0.00 | 96.64 0.07 | 76.67 0.23 | 97.50 0.44 | 84.76 0.47 | 98.43 0.34 |
| DBLP-Scholar | 91.93 0.01 | 92.57 0.03 | 90.46 0.00 | 90.21 0.00 | 90.12 0.00 | 88.98 0.05 | 88.91 0.33 | 76.63 0.39 | 95.64 0.26 |
| Dn7 Walmart-Amazon | 75.71 0.50 | 77.22 0.44 | 74.37 0.00 | 40.40 0.00 | 74.68 0.00 | 68.40 0.09 | 77.72 0.27 | 32.14 1.58 | 72.20 3.93 |
Table 4. Test set F1 scores (%) using the default prompt B0, mean SD over three runs (Ditto: three seeds). Qwen3.8-27B: base model of PPLX v1.2. Shaded bold: best, underlined: second best of the seven models. rows: blue/red for higher/lower means.
| Jev 1.13 | PPLX v1.2 | Clef | Clef-flash | Luna Dec. | Kev | Luna Gen. | |
| Default (B0) | 86.37 | 88.09 | 85.97 | 72.72 | 85.93 | 77.11 | 87.15 |
| Earlier study, general simple (GS) | 85.37 | 86.81 | 83.92 | 69.85 | 80.59 | 77.98 | 83.57 |
| Earlier study, general complex (GC) | 85.46 | 87.41 | 84.63 | 70.75 | 81.29 | 77.00 | 83.75 |
| Earlier study, domain simple (DS) | 84.99 | 86.17 | 82.79 | 67.69 | 81.75 | 75.28 | 84.51 |
| Earlier study, domain complex (DC) | 84.92 | 86.64 | 84.11 | 66.36 | 81.31 | 70.66 | 84.90 |
| Narayan simple (NS) | 86.15 | 87.99 | 85.17 | 73.08 | 83.02 | 77.12 | 85.48 |
Table 5. F1 (%) of the twelve prompts averaged over the eight benchmarks, one run. Shaded bold: best, underlined: second best per model. rows: blue/red for gain/loss. † : significant (paired bootstrap, 95%).
| Jev 1.13 | PPLX v1.2 | Clef | Clef-flash | Luna Dec. | Kev | Luna Gen. | |
| Abt-Buy | 91.60 AR ( +0.61 ) | 94.76 AR ( +0.76 ) | 92.38 C ( +1.52 ) | 87.16 AR ( +3.82 ) | 93.93 AR ( +2.89 ) | 87.56 B0 (0.00) | 94.89 B0 (0.00) |
| Walmart-Amazon | 88.49 C ( +1.38 ) | 90.44 B0 (0.00) | 88.02 B0 (0.00) | 74.53 AR ( +8.95 ) | 85.25 C ( +0.67 ) | 74.47 GS ( +0.10 ) | 90.68 D ( +0.05 ) |
| WDC Products | 87.14 E ( +1.91 ) | 87.86 AR ( +0.70 ) | 85.77 D ( +0.29 ) | 78.36 AR ( +6.19 ) | 86.96 AR ( +0.42 ) | 76.76 D ( +0.36 ) | 88.82 DC ( +0.43 ) |
| DBLP-ACM | 98.11 AR ( +0.96 ) | 98.33 NS ( +0.21 ) | 97.66 AR ( +0.42 ) | 95.47 GC ( +3.42 ) | 98.00 D ( +1.40 ) | 84.24 GC ( +7.62 ) | 98.22 NS ( +0.97 ) |
| DBLP-Scholar | 92.78 AR ( +0.87 ) | 93.63 GC ( +1.03 ) | 92.50 DS ( +2.04 ) | 92.84 D ( +2.63 ) | 92.33 AR ( +2.21 ) | 90.07 NS ( +1.06 ) | 91.10 AR ( +2.55 ) |
| Dn7 Walmart-Amazon | 77.81 AR ( +2.66 ) | 77.69 D ( +0.98 ) | 75.47 AR ( +1.11 ) | 47.38 AR ( +6.98 ) | 75.26 C ( +0.58 ) | 68.97 GS ( +0.52 ) | 77.78 B0 (0.00) |
Table 6. Selected prompt per model and benchmark: F1 (%), prompt code (Table 5 ), and difference to B0 in parentheses (pp, blue/red: gain/loss). Ties go to the prompt listed first. Bold/underlined: best/second-best mean.
| Serialization | Jev 1.13 | PPLX v1.2 | Clef | Clef-flash | Luna Dec. | Kev | Luna Gen. |
|---|---|---|---|---|---|---|---|
| F0 (F1) | 88.06 | 89.13 | 87.70 | 77.78 | 87.00 | 78.72 | 88.92 |
| Values only | 2.30 † (1.5) | 1.43 † ( 1.1 ) | 3.51 † (2.1) | 0.06 (3.2) | 1.95 † (1.6) | 3.11 † (4.3) | 1.81 † (1.7) |
| JSON | 0.66 † (0.6) | 0.31 ( 0.3 ) | 0.90 † (1.0) | 1.46 † (1.9) | 1.52 † (1.3) | 4.21 † (3.2) | 0.05 (1.0) |
| Attribute pairs | 3.32 † (1.8) | 1.39 † ( 1.0 ) | 2.61 † (1.9) | 0.70 (3.5) | 4.49 † (2.4) | 7.23 † (6.1) | 1.95 † (1.4) |
| Omit missing values | 0.41 (0.5) | 0.10 ( 0.2 ) | 0.64 † (0.5) | 0.31 (0.7) | 0.53 † (0.6) | +0.38 (0.7) | 0.09 (0.8) |
| Mean over variants | 1.67 (1.1) | 0.81 ( 0.7 ) | 1.91 (1.4) | 0.63 (2.3) | 2.12 (1.5) | 3.54 (3.6) | 0.97 (1.2) |
Table 7. Input serialization: F1 difference to F0 (pp), averaged over the eight benchmarks (blue/red: gain/loss), and flip rate (%) in parentheses. Bold: lowest flip rate per row. † : significant (paired bootstrap, 95%).
| Perturbation | Jev 1.13 | PPLX v1.2 | Clef | Clef-flash | Luna Dec. | Kev | Luna Gen. |
|---|---|---|---|---|---|---|---|
| F0 (F1) | 88.06 | 89.13 | 87.70 | 77.78 | 87.00 | 78.72 | 88.92 |
| Entry swap | 1.38 † (1.5) | 0.08 ( 0.8 ) | 0.65 (1.5) | 0.48 (2.6) | 0.58 (1.5) | 0.56 (3.7) | 0.82 † (1.4) |
| Reverse attribute order | 0.44 (0.9) | +0.14 ( 0.7 ) | 0.17 (1.1) | +0.33 (2.0) | 0.51 (1.3) | 1.64 † (3.3) | 0.38 (1.2) |
| Lowercase values ∗ | 0.48 (0.8) | 0.17 ( 0.4 ) | 0.52 (1.0) | +0.60 (2.3) | 0.53 (1.2) | 0.43 (1.6) | +0.05 (1.2) |
| Attribute deletion | 3.40 † (1.8) | 2.70 † ( 1.3 ) | 3.62 † (2.0) | 2.59 † (2.9) | 3.65 † (2.0) | 3.78 † (3.7) | 3.53 † (2.0) |
| Title typos | 1.08 † (0.7) | 0.49 † ( 0.5 ) | 1.22 † (1.2) | 0.43 (1.6) | 1.64 † (1.3) | 0.60 † (1.5) | 1.27 † (1.2) |
Table 8. Perturbations: F1 difference to F0 (pp), averaged over the eight benchmarks (blue/red: gain/loss), and flip rate (%) in parentheses. Bold: lowest flip rate per row. † : significant (paired bootstrap, 95%). ∗ Four benchmarks not lowercase already.
| Model | Brier | REL | RES | Mid (%) |
|---|---|---|---|---|
| PPLX v1.2 | 1.68 | 0.10 | 7.56 | 8.4 |
| Clef | 1.88 | 0.15 | 7.41 | 10.7 |
| Jev 1.13 | 1.94 | 0.22 | 7.42 | 14.1 |
| Luna Dec. | 2.12 | 0.34 | 7.32 | 7.9 |
| Kev | 3.68 | 0.70 | 6.16 | 19.4 |
| Clef-flash | 4.54 | 1.89 | 6.48 | 36.1 |
Table 9. Calibration with F0, scores . Bold/underl.: best/second, arrows: direction. Mid: share with .
Figure 2. Reliability diagrams with the selected prompt, ten bins over the 30,225 pooled pairs. Dashed: perfect calibration. Lower panels: pairs per bin, log scale. Six reliability diagrams, one per decision model, plot the share of matches against the predicted match probability in ten bins, with a dashed diagonal for perfect calibration and a log-scale histogram of pairs per bin below each diagram.
| Jev 1.13 | PPLX v1.2 | Clef | Clef-flash | Luna Dec. | Kev | |||||||
| Benchmark | Per bench. | Per model | Per bench. | Per model | Per bench. | Per model | Per bench. | Per model | Per bench. | Per model | Per bench. | Per model |
| Abt-Buy | 0.20 | 0.00 | 0.39 | +0.48 | +0.11 | +0.11 | 1.58 | 4.59 | +0.64 | +0.97 | +1.38 | +0.62 |
| Walmart-Amazon | 0.19 | 0.00 | 0.41 | 0.57 | +0.45 | 1.06 | +11.03 | +10.73 | +3.00 | +2.41 | +7.64 | +6.64 |
| WDC Products | 0.30 | 0.00 | 0.62 | 0.73 | 0.30 | 0.11 | +1.24 | +1.00 | +0.52 | +0.40 | +2.07 | +1.71 |
| DBLP-ACM | 0.27 | 0.00 | +0.77 | +0.11 | +0.28 | 0.21 | +0.40 | 3.06 | +0.42 | +0.21 | 0.55 | +0.06 |
| DBLP-Scholar | +0.18 | 0.00 | 0.03 | 0.37 | 0.10 | 0.51 | 0.03 | 2.50 | 0.05 | 0.85 | 0.28 | 2.20 |
Table 10. Test F1 difference (pp) to with the selected prompt when is tuned on validation, either per benchmark or once per model for all benchmarks. Test-optimal: tuned on test, an upper bound. Blue/red: gain/loss. Bold: largest mean gain.
| Configuration | Abt-Buy | Walmart | WDC | DBLP-ACM | DBLP-Sch. | Dn7 | billiger | Semi-H. | Mean | Errors | Cost (USD) | |
| Jev 1.13 : , 11.7 % deferred, selective risk 0.28 % | ||||||||||||
| Alone | 91.60 | 88.49 | 87.14 | 98.11 | 92.78 | 77.81 | 73.06 | 95.48 | 88.06 | – | 701 | 0.97 |
| + Sol | 95.94 | 94.18 | 91.24 | 97.91 | 92.83 | 79.79 | 78.22 | 95.51 | 90.70 | +2.64 | 582 | 5.24 |
| + Ensemble | 95.96 | 93.43 | 91.11 | 98.01 | 93.65 | 81.18 | 79.19 | 95.51 | 91.01 | +0.31 | 545 | 25.90 |
| PPLX v1.2 : , 10.3 % deferred, selective risk 0.28 % | ||||||||||||
| Alone | 94.76 | 90.44 | 87.86 | 98.33 | 93.63 | 77.69 | 74.49 | 95.82 | 89.13 | – | 641 | 0.27 |
Table 11. Cascade at a target risk of 0.5%: F1 (%) of each decision model alone, with Sol, and with the ensemble, and of the ensemble members alone. : gain over the row above (blue/red). Shaded bold: best mean per group. Cost at list prices.
| Best trained | Best zero-shot LLM | PPLX v1.2 (ours) | ||||
| Benchmark | F1 | Method | F1 | Method | Alone | Cascade |
| Abt-Buy | 94.9 | Qwen3-8B cross-encoder ( 50 ) | 93.81 | Kimi K2.6 ( 38 ) | 94.76 ( 0.14 ) | 96.44 ( +1.54 ) |
| Walmart-Amazon | 91.6 | Qwen3-4B cross-encoder ( 50 ) | 90.54 | Kimi K2.6 ( 38 ) | 90.44 ( 1.16 ) | 93.43 ( +1.83 ) |
| WDC Products | 83.9 | Qwen3-8B generative ( 50 ) | 87.20 | Kimi K2.6 ( 38 ) | 87.86 ( +0.66 ) | 91.53 ( +4.33 ) |
| DBLP-ACM | 99.21 | EMTransformer-B ( 28 ) | 97.44 | GPT-4 ( 48 ) | 98.33 ( 0.88 ) | 98.01 ( 1.20 ) |
| DBLP-Scholar | 98.51 | Jellyfish-13B † ( 48 ) | 92.49 | Kimi K2.6 ( 38 ) | 93.63 ( 4.88 ) | 93.68 ( 4.83 ) |
Table 12. Best published F1 on our test splits and PPLX v1.2 (selected prompt), alone and in the cascade at 0.5% target risk. In parentheses: gap to the best published F1 (blue/red: above/below). Bold: highest per row. † Instruction-tuned on training data of this benchmark.
| Tokens | USD | USD per 1M pairs | vs. | ||
| Model | per pair | test sets | B0 | F0 | PPLX |
| Jev 1.13 | 760 | 0.97 | 24.1 | 31.9 | 3.6 |
| PPLX v1.2 | 445 | 0.27 | 7.1 | 8.9 | 1.0 |
| Clef | 527 | 3.82 | 96.4 | 126.5 | 14 |
| Clef-flash | 585 | 0.67 | 15.3 | 22.2 | 2.5 |
| Luna Dec. | 502 | 1.52 | 41.2 | 50.2 | 5.7 |
Table 13. Cost with B0 and F0 at list prices; cascades at 0.5% target risk. vs. PPLX: factor to the cheapest model (bold). ∗ B0.
| Model | Pairs/s | 1k pairs | 1M pairs | Time vs. | |
|---|---|---|---|---|---|
| Ditto | Luna Gen. | ||||
| Jev 1.13 | 26.9 | 37.2 s | 10.3 h | 14.1 | 0.56 |
| PPLX v1.2 | 21.7 | 46.0 s | 12.8 h | 17.4 | 0.69 |
| Clef | 22.4 | 44.7 s | 12.4 h | 16.9 | 0.67 |
| Clef-flash | 24.6 | 40.6 s | 11.3 h | 15.4 | 0.61 |
| Luna Dec. | 26.6 | 37.6 s | 10.4 h | 14.2 | 0.56 |
Table 14. Throughput, 32 parallel requests, projected to 1M pairs. Bold: fastest per block. † One A40 GPU.
| Class | Name | Jev 1.13 | PPLX v1.2 | Sol | Opus | Argon | Range | |
|---|---|---|---|---|---|---|---|---|
| Errors (FP / FN) | 482 / 219 | 432 / 209 | 465 / 123 | 370 / 182 | 393 / 151 | |||
| C1 | Attribute value discrepancy within the same product type | 36.5 | 37.4 | 32.7 | 30.8 | 32.0 | 6.6 | 0.91 |
| C2 | Product type, role, or offer scope discrepancy | 9.6 | 10.8 | 10.5 | 11.2 | 11.4 | 1.8 | 0.90 |
| C3 | Brand or catalog category discrepancy | 14.4 | 13.4 | 12.2 | 11.1 | 13.4 | 3.3 | 0.98 |
| C4 | Publication title, author, or venue discrepancy | 25.4 | 24.0 | 29.3 | 29.9 | 27.6 | 5.9 | 1.00 |
| C5 | Missing, misplaced, incomparable, or noisy information | 96.6 | 96.1 | 96.3 | 97.1 | 96.9 | 1.0 | 0.95 |
Table 15. Error classes of all test errors of Jev 1.13, PPLX v1.2 (selected prompts), and the ensemble members (B0): share (%) of the errors of each model, by GPT-6.1 Sol (majority of three runs). Bold: highest share. Range: highest lowest. : Fleiss’ kappa.
More in this topic
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 242 papers