cs.CLOct 13, 2026
SavePre-Trained Language Models as Zero-Shot Tools for Social Science Research
Organizations: PhD Candidate, Department of Government, Harvard University.
Abstract
Record linkage and demographic inference from names are routine demands in social science. Most approaches to these problems require labeled training data that are difficult or costly to acquire. I show that pre-trained language models perform both tasks accurately without requiring task-specific training. For record linkage, I introduce EnsembleLink, an algorithm that uses pre-trained embeddings alongside string-based scorers to retrieve a short list of candidates and then identify the likeliest matches. EnsembleLink matches or exceeds the performance of most prior approaches while generalizing across scripts and formats that those methods are unable to handle. For ethnicity inference from names, I show that large language models (LLMs) achieve up to 83.8% accuracy on U.S. voter files, compared to 78.0% for name-supplemented BISG. Open-weight models offer reproducible implementations that approach these models in accuracy. Moreover, LLMs are able to infer group membership for religious, caste-based, and other ethnic group categories outside of the United States, and reduce bias in downstream estimates of geographic racial gaps. Where pre-training encodes the world knowledge required by each task, task-specific labels add less information.
Figures & tables
Table 1: Examples of a query and its correct match from each application. The North Carolina row is illustrative and shows the format of the voter records. Every other row comes from the test data.
Figure 1: Record linkage F1 (%) on held-out test data, with 95% bootstrap intervals over queries. Raw values are provided in Appendix Table 4 .
| Florida | North Carolina | |||||||||||||
| Method | Gem. | 4o | 4.1-m | DS4 | GLM5 | Q3.5 | BISG | Gem. | 4o | 4.1-m | DS4 | GLM5 | Q3.5 | BISG |
| Panel A: Accuracy by input configuration | ||||||||||||||
| Full name + geo | 83.8 | 78.5 | 70.5 | 76.6 | 74.0 | 74.9 | 78.0 | 83.5 | 80.5 | 74.1 | 80.0 | 76.8 | 78.3 | 78.4 |
| Full name only | 82.8 | 78.2 | 69.7 | 77.0 | 73.8 | 75.0 | — | 82.3 | 79.6 | 72.9 | 78.5 | 75.8 | 76.8 | — |
| Surname + geo | 76.0 | 69.9 | 65.9 | 70.4 | 65.2 | 66.6 | 68.2 | 74.3 | 69.1 | 66.9 | 71.3 | 66.1 | 67.9 | 68.9 |
| Surname only | 69.7 | 64.7 | 58.4 | 69.7 | 62.5 | 62.9 | — | 68.3 | 64.6 | 60.6 | 69.1 | 63.5 | 64.2 | — |
Table 2: Panel A: overall accuracy (%) by input configuration. Panel B: recall (%) by racial category with full name and geography, where the BISG column is name-supplemented BISG. Gem. = Gemini 3.0 Flash; 4o = GPT-4o; 4.1-m = GPT-4.1 mini; DS4 = DeepSeek-V4; GLM5 = GLM-5-Turbo; Q3.5 = Qwen3.5-Plus. BISG ( wru v3.0.3) requires geography and has no counterpart in the name-only rows. Bold marks the best value in each row and state. fBISG and recall for standard BISG are in Appendix B.2.2 .
| Model | FL | NC | AL | SC | BISG | +names |
|---|---|---|---|---|---|---|
| BISG (surname + geography) | 67.4 | 69.5 | 64.6 | 62.8 | ||
| BISG (+ first/middle names) | 77.8 | 79.9 | 75.2 | 70.5 | ||
| Proprietary (API-only) | ||||||
| Gemini 3.8 Flash | 81.2 | 80.3 | 76.0 | 75.0 | ✓ (4) | ✓ (4) |
| Claude Opus 4.8 | 80.7 | 79.7 | 75.6 | 74.6 | ✓ (4) | ✓ (3) |
| Claude Sonnet 4.6 | 80.5 | 79.8 | 74.6 | 72.5 | ✓ (4) | ✓ (2) |
Table 3: Accuracy (%) classifying race from full name and geography, 2,000 voters per state. Each model is run at the lowest reasoning setting accepted by the respective API (Appendix B.3 ). Bold marks the best model in each column. A check mark indicates that the model outperforms standard BISG ( BISG) or name-supplemented BISG ( +names) in at least one state, with the number of states in parentheses. Responses outside the four categories count as incorrect. BISG rows use wru v3.0.3, without and with first- and middle-name supplements, on the same voters. Model identifiers, reasoning settings, and licenses are in Appendix B.3 .
Figure 3: Left, probability calibration on the Florida test set, with expected calibration error per method. Right, discrimination in Florida and North Carolina, with one-versus-rest average precision.
Figure 4: Estimated gap between each minority group and White voters minus the gap measured with true race. Frequent turnout and living in an above-median-income ZCTA are the geographic outcomes, and being a registered Democrat is the non-geographic outcome (Appendix B.9 ).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: F1 on the altered test queries of each simulation condition. Jaro-Winkler averages the similarity of the fields that both records observe. Conditions under “Fields available” combine spelling errors, nicknames, and replaced surnames.
| Change in validation F1 (points) | |||||||
|---|---|---|---|---|---|---|---|
| Configuration | Mean | NC | Names | MaDI | Occupations | Paragraphs | License added |
| Jina v2 + BGE (baseline) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | — |
| Rank-average confidence (adopted) | +1.2 | 0.0 | +1.7 | +1.1 | +0.4 | +1.6 | — |
| Jina v2 + BGE + zerank-2 (not adopted) | +1.1 | 0.0 | +2.4 | +0.6 | +1.5 | +0.4 | Apache-2.0 |
| Jina v2 + BGE + gte-multilingual-reranker | +0.5 | 0.0 | +0.3 | +0.7 | 0.2 | +1.2 | Apache-2.0 |
| Jina v2 + BGE + mxbai-rerank-base-v2 | +0.2 | 0.2 | +0.5 | 0.0 | +0.6 | +0.2 | Apache-2.0 |
Table 14: Change in validation F1 (percentage points) from the baseline configuration (Harrier-OSS-v1 embedding; Jina Reranker v2 and BGE Reranker v2-m3 as rerankers; the margin between the top two candidates as match confidence).
| Retrieval recall | Top-1 | F1 | Extra pairs | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Benchmark | Both | Dense 60 | Dense 30 | Both | Dense 60 | Dense 30 | Both | Dense 60 | Dense 30 | (Dense 60) |
| North Carolina voters | 99.8 | 99.6 | 99.5 | 97.5 | 97.5 | 97.4 | 97.0 | 97.0 | 97.0 | +8% |
| ParaNames | 92.8 | 91.2 | 88.9 | 76.4 | 76.2 | 75.5 | 77.3 | 77.2 | 76.6 | +7% |
| GeoNames | 95.0 | 93.8 | 92.1 | 82.5 | 81.9 | 80.9 | 83.0 | 82.5 | 81.8 | +20% |
| Products join | 99.4 | 99.8 | 99.3 | 66.3 | 67.1 | 67.1 | 66.6 | 67.6 | 67.4 | +48% |
| Occupations | 82.2 | 82.7 | 77.8 | 53.9 | 53.6 | 53.8 | 56.1 | 56.2 | 56.1 | +15% |
Table 16: Retrieval with the dense and sparse candidates combined (“Both”, the default) and with the dense top 60 or top 30 alone, validation data (%). “Retrieval recall” is the share of queries whose true match enters the candidate list. “Extra pairs” is the increase in pairs scored by the rerankers under “Dense 60” relative to “Both”.
| Florida | North Carolina | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Race | Gem. | 4o | 4.1-m | DS4 | GLM5 | Q3.5 | BISG | +nm | Gem. | 4o | 4.1-m | DS4 | GLM5 | Q3.5 | BISG | +nm |
| White | 88.4 | 88.5 | 90.6 | 86.8 | 90.4 | 86.8 | 86.8 | 91.5 | 84.4 | 88.2 | 90.5 | 82.4 | 89.2 | 87.3 | 85.0 | 92.9 |
| Black | 81.7 | 72.2 | 48.9 | 62.8 | 63.0 | 63.0 | 48.8 | 67.0 | 80.0 | 68.3 | 52.1 | 71.4 | 63.3 | 64.1 | 53.1 | 65.2 |
| Hispanic | 92.3 | 91.6 | 91.4 | 92.2 | 90.0 | 92.6 | 88.8 | 93.1 | 84.9 | 85.7 | 83.8 | 84.7 | 81.5 | 85.8 | 77.0 | 81.2 |
| Asian | 72.8 | 61.6 | 51.2 | 64.4 | 52.7 | 57.2 | 48.5 | 60.4 | 84.5 | 79.6 | 70.2 | 81.6 | 73.4 | 76.2 | 60.5 | 74.4 |
| Average | 83.8 | 78.5 | 70.5 | 76.6 | 74.0 | 74.9 | 68.2 | 78.0 | 83.5 | 80.5 | 74.1 | 80.0 | 76.8 | 78.3 | 68.9 | 78.4 |
Table 24: Recall by racial category (%), full name and geography. +nm is BISG with first and middle name supplements ( Imai et al., 2022 ) . Bold indicates the most accurate LLM in each row.
| Florida | North Carolina | |||||||||||||
| Gem. | 4o | 4.1-m | DS4 | GLM5 | Q3.5 | BISG | Gem. | 4o | 4.1-m | DS4 | GLM5 | Q3.5 | BISG | |
| F1 by racial category | ||||||||||||||
| White | 79.8 | 76.6 | 71.6 | 74.5 | 73.6 | 74.2 | 65.2 | 76.4 | 74.7 | 70.1 | 73.0 | 72.1 | 73.0 | 62.2 |
| Black | 83.9 | 78.7 | 63.1 | 72.3 | 72.8 | 72.2 | 60.3 | 80.0 | 74.7 | 63.8 | 73.6 | 71.3 | 71.4 | 58.8 |
| Hispanic | 88.3 | 86.4 | 83.7 | 82.4 | 85.5 | 82.8 | 80.2 | 89.0 | 88.4 | 87.0 | 87.2 | 86.6 | 87.4 | 84.4 |
| Asian | 83.8 | 75.7 | 67.4 | 77.3 | 68.7 | 72.5 | 64.9 | 91.1 | 88.2 | 82.2 | 88.8 | 84.4 | 86.1 | 74.8 |
Table 25: Per-class F1 and macro-averaged precision and F1 (%), full name and geography. BISG uses surname and county.
| Gemini 3.0 Flash | DeepSeek-V4 | GLM-5-Turbo | Qwen3.5-Plus | |||||
|---|---|---|---|---|---|---|---|---|
| Category | Rajasthan | Bihar | Rajasthan | Bihar | Rajasthan | Bihar | Rajasthan | Bihar |
| Scheduled Caste (SC) | 56.0 | 93.3 | 33.3 | 72.0 | 2.7 | 56.0 | 5.3 | 49.3 |
| Scheduled Tribe (ST) | 66.7 | 68.0 | 77.3 | 85.3 | 50.7 | 50.7 | 97.3 | 97.3 |
| Other Backward Class (OBC) | 61.3 | 68.0 | 24.0 | 60.0 | 18.7 | 45.3 | 4.0 | 22.7 |
| General (unreserved) | 46.7 | 72.0 | 21.3 | 38.7 | 86.7 | 82.7 | 2.7 | 37.3 |
| Overall | 57.7 | 75.3 | 39.0 | 64.0 | 39.7 | 58.7 | 27.3 | 51.7 |
Table 28: Accuracy for four-way caste classification (%). Rajasthan: elected village heads ( sarpanch ) with official category labels. Bihar: land records with recorded jati .
Figure 6: One-versus-rest precision and recall by racial category, scoring each person by the first-token probability of the category and by the BISG posterior. The dotted line indicates the share of each category in the sample. Circles mark the most probable category of each model, and diamonds the precision and recall of Gemini 3.0 Flash, which returns no token probabilities.
Figure 7: Reliability diagram (predicted confidence against accuracy) for North Carolina voters, corresponding to the Florida figure in Section 5.5 .
Figure 8: Misclassification rate by ZCTA median household income and race, North Carolina voters. Points are averages within income ventiles, sized by the number of voters, and lines are loess fits.
More in this topic
Entity Resolution
Entity resolution links records that refer to the same real-world entity across datasets despite differences in names, attributes, formats, and languages.
See all 38 papers