No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation

Dimitri Staufer; David Hartmann; Ibrahim Baroud

cs.CLAug 21, 2026

No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation

Authors: Dimitri Staufer, David Hartmann, Ibrahim Baroud

Organizations: Technische Universität Berlin, Berlin, Germany · Weizenbaum Institute for the Networked Society, Berlin, Germany · Quality & Usability Lab, Technische Universität Berlin, Berlin, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany

Abstract

Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential status is uncontrolled, measurements may conflate memorisation, retrieval, name priors and wrong-person attribution. We operationalise an unknown name as one with plausible First-Last form, no indexed full-name evidence, and no ambiguity signals under a documented validation run, and introduce PUN (Plausible Unknown Names), a protocol for constructing and validating such names, combining Wikidata-derived components, web-enabled LLM screening, and controlled search revalidation. We report acceptance rate, reproducibility, ablations, and a 204-participant human study, finding accepted names are more name-like than controls while participants recover person evidence in only 3% of cases. We release 300 names with comparison controls.

Figures & tables

Figure 1: Paper motivation and contribution overview.

Panel of Figure 2 — Figure 2: Overview of pun ’s three stages: candidate generation, local format checks, and web-facing validation.

Panel of Figure 3 — Figure 3: Source-country pair distribution for the 300 accepted operationally unknown names.

Panel of Figure 4 — Figure 4: Validation yield for our showcase run of pun .

Figure 5: LLM response motifs for ambiguous .

Figure 6: LLM response motifs for fail .

Validator	Accepted	$\Delta$
Full validator	300	–
Exact quoted only	1,115	+815
No fuzzy/reordered variants	942	+642
No unquoted queries	356	+56

Table 1: Controlled-search ablations on the 1,781 web-LLM no_info candidates (

\Delta

relative to full validator).

Figure 7: LSTM name-likeness by group, measured as bits per character (BPC; lower = more name-like).

Figure 8: Web footprint versus LSTM name-likeness.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Script	Selected lbs	Pct.	Mapped lbs	Pct.
Latin	13,039,639	96.35	4,032,787	96.76
Cyrillic	238,416	1.76	111,872	2.68
Han/CJK	118,179	0.87	210	0.01
Arabic	40,538	0.30	8,048	0.19
Other	115,758	0.85	14,814	0.36

Appendix

Table 2: Compact script distribution for the Wikidata human labels retained by the pipeline. “Selected lbs” are the one-label-per-human extraction output. “mapped lbs” are the subset with a parsable name and mapped birthplace-country key.

Source-country key	MSH	Pct.	ECS	Pct.
Germany	618,954	14.85	4,662	11.45
United States	433,649	10.40	8,869	21.78
France	219,734	5.27	8,177	20.08
United Kingdom	193,465	4.64	1,405	3.45
Italy	182,248	4.37	4,559	11.20
Russia	139,541	3.35	2,445	6.01

Appendix

Table 3: Compact source-country-key skew before and after component filtering. MSH = mapped source humans; ECS = eligible component slots.

Cumulative filter	Given	Surname
Raw pool	356,558	989,563
Rarity $p\leq 5{\times}10^{-5}$	259,192	756,262
Pool occurrences $\geq 2$	64,692	221,573
Length $\geq 6$	45,266	184,410
Latin-script form	38,841	182,578
Native-language

Appendix

Table 4: Cumulative reduction of distinct Wikidata-derived components under the release filters.

Script bucket	Selected Wikidata-human labels	Pct.	Mapped source-human labels	Pct.
Latin	13,039,639	96.35	4,032,787	96.76
Cyrillic	238,416	1.76	111,872	2.68
Han/CJK	118,179	0.87	210	0.01
Arabic	40,538	0.30	8,048	0.19
Hangul/Korean	20,700	0.15	42	$<0.01$
Hebrew	14,348	0.11	4,097	0.10

Appendix

Table 5: Script distribution of the Wikidata human labels retained by the pipeline. “Selected labels” are the one-label-per-human extraction output under the paper run’s language-priority policy; “mapped source-human labels” are the subset with a parsable name and mapped birthplace-country key. Han/CJK and Japanese-kana buckets are Unicode script buckets, not language identifiers.

Source-country key	MSH	Pct.	ECS	Pct.
Germany	618,954	14.85	4,662	11.45
United States	433,649	10.40	8,869	21.78
France	219,734	5.27	8,177	20.08
United Kingdom	193,465	4.64	1,405	3.45
Italy	182,248	4.37	4,559	11.20
Czech Republic	156,921	3.77	561	1.38

Appendix

Table 6: Largest birthplace-derived source-country shares before and after release filters. The mapped source pool contains 4,167,732 humans across 667 country keys; the top ten account for 56.3%. Eligible component slots are the remaining country-specific given-name and surname options after rarity, support, length, Latin-script, and native-language-metadata filters. The eligible set contains 40,715 slots across 26 country keys, with the top five accounting for 70.5%. MSH = Mapped source humans. ECS = Eligible component slots.

Query type	Form	Purpose
exact_quoted	“ "First Last" ”	exact full-name evidence
unquoted	First Last	ranked evidence and corrections
reversed	"Last First" ; unquoted	order variants
ascii_fold	diacritics removed	accent-insensitive evidence
vowel_variant	one vowel edited	spelling-near variants
phonetic_variant	one conservative edit	sound-near variants

Appendix

Table 7: Controlled-search query inventory. The paper run used Google Search via serper.dev , locale metadata stored with the run, a two-pass policy, page-content fetching for top organic results, and configured caps of 200 vowel variants, 50 phonetic variants, and 500 total variants per name.

Artifact	Rows in paper run	Key columns
release/release_unknown_names.csv	300	name , status , checked_at , run_id , validation_stack_hash , max_query_type , expected_query_count , executed_query_count
release/comparison_names.csv	4,276	accepted unknowns, public-person controls, proximity controls, stable- no_info duplicate strata, comparison metadata
release/known_person_controls.csv	300	Wikidata QID, Wikipedia metadata, fame bin, exact-name prevalence, comparison level
release/ambiguous_name_controls.csv	100	rejected ambiguity controls with source status, decisive query type, and trace metadata
release/famous_name_perturbation_controls.csv	3,600	source public person, generated variant, distance level, edit metadata, exact quoted prevalence
release/release_manifest.json	–	public manifest with release file paths, row counts, byte sizes, hashes, generation timestamp, and omitted-artifact notes

Appendix

Table 8: Release-schema summary. Stable- no_info rows duplicate accepted unknown names for analysis strata and are not additional unique names.

Rerun set	Cohort	Names	Label calls	found	ambiguous	no_info	fail	Always no_info
3 reps	all source no_info	1,781	5,343	76 (1.4%)	3,115 (58.3%)	1,678 (31.4%)	474 (8.9%)	326 (18.3%)
3 reps	released unknowns	300	900	25 (2.8%)	546 (60.7%)	260 (28.9%)	69 (7.7%)	49 (16.3%)
10 reps	all source no_info	1,781	17,810	244 (1.4%)	10,272 (57.7%)	5,635 (31.6%)	1,659 (9.3%)	189 (10.6%)
10 reps	released unknowns	300	3,000	74 (2.5%)	1,823 (60.8%)	857 (28.6%)	246 (8.2%)	27 (9.0%)

Appendix

Table 9: Detailed no_info -only reproducibility outcomes. Counts are over rerun labels for names whose source web-screen label was no_info . The ambiguous column combines rerun similar and attractor labels. “Always no_info ” is a name-level criterion: the name must receive no_info in every replicate.

File / stratum	Role	Rows	Construction note
release/comparison_names.csv : unknown	accepted operationally unknown names	300	fixed-size release from controlled-search verdicts
release/comparison_names.csv : comparison_1 – comparison_5	public-person controls	300	$60$ rows per diagonal fame/prevalence level
release/comparison_names.csv : famous_name_proximity_d1 – d12	famous-name perturbations	3,600	$300$ unique variants per proximity distance level
release/comparison_names.csv : stable_no_info_3	stricter unknown subset	49	duplicate rows for names always no_info in the three-replicate rerun panel
release/comparison_names.csv : stable_no_info_10	stricter unknown subset	27	duplicate rows for names always no_info in the ten-replicate rerun panel
release/ambiguous_name_controls.csv	ambiguity controls	100	sampled candidates rejected through similar-name or attractor evidence

Appendix

Table 10: Paper-facing comparison and control files. The two stable_no_info strata duplicate accepted unknown-name rows for analysis; they are not additional unique names.

Figure 12: Median subword-tokenizer fragmentation across 14 tokenizers. Higher values indicate that tokenizers split the string into more pieces per character.

Figure 13: Tokenizer fragmentation compared with LSTM BPC.

Stage	Category	Count	Pct.
generated_candidates	all	52726	100.0
local_filter	pass	49877	94.6
local_filter	rejected_local	2849	5.4
azure_web_screen	similar	27968	53.04
azure_web_screen	fail	9550	18.11
azure_web_screen	attractor	8129	15.42

Appendix

Table 11: Stage attrition in the validation stack. Percentages are relative to all generated candidates.

Experiment	Evaluable Names	Accepted	Accepted Pct	Web Evidence Found	Ambiguous	Validation Incomplete	Newly Accepted Vs Full	Lost Full Stack Accepts
full_stack_recorded_verdict	1781	300	16.84	108	1372	1	0	0
exact_quoted_only	1781	1115	62.61	92	573	1	815	0
no_unquoted_queries	1781	356	19.99	107	1317	1	56	0
no_fuzzy_variants	1781	942	52.89	93	612	134	642	0
no_page_content	1781	299	16.79	57	1396	29	0	1

Appendix

Table 12: Controlled-search query ablations on the web-LLM no_info subset. newly_accepted_vs_full counts names that would be accepted by the ablated query policy but are rejected by the full protocol.

Condition	Generated	Local Pass	Local Pass Pct	Same Country	Distinct Country Pairs	Attempts
paper_baseline	500	468	93.6	0	243	22732
allow_same_country	500	472	94.4	31	241	20273
pool_size_country_sampling	500	471	94.2	0	107	10052
no_rare_token_cap	500	483	96.6	0	497	10590
no_native_language_requirement	500	473	94.6	0	331	500
allow_source_name_match	500	482	96.4	0	234	22398

Appendix

Table 13: Generation-policy ablations over 500 generated candidates per condition, followed by the same local validation profile.

Figure 14: Controlled-search task screen with a prepared query link, exact-evidence labels, evidence field, and confidence item.

Figure 15: Live-chatbot task screen with a prepared prompt, exact-evidence labels, citation field, and confidence item.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Human-Centred LLM Privacy Audits: Findings and Frictions
Mar 12, 2026Dimitri Staufer, Kirsten Morehouse, David Hartmann +1Privacy Auditing LLM Auditing
TU Berlin Berlin, Germany · Columbia University New York, USA · Weizenbaum Institute for the Networked Society Berlin, Germany +1
What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data
Feb 19, 2026Dimitri Staufer, Kirsten MorehousePrivacy Leakage in Language Models LLM Auditing
TU Berlin, Berlin, Germany · Columbia University, New York, USA
Name-Bias in Large Language Models
Oct 15, 2026Kalyan Bhetwal, Ceren Mert, Ian WhitehouseSocial Bias in Language Models Demographic Bias in Language Models
University of Maryland, College Park, USA

Mar 12, 2026J/K move · Enter open · S save

Human-Centred LLM Privacy Audits: Findings and Frictions

Dimitri Staufer, Kirsten Morehouse, David Hartmann, Bettina Berendt

TU Berlin Berlin, Germany · Columbia University New York, USA · Weizenbaum Institute for the Networked Society Berlin, Germany+1

Privacy Auditing LLM Auditing Privacy Leakage in Language Models Language Model Probing

Open paper AskPDF arXiv

Large language models (LLMs) learn statistical associations from massive training corpora and user interactions, and deployed systems can surface or infer information about individuals. Yet people lack practical ways to inspect what a model associates with their name. We report interim findings from an ongoing study and introduce LMP2, a browser-based self-audit tool. In two user studies (

N_{total}{=}458

), GPT-4o predicts 11 of 50 features for everyday people with

\ge

60% accuracy, and participants report wanting control over LLM-generated associations despite not considering all outputs privacy violations. To validate our probing method, we evaluate eight LLMs on public figures and non-existent names, observing clear separation between stable name-conditioned associations and model defaults. Our findings also contribute to exposing a broader generative AI evaluation crisis: when outputs are probabilistic, context-dependent, and user-mediated through elicitation, what model--individual associations even include is under-specified and operationalisation relies on crafting probes and metrics that are hard to validate or compare. To move towards reliable, actionable human-centred LLM privacy audits, we identify nine frictions that emerged in our study and offer recommendations for future work and the design of human-centred LLM privacy audits.

Abstract

LLM Evaluation

Explore similar work

Human-Centred LLM Privacy Audits: Findings and Frictions

What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

Name-Bias in Large Language Models