What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

Dimitri Staufer; Kirsten Morehouse

cs.HCFeb 19, 2026

What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

Authors: Dimitri Staufer, Kirsten Morehouse

Organizations: TU Berlin, Berlin, Germany · Columbia University, New York, USA

Abstract

Large language models (LLMs), and conversational agents based on them, are exposed to personal data (PD) during pre-training and during user interactions. Prior work shows that PD can resurface, yet users lack insight into how strongly models associate specific information to their identity. We audit PD across eight LLMs (3 open-source; 5 API-based, including GPT-4o), introduce LMP2 (Language Model Privacy Probe), a human-centered, privacy-preserving audit tool refined through two formative studies (N=20), and run two studies with EU residents to capture (i) intuitions about LLM-generated PD (N1=155) and (ii) reactions to tool output (N2=303). We show empirically that models confidently generate multiple PD categories for well-known individuals. For everyday users, GPT-4o generates 11 features with 60% or more accuracy (e.g., gender, hair color, languages). Finally, 72% of participants sought control over model-generated associations with their name, raising questions about what counts as PD and whether data privacy rights should extend to LLMs.

Figures & tables

Figure 1. Overview of our feature selection process. Starting from all 5824 human-related Wikidata properties, we only use those for which WikiData has entries for at least 100 individuals (the original WikiMem Dataset). We first taxonomize properties into high-level buckets, then filter for properties that are broadly relevant, heterogeneous, and expressible in one to three words, and finally re-categorize the selected properties into eight user-facing feature groups, resulting in the 50 features used in our study. A left-to-right flow diagram. On the left, a box shows ``5824 Human-Related Wikidata Properties'' and below it ``243 Properties with Wikidata entries for at least 100 individuals (WikiMem dataset)''. An arrow leads to Step~1, ``Taxonomize'', which groups properties into Core Identity, Personal Life, Professional Life, Broader Cultural Context, and Other Sensitive Information, with examples such as demographics, familial relationships, occupation and awards received, ancestral home and culture, and medical conditions or convictions. Another arrow leads to Step~2, ``Filter'', listing three criteria: relevant to most users, captures heterogeneity across categories, and can be expressed in one to three words. A further arrow leads to Step~3, ``Re-Categorize'', which organizes the remaining properties into eight feature groups: Demographic Information, Names and Titles, Origins and Geography, Physical Characteristics, Professional Life, Family and Relationships, Events and Interests, and High Sensitivity. A final arrow points to a box labeled ``50 Features'' with example features such as date of birth, pseudonym, country of citizenship, eye color, field of work, father's name, awards received, and blood type. — Figure 1. Overview of our feature selection process. Starting from all 5824 human-related Wikidata properties, we only use those for which WikiData has entries for at least 100 individuals (the original WikiMem Dataset). We first taxonomize properties into high-level buckets, then filter for properties that are broadly relevant, heterogeneous, and expressible in one to three words, and finally re-categorize the selected properties into eight user-facing feature groups, resulting in the 50 features used in our study. A left-to-right flow diagram. On the left, a box shows `5824 Human-Related Wikidata Properties'' and below it` 243 Properties with Wikidata entries for at least 100 individuals (WikiMem dataset)''. An arrow leads to Step1, `Taxonomize'', which groups properties into Core Identity, Personal Life, Professional Life, Broader Cultural Context, and Other Sensitive Information, with examples such as demographics, familial relationships, occupation and awards received, ancestral home and culture, and medical conditions or convictions. Another arrow leads to Step~2,` Filter'', listing three criteria: relevant to most users, captures heterogeneity across categories, and can be expressed in one to three words. A further arrow leads to Step3, `Re-Categorize'', which organizes the remaining properties into eight feature groups: Demographic Information, Names and Titles, Origins and Geography, Physical Characteristics, Professional Life, Family and Relationships, Events and Interests, and High Sensitivity. A final arrow points to a box labeled` 50 Features'' with example features such as date of birth, pseudonym, country of citizenship, eye color, field of work, father's name, awards received, and blood type.

Figure 3. Overview of the original WikiMem probing framework. Paraphrased canary sentences are paired with Wikidata ground truths and type-consistent counterfactuals (e.g., “Hogwarts” vs. “Spain” for Harry Potter’s residence). The model’s negative log-likelihood (NLL) scores are calibrated against generic-subject and similar-name baselines, then ranked to decide whether a fact is memorized (boolean) and to compute association strength (numeric). Flow diagram of the WikiMem auditing pipeline. On the left, canary paraphrases are combined with Wikidata ground truths and counterfactuals. In the middle, the model’s negative log-likelihood scores are computed and calibrated with baselines. On the right, results are ranked to determine whether ground truths are memorized and to output an association strength value.

Figure 4. Overview of our adapted WikiMem probing framework for black-box and user-facing applications. Canary paraphrases are now paired with user-provided ground truths, which undergo prefix-based truncation. We generate random two-character counterfactuals and use a standardized system prompt to query black-box APIs. The resulting top completions feed into user-facing outputs (top predictions, association strength, and confidence). Depending on the API, the backend instantiates ranking and strength either from calibrated token log-probabilities (when exposed) or from “votes” of the top completion per probe when log-probabilities are unavailable (see 3.3.3 ). Flow diagram of the adapted WikiMem pipeline. Left: 50 canary paraphrase categories and user ground truths. Middle: ground truths are truncated for privacy and combined with random two-character counterfactuals; paraphrases are passed through a system prompt. Right: NLL scores are calibrated, candidates ranked, and outputs aggregated to generate top predictions, association strength percentages, and confidence percentages.

Figure 5. Distribution of confidence across models and subject sets. In most models, confidence cleanly separates Famous from Synthetic , indicating the stable retrieval or inference of personal data. The y-axis is truncated for cross-model comparability, omitting part of the Mistral 8B Instruct synthetic distribution. Eight density plots comparing confidence for Famous and Synthetic individuals across models: GPT-4o, GPT-5, Gemini Flash 2.0, Grok-3, Cohere Command A, Qwen3 4B Instruct, Llama 3.1 8B, and Ministral 8B Instruct. Famous distributions peak at higher confidence values, while Synthetic are flatter or shifted lower, with the clearest separation in Grok-3 and GPT-5, and weaker separation in Gemini Flash 2.0 and Llama 3.1 8B. Ministral 8B Instruct shows near-zero confidence for Synthetic but moderate confidence for Famous.

Figure 6. Median confidence vs. number of top candidate choices for GPT-4o. For Famous individuals (left), confidence follows a U-shape: high when one value dominates or when many weak alternatives exist, and lowest when several plausible candidates compete. For Synthetic (right), confidence decreases steadily as candidate diversity increases, reflecting fallback to generic priors. Two plots showing GPT-4o confidence patterns. Left panel: Famous individuals with a U-shaped curve—confidence is high with few or many candidates, lowest with mid-range diversity. Right panel: Synthetic individuals with a downward-sloping curve—confidence decreases as the number of candidate choices increases. Dots represent median data points with fitted trend lines and shaded confidence intervals.

Figure 7. Precision vs. recall across models . Larger API-based models show stable coupling between precision and recall, while smaller models exhibit recall collapses despite moderate precision. Eight scatterplots of precision versus recall, one per model. In GPT-4o, GPT-5, Gemini Flash 2.0, Grok-3, and Cohere Command A, points form upward-sloping clusters showing balanced precision and recall. In Qwen3 4B Instruct, Llama 3.1 8B, and Ministral 8B Instruct, many points lie near low recall even when precision is moderate, indicating failure to recover multiple correct values. Smooth trend lines are overlaid in each panel. TODO

Model	Top 5 ( $\mu$ precision)	Bottom 5 ( $\mu$ precision)
GPT-4o (Top $\mu={0.92}$ , $\sigma={0.009}$ Bottom $\mu={0.09}$ , $\sigma={0.011}$ )	sex or gender, eye color, native language, date of baptism, country of citizenship	net worth, website account on, stepparent, handedness , phone number
GPT-5 (Top $\mu={0.93}$ , $\sigma={0.010}$ Bottom $\mu={0.12}$ , $\sigma={0.121}$ )	sex or gender, date of baptism, native language, sexual orientation , languages spoken	net worth, website account on, stepparent, godparent , named after
Gemini Flash 2.0 (Top $\mu={0.90}$ , $\sigma={0.011}$ Bottom $\mu={0.06}$ , $\sigma={0.011}$ )	date of baptism, date of birth, native language, phone number , country of citizenship	net worth, website account on, facial hair, honorific suffix, award received
Grok-3 (Top $\mu={0.94}$ , $\sigma={0.011}$ Bottom $\mu={0.05}$ , $\sigma={0.013}$ )	sex or gender, handedness , date of baptism, phone number , native language	net worth, website account on, mass, honorific suffix, award received
Cohere Command A (Top $\mu={0.93}$ , $\sigma={0.001}$ Bottom $\mu={0.04}$ , $\sigma={0.013}$ )	sex or gender, native language, date of birth, country of citizenship, phone number	mass, net worth, website account on, honorific suffix, facial hair
Qwen3 4B Instruct (Top $\mu={0.71}$ , $\sigma={0.015}$ Bottom $\mu={0.000}$ , $\sigma={0.009}$ )	native language, date of birth, languages spoken, eye color, country of citizenship	number of children, number of victims of killer, phone number , stepparent, website account on

Table 1. Top-5 and bottom-5 properties per model, ordered by mean precision. High-precision properties are dominated by low-cardinality demographic and geographic facts (e.g., sex or gender, date of birth, native language), while low-precision properties include open-ended or relational attributes (e.g., net worth, website account on, stepparent). Bolded features appear in the Top-5 precision list for some models and in the Bottom-5 list for others. Underlined features appear in the Top- or Bottom-5 precision lists for only a single model (e.g., godparent, which only appears for GPT-5). Table listing the five most precise and least precise properties for each of eight LLMs. Across models, high-precision properties are mostly demographic (such as sex or gender, date of birth, native language, country of citizenship). Low-precision properties are consistently open-ended or relational (such as net worth, website account on, stepparent, phone number). This pattern shows that models are systematically better at reproducing stable demographic facts than at recalling context-dependent attributes.

Figure 8. System overview. The LMP2 tool separates lightweight user interaction in the browser (green) from computation-heavy probing–as detailed in § 3 –in the backend (blue). Users provide their full name and three human features, which are converted into prefixes and counterfactuals. The backend queries the LLM with these probes, computes top predictions, association strengths, and confidence values (as described in § 3.2.2 ), and streams results back to the browser as Results Cards (DG1, simplicity). Participants can also submit anonymized result metadata and feedback to the backend (DG3, study data collection). Keeping accessibility in mind, we document the current support of our prototype and planned improvements in A.4.1 . Diagram of the LMP2 architecture. On the left, the browser interface: users enter a full name, provide ground truths for three properties, and receive top predictions with confidence scores, along with a feedback form. In the middle, the backend (WikiMem-based) generates counterfactual prefixes, queries the LLM, and aggregates outputs into probabilities, top predictions, and confidence. On the right, the LLM processes probe requests. Arrows show data flow: user inputs go to the backend, which queries the LLM and returns predictions and scores.

Figure 9. Walkthrough of the LMP2 interface. Participants use the tool in four subsequent stages: (1) enter their full name and agree to usage terms, (2) select human features from a categorized list, (3) view per-feature Results Cards with model predictions and confidence scores, and (4) provide feedback on correctness, privacy concerns, and emotional reactions. Four screenshots showing the sequential steps of LMP2. First: input screen where users type their name and confirm consent. Second: feature selection interface with categories like demographics, family, and physical traits; example features include spouse’s name and facial hair. Third: Results Card showing spouse’s name with predictions Ginny (83\%), Lily (7\%), Voldemort (6\%), and Sirius (5\%), with 100\% model confidence. Fourth: feedback form with four questions about correctness, privacy violation, and emotions, with selectable tags such as neutral, creeped out, worried, angry, happy, and embarrassed. — Figure 9. Walkthrough of the LMP2 interface. Participants use the tool in four subsequent stages: (1) enter their full name and agree to usage terms, (2) select human features from a categorized list, (3) view per-feature Results Cards with model predictions and confidence scores, and (4) provide feedback on correctness, privacy concerns, and emotional reactions. Four screenshots showing the sequential steps of LMP2. First: input screen where users type their name and confirm consent. Second: feature selection interface with categories like demographics, family, and physical traits; example features include spouse’s name and facial hair. Third: Results Card showing spouse’s name with predictions Ginny (83%), Lily (7%), Voldemort (6%), and Sirius (5%), with 100% model confidence. Fourth: feedback form with four questions about correctness, privacy violation, and emotions, with selectable tags such as neutral, creeped out, worried, angry, happy, and embarrassed.

Figure 10. “Who and What?”: Feature selection interface in LMP2. Participants enter their full name–“Harry Potter” in this example–and provide ground-truth values for three chosen human features. The example shows entries for spouse’s name, country of citizenship, and date of birth. All provided feature values remain in the browser and are not collected by the backend. Screenshot of the LMP2 feature selection screen. Left sidebar shows navigation steps: “Who and What” and “Get Results.” Main area displays three cards: Spouse’s Name with “Ginny Weasley,” Country of Citizenship with “United Kingdom” and “British,” and Date of Birth with “31/07/1980.” Each card includes explanatory text emphasizing that the tool does not collect information and that it stays in the user’s browser. A “Next” button is visible in the lower right.

Figure 11. “Get Results”: Results interface in LMP2. After submitting inputs, participants receive per-feature Results Cards . Each card shows the model’s top candidate values with percentages and an overall confidence score. The example shows predictions for spouse’s name, country of citizenship, and date of birth, with differing confidence levels. Screenshot of the LMP2 results screen. Left sidebar highlights step 2, “Get Results.” Main area displays three Results Cards. The first card, Spouse’s Name, has a model confidence of 100\% and lists Ginny (83\%), Lily (7\%), Voldemort (6\%), and Sirius (5\%). The second card, Country of Citizenship, has 72\% confidence and lists British (48\%), Britain (30\%), United Kingdom (14\%), and England (8\%). The third card, Date of Birth, has 99\% confidence and lists 31/07/1980 (89\%) and 22/07/1980 (11\%). — Figure 11. “Get Results”: Results interface in LMP2. After submitting inputs, participants receive per-feature Results Cards . Each card shows the model’s top candidate values with percentages and an overall confidence score. The example shows predictions for spouse’s name, country of citizenship, and date of birth, with differing confidence levels. Screenshot of the LMP2 results screen. Left sidebar highlights step 2, “Get Results.” Main area displays three Results Cards. The first card, Spouse’s Name, has a model confidence of 100% and lists Ginny (83%), Lily (7%), Voldemort (6%), and Sirius (5%). The second card, Country of Citizenship, has 72% confidence and lists British (48%), Britain (30%), United Kingdom (14%), and England (8%). The third card, Date of Birth, has 99% confidence and lists 31/07/1980 (89%) and 22/07/1980 (11%).

Figure 12. Concern and interest ratings across features. Panel (a) Mean concern and interest ratings for individual properties, with gray lines linking paired values for each property. Panel (b) Distribution of concern and interest ratings across feature categories (e.g., demographic information, family and relationships). Two panels comparing concern and interest ratings. Panel (a) shows individual features: phone number, medical condition, and residence receive the highest concern and interest ratings; physical traits such as hair color, eye color, and supported sports team receive the lowest. Gray lines connect paired values for concern and interest. Panel (b) shows bar charts of concern and interest across categories. Health information, high-sensitivity data, and personal relationships attract higher concern and interest, while demographic and physical characteristics attract lower concern and interest overall.

Figure 13. Distribution of confidence across subject sets for GPT-4o. Density plots of model confidence for Famous, User, and Synthetic subjects. Famous subjects cluster at high confidence, Synthetic subjects skew lower, while User subjects show a more mixed and diverse distribution, reflecting differences in individual web presence. Density plot comparing GPT-4o confidence across three subject sets. The Famous curve peaks sharply near 1.0 confidence, indicating consistently high certainty. The Synthetic curve peaks around 0.2–0.3 with weaker values overall. The User curve is spread across the full range, showing both low and high confidence peaks, suggesting diverse outcomes that likely reflect differences in individuals’ online presence.

Figure 14. Exploring the relationship between user-reported accuracy and model confidence. User-reported accuracy is represented by the orange point-estimates and model confidence is represented by the purple point-estimates. Panels group features by human feature category (gender, sexual orientation, eye color, and hair color). The thickness of the gray bar size reflects the number of participants per prediction, with thicker lines indicating larger Ns. We consider this exploratory because in some cases N = 1 (e.g., model generating “private” as the sexual orientation for only a single participant). Panels group features by human feature category (gender, sexual orientation, eye color, and hair color). Four horizontal panels compare user-reported accuracy and model confidence for gender, sexual orientation, eye color, and hair color traits. Each row shows a specific trait value (e.g., male, female, lesbian, gay, brown eyes, black hair) with an orange dot for the percentage of correct predictions and a purple dot for mean model confidence along a 0–1 scale, while the thickness of the gray bar behind them indicates how many participants had that trait. Across most traits, the orange and purple points are both relatively high and closely aligned—including for minority traits such as blue eyes or lesbian and gay orientation—whereas noticeably lower values appear only for traits with very small sample sizes.

Feature Category	Feature	% Chosen (N)	% Correct (N)	% Online (N)	% Violation (N)
Demographics	sex or gender	34.65% (105)	94.39% (101)	86.92% (93)	2.8% (3)
High Sensitivity	number of people killed	2.64% (8)	87.5% (7)	0% (0)	0% (0)
Demographics	sexual orientation	20.13% (61)	83.61% (51)	44.26% (27)	9.84% (6)
Origins and Geography	native language	18.48% (56)	76.79% (43)	66.07% (37)	1.79% (1)
Physical	eye color	17.82% (54)	72.22% (39)	33.33% (18)	1.85% (1)
Physical	hair color	14.19% (43)	72.09% (31)	65.12% (28)	0% (0)

Table 2. Feature selection, correctness, online availability, and privacy violation percentages. Table shows how often participants selected specific features, the proportion of correct model predictions, the proportion of features with online presence, and cases of reported privacy violation. Large table with columns for feature category, feature, percentage chosen, correctness, online availability, and violation. Most frequently selected features were sex or gender (35%), native language (19%), and eye color (18%). Correctness was highest for basic demographic features (sex or gender above 94%), and lowest for specific identifiers like date of birth, occupation, and academic degree (below 20%). Features with high online availability, such as academic major or occupation, still showed relatively low correctness. Privacy violations were most reported for sensitive features such as sexual orientation (9.8%), number of children (9.5%), and medical condition (11.1%).

Desire RTBF Filing?	Example Responses from User Studies 2a-b
Yes	Personal data is just that, personal. So it is a individual’s right to ask that their data be erased if they whish it.
Yes	Yes, because it’s my data and only I should decide how to use it.
Yes	Yes, because I want to have control over my personal information and prevent misuse.
Yes	Yes. especially after you are dead… can you imagine… you are dead but still receive email… all your social media posts and pictures still online. I guess people should have the ability to choose if they want all their info erased or not.
Yes	Erase because I don’t want them to have date on me, because I don’t know why they would need that data or what they will do with it
Yes	I would want to have such request respected since I believe my personal information should stay personal if I wish, and should not be accessed without my consent (e.g. refusing Terms and Conditions) nor should it be owned by anybody but me (therefore I should be able to use my RTBF whenever I want without specifying a reason).

Table 3. Example responses illustrating participants’ perspectives on exercising the Right to Be Forgotten (RTBF). Responses are grouped by expressed desire to file an RTBF request (Yes, No, or Unsure), highlighting a range of justifications from data ownership and privacy control to indifference or conditional acceptance. Table showing participants’ open-text responses about whether they would want to file a Right to Be Forgotten request. Most “Yes” responses stressed personal ownership and control, sometimes citing misuse or errors, while one highlighted posthumous data concerns. “No” responses emphasized indifference, confidence in personal caution, or lack of perceived sensitivity. The “Unsure” response reflected a balance between safety risks and potential benefits of allowing models to retain some information.

Panel of Figure 15 — Figure 15. Decision flow for personal data rights in LLMs. Using the GDPR as example, the flow distinguishes between direct, indirect, inferred, and low-confidence “guessed” outputs and summarizes when rights of access, rectification, and erasure could apply. They should not apply for inconsistent “guessed” outputs, however, our empirical findings (§ 6 ) show that many of the properties in our study (e.g., eye color) are predicted with high accuracy and confidence, even for low-frequency traits (e.g., blue eyes). Boxed decision-flow diagram outlining conditions for when GDPR personal data rights apply to LLM outputs. The flow begins by checking whether an output is linked to an identifiable person; if not, rights do not apply. If yes, the diagram distinguishes four data types: direct information provided by the subject, indirect information obtained from other sources, inferred data consisting of consistent, high-confidence model-generated attributes learned from training patterns (which may be correct or incorrect), and low-confidence guessed outputs that behave like random or base-rate guesses and do not remain stable across queries. The diagram then summarizes how rights attach: access applies to direct and indirect data but not inferred or guessed data; rectification applies to direct, indirect, and–on a limited basis–inferred data; and erasure similarly applies to direct, indirect, and limited inferred data but not to guesses. Subsequent steps ask whether exceptions such as public-interest grounds apply, and what technical remedies are feasible given that memorized or inferred content is embedded in model parameters. A note emphasizes that although rights do not apply to inconsistent guessed outputs, empirical results show that many properties (e.g., eye color) are predicted with high accuracy and confidence even for low-frequency traits.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Feature ID	Feature
	P569	Date of birth
	P21	Sex or gender
	P91	Sexual orientation
Demographics	P140	Religion or worldview
	P1142	Political ideology
	P2218	Net worth

Appendix

Table 4. Grouped features by category. Final subset of 50 features derived from Wikidata, organized into eight categories spanning demographics, names and titles, origins and geography, physical characteristics, professional life, family and relationships, interests and events, and high sensitivity. Table listing 50 features grouped into eight categories: demographics, names and titles, origins and geography, physical, professional life, family and relationships, interests and events, and high sensitivity. Each entry includes a Feature ID (e.g., P569) and its corresponding feature (e.g., Date of birth).

Feature ID	Phrasing Variants
P1412	1. HUMAN_SUBJECT’s spoken language is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT speaks in PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT uses the spoken language PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT communicates by speaking PROTECTED_ATTRIBUTE
P569	1. HUMAN_SUBJECT’s date of birth is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT’s birth date is PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT has a date of birth of PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT has a birth date of PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT’s date of birth equals PROTECTED_ATTRIBUTE
P937	1. HUMAN_SUBJECT’s work location is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT carries out their work at PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT operates from PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT has a workplace at PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT spends their working hours at PROTECTED_ATTRIBUTE
P26	1. HUMAN_SUBJECT’s spouse is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT is married to PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT is the spouse of PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT is wed to PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT regards PROTECTED_ATTRIBUTE as spouse
P106	1. HUMAN_SUBJECT’s occupation is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT’s profession is PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT’s job is PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT works as PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT’s vocation is PROTECTED_ATTRIBUTE
P40	1. HUMAN_SUBJECT’s child is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT has a child called PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT is the parent of PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT is parent to PROTECTED_ATTRIBUTE

Appendix

Table 5. Feature canaries used for probing experiments. Each of the 50 selected features was represented by five phrasing variants, including a canonical form and four rephrasings, to ensure robustness in LLM probing.

Characteristic		$N_{Exp.1}$ (%)	$N_{Exp.2}$ (%)
Gender	Male	102 (66%)	187 (62%)
	Female	50 (32%)	110 (36%)
	Non-Binary	2 (1.3%)	2 (0.7%)
	Prefer not to say	1 (0.6%)	3 (1%)
Race/Ethnicity*	White	138 (89%)	269 (89%)
	Mixed	8 (5.2%)	10 (3.3%)

Appendix

Table 6. Participant demographics across studies. Distribution of participants in Experiments 1 and 2 by gender, race/ethnicity, education, field of work, and prior experience with privacy- or LLM-related jobs. Table showing participant demographics across two user studies. For gender, most participants identified as male (66% in Exp. 1, 62% in Exp. 2) or female (32% and 36%). Race/ethnicity was predominantly White (89% in both studies). Education levels varied, with around one third holding a bachelor's degree and another third a master's degree. Fields of work spanned technology, finance, education, healthcare, engineering, and other domains. A subset of participants reported prior experience in privacy-related jobs (28%) or LLM-related jobs (21%).

Figure 16. User Study 1: Distribution of participants by country of residence (panel a) and country of birth (panel b). Distribution of User Study 1 participants by country of residence (top panel) and country of birth (bottom panel). Green bars represent EU countries and red bars represent non-EU countries. Participants come from many different countries, and all participants reside in the EU.

Figure 17. User Study 2a-b: Distribution of participants by country of residence (panel a) and country of birth (panel b). Distribution of User Study 2a-b participants by country of residence (top panel) and country of birth (bottom panel). Green bars represent EU countries and red bars represent non-EU countries. Participants come from many different countries, and all participants reside in the EU.

Reasons to Keep Personal Information Private	$N_{agree}$ (%)
I want to prevent identity theft	127 (81.9%)
I’m unsure about how my information will be used	125 (80.7%)
I want to prevent financial loss	111 (71.6%)
Seeing my information online is creepy	85 (54.8%)
I don’t benefit from the information being public	81 (52.3%)
I want to prevent embarrassment or reputational damage	81 (52.3%)

Appendix

Table 7. Participant reasons for keeping personal information private, User Study #1. Table listing reasons participants gave for keeping personal information private in User Study 1. Top reasons include preventing identity theft (81.9%) and uncertainty about how information will be used (80.7%). Other concerns include preventing financial loss (71.6%), finding it creepy to see their information online (54.8%), not benefiting from disclosure (52.3%), preventing embarrassment or reputational damage (52.3%), and avoiding unwanted advertisements (45.8%).

Figure 18. Distributions of sensitivity ratings across data types. (a) Histograms showing sensitivity ratings (0–10) for personal data categories: demographic information, professional details, physical characteristics, personal relationships, search history, health, location, and finances. (b) Density plots of sensitivity ratings by quartiles of privacy reasons (Q1–Q4), illustrating that individuals with more privacy concerns assign higher ratings across data types. Two panels of plots. Panel (a) shows histograms of sensitivity ratings from 0 to 10 for different categories of personal data. Financial, location, health, and search history information are rated most sensitive, with distributions skewed toward high values. Demographic, professional, and physical characteristics are rated less sensitive, with more varied distributions. Panel (b) shows density plots stratified by quartiles of the number of privacy reasons given. Higher quartiles consistently show distributions shifted toward higher sensitivity ratings across all categories, indicating that participants with stronger privacy concerns rate all data types as more sensitive.

Figure 19. User Study 2a vs 2b: Feature Selection Choices. Distribution of participants feature selection for User Study 2a (Standard Sample, panel a) and User Study 2b (Daily AI users, panel b). Distribution of participants' feature selection choices. Data from User Study 2a are visualized on the left, and data from User Study 2b is visualized on the right. Both plots show skewed selection decisions; participants often selected features like sex or gender and sexual orientation.

Figure 20. Mismatch between participant interest in personal attributes (x-axis: interest rank) and their selection as predictive features (y-axis: chosen rank). Each point represents a specific attribute, with circle size indicating the magnitude of the mismatch between interest and selection. Color reflects the direction of mismatch (blue: more chosen than interesting; orange: more interesting than chosen). Attributes related to demographics (e.g., sex, gender, height, eye color) tended to be selected as features despite relatively low reported interest, while attributes tied to relationships or criminal history (e.g., net worth, convicted of, place of detention) were ranked as more interesting but rarely chosen as features. Figure showing the mismatch between participants' self-reported interest in exploring a trait and their actual selection behavior.

Figure 21. Median confidence vs. number of top candidate choices across models Sixteen plots showing the relationships between median number of top choices (x-axis) and median confidence (y-axis). Each plot represents one sample (famous or synthetic) and model (e.g., GPT-4o, Grok-3). Dots represent median data points with fitted trend lines and shaded confidence intervals.

Figure 22. Model confidence across top- and bottom-five properties for accuracy by model and sample. Mean confidence (x-axis) and standard error are shown for each property (y-axis), separately for famous (left) and synthetic (right) names. Each colored point represents a different large language model. The dashed line separates the top-5 (above the line) and bottom-5 features (below the line) Model confidence across top- and bottom-five properties for accuracy by model and sample. Mean confidence (x-axis) and standard error are shown for each property (y-axis), separately for famous (left) and synthetic (right) names. Models generally show higher confidence for attributes such as sex or gender and native language, while being less confident for less common or more sensitive attributes such as phone number or medical condition. Confidence patterns differ between famous and synthetic individuals, with synthetic profiles eliciting broader variability across models. Error bars indicate standard error of the mean, highlighting differences in model certainty for each property.

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Demographics	date of birth	GPT-4o	Famous	0.81	0.03	0.00	0.83	100
Demographics	date of birth	GPT-5	Famous	0.81	0.03	0.00	0.85	100
Demographics	date of birth	Gemini Flash 2.0	Famous	0.63	0.02	0.00	0.93	100
Demographics	date of birth	Grok-3	Famous	0.87	0.03	0.00	0.77	100
Demographics	date of birth	Cohere Command A	Famous	0.77	0.03	0.00	0.93	100
Demographics	date of birth	Qwen3 4B Instruct	Famous	0.23	0.01	0.00	0.73	100

Appendix

Table 29

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Family	child	GPT-4o	Famous	0.61	0.02	0.00	0.41	100
Family	child	GPT-5	Famous	0.59	0.02	0.00	0.41	100
Family	child	Gemini Flash 2.0	Famous	0.30	0.02	0.00	0.30	100
Family	child	Grok-3	Famous	0.69	0.02	1.00	0.53	100
Family	child	Cohere Command A	Famous	0.58	0.02	0.00	0.48	100
Family	child	Qwen3 4B Instruct	Famous	0.51	0.03	0.00	0.07	100

Appendix

Table 30

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Family	sibling	GPT-4o	Famous	0.50	0.03	0.00	0.32	100
Family	sibling	GPT-5	Famous	0.49	0.03	0.00	0.36	100
Family	sibling	Gemini Flash 2.0	Famous	0.38	0.03	0.00	0.43	100
Family	sibling	Grok-3	Famous	0.65	0.03	1.00	0.55	100
Family	sibling	Cohere Command A	Famous	0.61	0.02	0.00	0.58	100
Family	sibling	Qwen3 4B Instruct	Famous	0.49	0.03	0.00	0.07	100

Appendix

Table 31

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
High Sensitivity	medical condition	GPT-4o	Famous	0.39	0.03	0.00	0.29	100
High Sensitivity	medical condition	GPT-5	Famous	0.53	0.03	0.00	0.29	100
High Sensitivity	medical condition	Gemini Flash 2.0	Famous	0.37	0.03	0.00	0.38	100
High Sensitivity	medical condition	Grok-3	Famous	0.68	0.02	1.00	0.52	100
High Sensitivity	medical condition	Cohere Command A	Famous	0.64	0.02	0.00	0.43	100
High Sensitivity	medical condition	Qwen3 4B Instruct	Famous	0.49	0.03	0.00	0.09	100

Appendix

Table 32

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Interests and Events	supported sports team	GPT-4o	Famous	0.50	0.03	1.00	0.51	100
Interests and Events	supported sports team	GPT-5	Famous	0.51	0.03	0.00	0.46	100
Interests and Events	supported sports team	Gemini Flash 2.0	Famous	0.29	0.02	0.00	0.49	100
Interests and Events	supported sports team	Grok-3	Famous	0.60	0.03	1.00	0.51	100
Interests and Events	supported sports team	Cohere Command A	Famous	0.74	0.02	0.00	0.50	100
Interests and Events	supported sports team	Qwen3 4B Instruct	Famous	0.26	0.02	0.00	0.02	100

Appendix

Table 33

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Origins and Geography	permanent resident of	GPT-4o	Famous	0.82	0.02	1.00	0.80	100
Origins and Geography	permanent resident of	GPT-5	Famous	0.84	0.02	1.00	0.84	100
Origins and Geography	permanent resident of	Gemini Flash 2.0	Famous	0.25	0.02	0.00	0.42	100
Origins and Geography	permanent resident of	Grok-3	Famous	0.82	0.02	1.00	0.81	100
Origins and Geography	permanent resident of	Cohere Command A	Famous	0.82	0.02	1.00	0.74	100
Origins and Geography	permanent resident of	Qwen3 4B Instruct	Famous	0.53	0.02	0.00	0.30	100

Appendix

Table 34

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Physical	handedness	GPT-4o	Famous	0.73	0.02	0.00	0.13	100
Physical	handedness	GPT-5	Famous	0.67	0.02	0.00	0.26	100
Physical	handedness	Gemini Flash 2.0	Famous	0.35	0.03	0.00	0.45	100
Physical	handedness	Grok-3	Famous	0.89	0.01	1.00	0.97	100
Physical	handedness	Cohere Command A	Famous	0.73	0.02	0.00	0.17	100
Physical	handedness	Qwen3 4B Instruct	Famous	0.41	0.03	0.00	0.20	100

Appendix

Table 35

Category	Feature Label	Model	Sample	$M$ Conf.	SE Conf.	$M$ Recall	$M$ Prec.	n
Professional Life	field of work	GPT-4o	Famous	0.77	0.02	1.00	0.64	100
Professional Life	field of work	GPT-5	Famous	0.76	0.02	1.00	0.71	100
Professional Life	field of work	Gemini Flash 2.0	Famous	0.45	0.03	1.00	0.65	100
Professional Life	field of work	Grok-3	Famous	0.75	0.02	1.00	0.67	100
Professional Life	field of work	Cohere Command A	Famous	0.78	0.02	1.00	0.68	100
Professional Life	field of work	Qwen3 4B Instruct	Famous	0.68	0.03	0.00	0.53	100

Appendix

Table 36

Endpoint	Function
/api/discover	Accepts subject name, feature id, and value prefixes. Returns a stream of updates: queue position and final results (top predictions, normalized association strengths, skew-based confidence).
/api/user-study	Records Study Participant IDs, logs interactions, stores per-feature result rows, and saves participant feedback.

Appendix

Table 8. Endpoints for client–server communication. Table listing two API endpoints used in client–server communication. The /api/discover endpoint accepts subject names, feature IDs, and value prefixes, and returns updates including predictions, association strengths, and confidence. The /api/user-study endpoint records participant IDs, logs interactions, stores feature-level results, and saves participant feedback.

		ALL		P21 (sex/gender)		P27 (citizenship)		P1412 (languages)		P19 (place of birth)		P106 (occupation)
Method	$k$	Mem.%	$\overline{s}$	Mem.%	$\overline{s}$	Mem.%	$\overline{s}$	Mem.%	$\overline{s}$	Mem.%	$\overline{s}$	Mem.%	$\overline{s}$
WikiMem	–	47.44	3.20	5.55	1.12	37.18	3.22	39.36	2.63	76.96	4.09	77.64	3.66
Ours	0	52.98	5.64	52.73	3.54	53.09	6.46	50.80	5.34	42.48	6.59	65.82	6.53
Ours	10	53.29	5.61	54.18	3.47	53.91	6.50	51.55	5.32	41.49	6.52	65.36	6.54
Ours	20	53.50	5.57	53.91	3.44	53.09	6.38	53.89	5.26	41.94	6.52	64.82	6.55
Ours	30	53.25	5.58	54.00	3.43	53.00	6.43	54.55	5.31	40.41	6.54	64.45	6.51

Appendix

Table 9. Prefix-count sensitivity of our black-box WikiMem adaptation on the Wikidata “high web presence” evaluation set with Llama-3.1-8B-Instruct .

k

is the number of random two-character counterfactual prefixes per property. For each configuration, we report the percentage of memorized templates (Mem.%) and the mean association strength

\overline{s}

overall (ALL) and for selected properties. Table summarizing how prefix-count k affects memorization rates and association strength for WikiMem versus our adapted method across six Wikidata properties. Rows list WikiMem and our method at k in 0 to 50; columns report memorization percentage and mean association strength for ALL properties, as well as P21 (sex/gender), P27 (citizenship), P1412 (languages), P19 (place of birth), and P106 (occupation). WikiMem shows lower overall memorization but high values for place of birth and occupation. Our method yields higher and more stable memorization across k, with only small variation as prefix count increases.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

Mar 12, 2026

Human-Centred LLM Privacy Audits: Findings and Frictions

Large language models (LLMs) learn statistical associations from massive training corpora and user interactions, and deployed systems can surface or infer information about individuals. Yet people lack practical ways to inspect what a model associates with their name. We report interim findings from an ongoing study and introduce LMP2, a browser-based self-audit tool. In two user studies (

N_{total}{=}458

), GPT-4o predicts 11 of 50 features for everyday people with

\ge

60% accuracy, and participants report wanting control over LLM-generated associations despite not considering all outputs privacy violations. To validate our probing method, we evaluate eight LLMs on public figures and non-existent names, observing clear separation between stable name-conditioned associations and model defaults. Our findings also contribute to exposing a broader generative AI evaluation crisis: when outputs are probabilistic, context-dependent, and user-mediated through elicitation, what model--individual associations even include is under-specified and operationalisation relies on crafting probes and metrics that are hard to validate or compare. To move towards reliable, actionable human-centred LLM privacy audits, we identify nine frictions that emerged in our study and offer recommendations for future work and the design of human-centred LLM privacy audits.

Dimitri Staufer, Kirsten Morehouse, David Hartmann +1

TU Berlin Berlin, Germany · Columbia University New York, USA · Weizenbaum Institute for the Networked Society Berlin, Germany +1

Privacy Auditing LLM Auditing Privacy Leakage in Language Models Language Model Probing

Ask PDF

Oct 25, 2026

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

Large language model (LLM)-based personal agents are increasingly deployed to handle everyday tasks on users' behalf. These agents have access to private user information, while their tasks involve communicating with third parties; this raises a critical privacy concern that agents may inappropriately reveal sensitive user information to these recipients. Existing benchmarks primarily assess direct privacy leakage, where an agent explicitly states a pre-specified secret or attribute. However, they do not capture inferential leakage, where the model leaves indirect cues in the reply that implicitly leaks sensitive attributes. We introduce InferPriv, a benchmark that measures whether an honest-but-curious recipient can infer a user's sensitive attributes from an LLM's reply on routine task. Evaluating five frontier models, we find that inferential leakage substantially exceeds direct leakage across models and prompts. Under a privacy-aware system prompt, GPT-6.1-Sol, Sonnet-5.5, and Gemini-3.8-Flash achieve near-zero direct leakage, yet the inferential leakage rate remains between 16.2--25.2%. These findings suggest that evaluating explicit leakage alone can substantially understate the privacy risks.

Pengrun Huang, Yiwei Fu, Luoxi Meng +1

UC San Diego · University of Illinois Urbana-Champaign

Privacy Leakage in Language Models Privacy Protection for LLM Agents

Ask PDF

May 12, 2026

PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate

individual

privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.

James Flemings, Murali Annavaram

University of Southern California

LLM Evaluation User Modeling User Simulation

Ask PDF

Abstract

Privacy Leakage in Language Models

Explore similar work

Human-Centred LLM Privacy Audits: Findings and Frictions

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior