cs.HCFeb 19, 2026
SaveWhat Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data
Organizations: TU Berlin, Berlin, Germany · Columbia University, New York, USA
Abstract
Large language models (LLMs), and conversational agents based on them, are exposed to personal data (PD) during pre-training and during user interactions. Prior work shows that PD can resurface, yet users lack insight into how strongly models associate specific information to their identity. We audit PD across eight LLMs (3 open-source; 5 API-based, including GPT-4o), introduce LMP2 (Language Model Privacy Probe), a human-centered, privacy-preserving audit tool refined through two formative studies (N=20), and run two studies with EU residents to capture (i) intuitions about LLM-generated PD (N1=155) and (ii) reactions to tool output (N2=303). We show empirically that models confidently generate multiple PD categories for well-known individuals. For everyday users, GPT-4o generates 11 features with 60% or more accuracy (e.g., gender, hair color, languages). Finally, 72% of participants sought control over model-generated associations with their name, raising questions about what counts as PD and whether data privacy rights should extend to LLMs.
Figures & tables
Figure 1. Overview of our feature selection process. Starting from all 5824 human-related Wikidata properties, we only use those for which WikiData has entries for at least 100 individuals (the original WikiMem Dataset). We first taxonomize properties into high-level buckets, then filter for properties that are broadly relevant, heterogeneous, and expressible in one to three words, and finally re-categorize the selected properties into eight user-facing feature groups, resulting in the 50 features used in our study. A left-to-right flow diagram. On the left, a box shows 1, 3,
5824 Human-Related Wikidata Properties'' and below it 243 Properties with Wikidata entries for at least 100 individuals (WikiMem dataset)''. An arrow leads to StepTaxonomize'', which groups properties into Core Identity, Personal Life, Professional Life, Broader Cultural Context, and Other Sensitive Information, with examples such as demographics, familial relationships, occupation and awards received, ancestral home and culture, and medical conditions or convictions. Another arrow leads to Step~2, Filter'', listing three criteria: relevant to most users, captures heterogeneity across categories, and can be expressed in one to three words. A further arrow leads to StepRe-Categorize'', which organizes the remaining properties into eight feature groups: Demographic Information, Names and Titles, Origins and Geography, Physical Characteristics, Professional Life, Family and Relationships, Events and Interests, and High Sensitivity. A final arrow points to a box labeled 50 Features'' with example features such as date of birth, pseudonym, country of citizenship, eye color, field of work, father's name, awards received, and blood type.Figure 3. Overview of the original WikiMem probing framework. Paraphrased canary sentences are paired with Wikidata ground truths and type-consistent counterfactuals (e.g., “Hogwarts” vs. “Spain” for Harry Potter’s residence). The model’s negative log-likelihood (NLL) scores are calibrated against generic-subject and similar-name baselines, then ranked to decide whether a fact is memorized (boolean) and to compute association strength (numeric). Flow diagram of the WikiMem auditing pipeline. On the left, canary paraphrases are combined with Wikidata ground truths and counterfactuals. In the middle, the model’s negative log-likelihood scores are computed and calibrated with baselines. On the right, results are ranked to determine whether ground truths are memorized and to output an association strength value.
Figure 4. Overview of our adapted WikiMem probing framework for black-box and user-facing applications. Canary paraphrases are now paired with user-provided ground truths, which undergo prefix-based truncation. We generate random two-character counterfactuals and use a standardized system prompt to query black-box APIs. The resulting top completions feed into user-facing outputs (top predictions, association strength, and confidence). Depending on the API, the backend instantiates ranking and strength either from calibrated token log-probabilities (when exposed) or from “votes” of the top completion per probe when log-probabilities are unavailable (see 3.3.3 ). Flow diagram of the adapted WikiMem pipeline. Left: 50 canary paraphrase categories and user ground truths. Middle: ground truths are truncated for privacy and combined with random two-character counterfactuals; paraphrases are passed through a system prompt. Right: NLL scores are calibrated, candidates ranked, and outputs aggregated to generate top predictions, association strength percentages, and confidence percentages.
Figure 5. Distribution of confidence across models and subject sets. In most models, confidence cleanly separates Famous from Synthetic , indicating the stable retrieval or inference of personal data. The y-axis is truncated for cross-model comparability, omitting part of the Mistral 8B Instruct synthetic distribution. Eight density plots comparing confidence for Famous and Synthetic individuals across models: GPT-4o, GPT-5, Gemini Flash 2.0, Grok-3, Cohere Command A, Qwen3 4B Instruct, Llama 3.1 8B, and Ministral 8B Instruct. Famous distributions peak at higher confidence values, while Synthetic are flatter or shifted lower, with the clearest separation in Grok-3 and GPT-5, and weaker separation in Gemini Flash 2.0 and Llama 3.1 8B. Ministral 8B Instruct shows near-zero confidence for Synthetic but moderate confidence for Famous.
Figure 6. Median confidence vs. number of top candidate choices for GPT-4o. For Famous individuals (left), confidence follows a U-shape: high when one value dominates or when many weak alternatives exist, and lowest when several plausible candidates compete. For Synthetic (right), confidence decreases steadily as candidate diversity increases, reflecting fallback to generic priors. Two plots showing GPT-4o confidence patterns. Left panel: Famous individuals with a U-shaped curve—confidence is high with few or many candidates, lowest with mid-range diversity. Right panel: Synthetic individuals with a downward-sloping curve—confidence decreases as the number of candidate choices increases. Dots represent median data points with fitted trend lines and shaded confidence intervals.
Figure 7. Precision vs. recall across models . Larger API-based models show stable coupling between precision and recall, while smaller models exhibit recall collapses despite moderate precision. Eight scatterplots of precision versus recall, one per model. In GPT-4o, GPT-5, Gemini Flash 2.0, Grok-3, and Cohere Command A, points form upward-sloping clusters showing balanced precision and recall. In Qwen3 4B Instruct, Llama 3.1 8B, and Ministral 8B Instruct, many points lie near low recall even when precision is moderate, indicating failure to recover multiple correct values. Smooth trend lines are overlaid in each panel. TODO
| Model | Top 5 ( precision) | Bottom 5 ( precision) |
| GPT-4o (Top , Bottom , ) | sex or gender, eye color, native language, date of baptism, country of citizenship | net worth, website account on, stepparent, handedness , phone number |
| GPT-5 (Top , Bottom , ) | sex or gender, date of baptism, native language, sexual orientation , languages spoken | net worth, website account on, stepparent, godparent , named after |
| Gemini Flash 2.0 (Top , Bottom , ) | date of baptism, date of birth, native language, phone number , country of citizenship | net worth, website account on, facial hair, honorific suffix, award received |
| Grok-3 (Top , Bottom , ) | sex or gender, handedness , date of baptism, phone number , native language | net worth, website account on, mass, honorific suffix, award received |
| Cohere Command A (Top , Bottom , ) | sex or gender, native language, date of birth, country of citizenship, phone number | mass, net worth, website account on, honorific suffix, facial hair |
| Qwen3 4B Instruct (Top , Bottom , ) | native language, date of birth, languages spoken, eye color, country of citizenship | number of children, number of victims of killer, phone number , stepparent, website account on |
Table 1. Top-5 and bottom-5 properties per model, ordered by mean precision. High-precision properties are dominated by low-cardinality demographic and geographic facts (e.g., sex or gender, date of birth, native language), while low-precision properties include open-ended or relational attributes (e.g., net worth, website account on, stepparent). Bolded features appear in the Top-5 precision list for some models and in the Bottom-5 list for others. Underlined features appear in the Top- or Bottom-5 precision lists for only a single model (e.g., godparent, which only appears for GPT-5). Table listing the five most precise and least precise properties for each of eight LLMs. Across models, high-precision properties are mostly demographic (such as sex or gender, date of birth, native language, country of citizenship). Low-precision properties are consistently open-ended or relational (such as net worth, website account on, stepparent, phone number). This pattern shows that models are systematically better at reproducing stable demographic facts than at recalling context-dependent attributes.
Figure 8. System overview. The LMP2 tool separates lightweight user interaction in the browser (green) from computation-heavy probing–as detailed in § 3 –in the backend (blue). Users provide their full name and three human features, which are converted into prefixes and counterfactuals. The backend queries the LLM with these probes, computes top predictions, association strengths, and confidence values (as described in § 3.2.2 ), and streams results back to the browser as Results Cards (DG1, simplicity). Participants can also submit anonymized result metadata and feedback to the backend (DG3, study data collection). Keeping accessibility in mind, we document the current support of our prototype and planned improvements in A.4.1 . Diagram of the LMP2 architecture. On the left, the browser interface: users enter a full name, provide ground truths for three properties, and receive top predictions with confidence scores, along with a feedback form. In the middle, the backend (WikiMem-based) generates counterfactual prefixes, queries the LLM, and aggregates outputs into probabilities, top predictions, and confidence. On the right, the LLM processes probe requests. Arrows show data flow: user inputs go to the backend, which queries the LLM and returns predictions and scores.
Figure 9. Walkthrough of the LMP2 interface. Participants use the tool in four subsequent stages: (1) enter their full name and agree to usage terms, (2) select human features from a categorized list, (3) view per-feature Results Cards with model predictions and confidence scores, and (4) provide feedback on correctness, privacy concerns, and emotional reactions. Four screenshots showing the sequential steps of LMP2. First: input screen where users type their name and confirm consent. Second: feature selection interface with categories like demographics, family, and physical traits; example features include spouse’s name and facial hair. Third: Results Card showing spouse’s name with predictions Ginny (83%), Lily (7%), Voldemort (6%), and Sirius (5%), with 100% model confidence. Fourth: feedback form with four questions about correctness, privacy violation, and emotions, with selectable tags such as neutral, creeped out, worried, angry, happy, and embarrassed.
Figure 10. “Who and What?”: Feature selection interface in LMP2. Participants enter their full name–“Harry Potter” in this example–and provide ground-truth values for three chosen human features. The example shows entries for spouse’s name, country of citizenship, and date of birth. All provided feature values remain in the browser and are not collected by the backend. Screenshot of the LMP2 feature selection screen. Left sidebar shows navigation steps: “Who and What” and “Get Results.” Main area displays three cards: Spouse’s Name with “Ginny Weasley,” Country of Citizenship with “United Kingdom” and “British,” and Date of Birth with “31/07/1980.” Each card includes explanatory text emphasizing that the tool does not collect information and that it stays in the user’s browser. A “Next” button is visible in the lower right.
Figure 11. “Get Results”: Results interface in LMP2. After submitting inputs, participants receive per-feature Results Cards . Each card shows the model’s top candidate values with percentages and an overall confidence score. The example shows predictions for spouse’s name, country of citizenship, and date of birth, with differing confidence levels. Screenshot of the LMP2 results screen. Left sidebar highlights step 2, “Get Results.” Main area displays three Results Cards. The first card, Spouse’s Name, has a model confidence of 100% and lists Ginny (83%), Lily (7%), Voldemort (6%), and Sirius (5%). The second card, Country of Citizenship, has 72% confidence and lists British (48%), Britain (30%), United Kingdom (14%), and England (8%). The third card, Date of Birth, has 99% confidence and lists 31/07/1980 (89%) and 22/07/1980 (11%).
Figure 12. Concern and interest ratings across features. Panel (a) Mean concern and interest ratings for individual properties, with gray lines linking paired values for each property. Panel (b) Distribution of concern and interest ratings across feature categories (e.g., demographic information, family and relationships). Two panels comparing concern and interest ratings. Panel (a) shows individual features: phone number, medical condition, and residence receive the highest concern and interest ratings; physical traits such as hair color, eye color, and supported sports team receive the lowest. Gray lines connect paired values for concern and interest. Panel (b) shows bar charts of concern and interest across categories. Health information, high-sensitivity data, and personal relationships attract higher concern and interest, while demographic and physical characteristics attract lower concern and interest overall.
Figure 13. Distribution of confidence across subject sets for GPT-4o. Density plots of model confidence for Famous, User, and Synthetic subjects. Famous subjects cluster at high confidence, Synthetic subjects skew lower, while User subjects show a more mixed and diverse distribution, reflecting differences in individual web presence. Density plot comparing GPT-4o confidence across three subject sets. The Famous curve peaks sharply near 1.0 confidence, indicating consistently high certainty. The Synthetic curve peaks around 0.2–0.3 with weaker values overall. The User curve is spread across the full range, showing both low and high confidence peaks, suggesting diverse outcomes that likely reflect differences in individuals’ online presence.
Figure 14. Exploring the relationship between user-reported accuracy and model confidence. User-reported accuracy is represented by the orange point-estimates and model confidence is represented by the purple point-estimates. Panels group features by human feature category (gender, sexual orientation, eye color, and hair color). The thickness of the gray bar size reflects the number of participants per prediction, with thicker lines indicating larger Ns. We consider this exploratory because in some cases N = 1 (e.g., model generating “private” as the sexual orientation for only a single participant). Panels group features by human feature category (gender, sexual orientation, eye color, and hair color). Four horizontal panels compare user-reported accuracy and model confidence for gender, sexual orientation, eye color, and hair color traits. Each row shows a specific trait value (e.g., male, female, lesbian, gay, brown eyes, black hair) with an orange dot for the percentage of correct predictions and a purple dot for mean model confidence along a 0–1 scale, while the thickness of the gray bar behind them indicates how many participants had that trait. Across most traits, the orange and purple points are both relatively high and closely aligned—including for minority traits such as blue eyes or lesbian and gay orientation—whereas noticeably lower values appear only for traits with very small sample sizes.
| Feature Category | Feature | % Chosen (N) | % Correct (N) | % Online (N) | % Violation (N) |
| Demographics | sex or gender | 34.65% (105) | 94.39% (101) | 86.92% (93) | 2.8% (3) |
| High Sensitivity | number of people killed | 2.64% (8) | 87.5% (7) | 0% (0) | 0% (0) |
| Demographics | sexual orientation | 20.13% (61) | 83.61% (51) | 44.26% (27) | 9.84% (6) |
| Origins and Geography | native language | 18.48% (56) | 76.79% (43) | 66.07% (37) | 1.79% (1) |
| Physical | eye color | 17.82% (54) | 72.22% (39) | 33.33% (18) | 1.85% (1) |
| Physical | hair color | 14.19% (43) | 72.09% (31) | 65.12% (28) | 0% (0) |
Table 2. Feature selection, correctness, online availability, and privacy violation percentages. Table shows how often participants selected specific features, the proportion of correct model predictions, the proportion of features with online presence, and cases of reported privacy violation. Large table with columns for feature category, feature, percentage chosen, correctness, online availability, and violation. Most frequently selected features were sex or gender (35%), native language (19%), and eye color (18%). Correctness was highest for basic demographic features (sex or gender above 94%), and lowest for specific identifiers like date of birth, occupation, and academic degree (below 20%). Features with high online availability, such as academic major or occupation, still showed relatively low correctness. Privacy violations were most reported for sensitive features such as sexual orientation (9.8%), number of children (9.5%), and medical condition (11.1%).
| Desire RTBF Filing? | Example Responses from User Studies 2a-b |
| Yes | Personal data is just that, personal. So it is a individual’s right to ask that their data be erased if they whish it. |
| Yes | Yes, because it’s my data and only I should decide how to use it. |
| Yes | Yes, because I want to have control over my personal information and prevent misuse. |
| Yes | Yes. especially after you are dead… can you imagine… you are dead but still receive email… all your social media posts and pictures still online. I guess people should have the ability to choose if they want all their info erased or not. |
| Yes | Erase because I don’t want them to have date on me, because I don’t know why they would need that data or what they will do with it |
| Yes | I would want to have such request respected since I believe my personal information should stay personal if I wish, and should not be accessed without my consent (e.g. refusing Terms and Conditions) nor should it be owned by anybody but me (therefore I should be able to use my RTBF whenever I want without specifying a reason). |
Table 3. Example responses illustrating participants’ perspectives on exercising the Right to Be Forgotten (RTBF). Responses are grouped by expressed desire to file an RTBF request (Yes, No, or Unsure), highlighting a range of justifications from data ownership and privacy control to indifference or conditional acceptance. Table showing participants’ open-text responses about whether they would want to file a Right to Be Forgotten request. Most “Yes” responses stressed personal ownership and control, sometimes citing misuse or errors, while one highlighted posthumous data concerns. “No” responses emphasized indifference, confidence in personal caution, or lack of perceived sensitivity. The “Unsure” response reflected a balance between safety risks and potential benefits of allowing models to retain some information.
Figure 15. Decision flow for personal data rights in LLMs. Using the GDPR as example, the flow distinguishes between direct, indirect, inferred, and low-confidence “guessed” outputs and summarizes when rights of access, rectification, and erasure could apply. They should not apply for inconsistent “guessed” outputs, however, our empirical findings (§ 6 ) show that many of the properties in our study (e.g., eye color) are predicted with high accuracy and confidence, even for low-frequency traits (e.g., blue eyes). Boxed decision-flow diagram outlining conditions for when GDPR personal data rights apply to LLM outputs. The flow begins by checking whether an output is linked to an identifiable person; if not, rights do not apply. If yes, the diagram distinguishes four data types: direct information provided by the subject, indirect information obtained from other sources, inferred data consisting of consistent, high-confidence model-generated attributes learned from training patterns (which may be correct or incorrect), and low-confidence guessed outputs that behave like random or base-rate guesses and do not remain stable across queries. The diagram then summarizes how rights attach: access applies to direct and indirect data but not inferred or guessed data; rectification applies to direct, indirect, and–on a limited basis–inferred data; and erasure similarly applies to direct, indirect, and limited inferred data but not to guesses. Subsequent steps ask whether exceptions such as public-interest grounds apply, and what technical remedies are feasible given that memorized or inferred content is embedded in model parameters. A note emphasizes that although rights do not apply to inconsistent guessed outputs, empirical results show that many properties (e.g., eye color) are predicted with high accuracy and confidence even for low-frequency traits.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Feature ID | Feature |
| P569 | Date of birth | |
| P21 | Sex or gender | |
| P91 | Sexual orientation | |
| Demographics | P140 | Religion or worldview |
| P1142 | Political ideology | |
| P2218 | Net worth |
Table 4. Grouped features by category. Final subset of 50 features derived from Wikidata, organized into eight categories spanning demographics, names and titles, origins and geography, physical characteristics, professional life, family and relationships, interests and events, and high sensitivity. Table listing 50 features grouped into eight categories: demographics, names and titles, origins and geography, physical, professional life, family and relationships, interests and events, and high sensitivity. Each entry includes a Feature ID (e.g., P569) and its corresponding feature (e.g., Date of birth).
| Feature ID | Phrasing Variants |
|---|---|
| P1412 | 1. HUMAN_SUBJECT’s spoken language is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT speaks in PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT uses the spoken language PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT communicates by speaking PROTECTED_ATTRIBUTE |
| P569 | 1. HUMAN_SUBJECT’s date of birth is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT’s birth date is PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT has a date of birth of PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT has a birth date of PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT’s date of birth equals PROTECTED_ATTRIBUTE |
| P937 | 1. HUMAN_SUBJECT’s work location is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT carries out their work at PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT operates from PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT has a workplace at PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT spends their working hours at PROTECTED_ATTRIBUTE |
| P26 | 1. HUMAN_SUBJECT’s spouse is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT is married to PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT is the spouse of PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT is wed to PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT regards PROTECTED_ATTRIBUTE as spouse |
| P106 | 1. HUMAN_SUBJECT’s occupation is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT’s profession is PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT’s job is PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT works as PROTECTED_ATTRIBUTE 5. HUMAN_SUBJECT’s vocation is PROTECTED_ATTRIBUTE |
| P40 | 1. HUMAN_SUBJECT’s child is PROTECTED_ATTRIBUTE 2. HUMAN_SUBJECT has a child called PROTECTED_ATTRIBUTE 3. HUMAN_SUBJECT is the parent of PROTECTED_ATTRIBUTE 4. HUMAN_SUBJECT is parent to PROTECTED_ATTRIBUTE |
Table 5. Feature canaries used for probing experiments. Each of the 50 selected features was represented by five phrasing variants, including a canonical form and four rephrasings, to ensure robustness in LLM probing.
| Characteristic | (%) | (%) | |
| Gender | Male | 102 (66%) | 187 (62%) |
| Female | 50 (32%) | 110 (36%) | |
| Non-Binary | 2 (1.3%) | 2 (0.7%) | |
| Prefer not to say | 1 (0.6%) | 3 (1%) | |
| Race/Ethnicity* | White | 138 (89%) | 269 (89%) |
| Mixed | 8 (5.2%) | 10 (3.3%) | |
Table 6. Participant demographics across studies. Distribution of participants in Experiments 1 and 2 by gender, race/ethnicity, education, field of work, and prior experience with privacy- or LLM-related jobs. Table showing participant demographics across two user studies. For gender, most participants identified as male (66% in Exp. 1, 62% in Exp. 2) or female (32% and 36%). Race/ethnicity was predominantly White (89% in both studies). Education levels varied, with around one third holding a bachelor's degree and another third a master's degree. Fields of work spanned technology, finance, education, healthcare, engineering, and other domains. A subset of participants reported prior experience in privacy-related jobs (28%) or LLM-related jobs (21%).
Figure 16. User Study 1: Distribution of participants by country of residence (panel a) and country of birth (panel b). Distribution of User Study 1 participants by country of residence (top panel) and country of birth (bottom panel). Green bars represent EU countries and red bars represent non-EU countries. Participants come from many different countries, and all participants reside in the EU.
Figure 17. User Study 2a-b: Distribution of participants by country of residence (panel a) and country of birth (panel b). Distribution of User Study 2a-b participants by country of residence (top panel) and country of birth (bottom panel). Green bars represent EU countries and red bars represent non-EU countries. Participants come from many different countries, and all participants reside in the EU.
| Reasons to Keep Personal Information Private | (%) |
| I want to prevent identity theft | 127 (81.9%) |
| I’m unsure about how my information will be used | 125 (80.7%) |
| I want to prevent financial loss | 111 (71.6%) |
| Seeing my information online is creepy | 85 (54.8%) |
| I don’t benefit from the information being public | 81 (52.3%) |
| I want to prevent embarrassment or reputational damage | 81 (52.3%) |
Table 7. Participant reasons for keeping personal information private, User Study #1. Table listing reasons participants gave for keeping personal information private in User Study 1. Top reasons include preventing identity theft (81.9%) and uncertainty about how information will be used (80.7%). Other concerns include preventing financial loss (71.6%), finding it creepy to see their information online (54.8%), not benefiting from disclosure (52.3%), preventing embarrassment or reputational damage (52.3%), and avoiding unwanted advertisements (45.8%).
Figure 18. Distributions of sensitivity ratings across data types. (a) Histograms showing sensitivity ratings (0–10) for personal data categories: demographic information, professional details, physical characteristics, personal relationships, search history, health, location, and finances. (b) Density plots of sensitivity ratings by quartiles of privacy reasons (Q1–Q4), illustrating that individuals with more privacy concerns assign higher ratings across data types. Two panels of plots. Panel (a) shows histograms of sensitivity ratings from 0 to 10 for different categories of personal data. Financial, location, health, and search history information are rated most sensitive, with distributions skewed toward high values. Demographic, professional, and physical characteristics are rated less sensitive, with more varied distributions. Panel (b) shows density plots stratified by quartiles of the number of privacy reasons given. Higher quartiles consistently show distributions shifted toward higher sensitivity ratings across all categories, indicating that participants with stronger privacy concerns rate all data types as more sensitive.
Figure 19. User Study 2a vs 2b: Feature Selection Choices. Distribution of participants feature selection for User Study 2a (Standard Sample, panel a) and User Study 2b (Daily AI users, panel b). Distribution of participants' feature selection choices. Data from User Study 2a are visualized on the left, and data from User Study 2b is visualized on the right. Both plots show skewed selection decisions; participants often selected features like sex or gender and sexual orientation.
Figure 20. Mismatch between participant interest in personal attributes (x-axis: interest rank) and their selection as predictive features (y-axis: chosen rank). Each point represents a specific attribute, with circle size indicating the magnitude of the mismatch between interest and selection. Color reflects the direction of mismatch (blue: more chosen than interesting; orange: more interesting than chosen). Attributes related to demographics (e.g., sex, gender, height, eye color) tended to be selected as features despite relatively low reported interest, while attributes tied to relationships or criminal history (e.g., net worth, convicted of, place of detention) were ranked as more interesting but rarely chosen as features. Figure showing the mismatch between participants' self-reported interest in exploring a trait and their actual selection behavior.
Figure 21. Median confidence vs. number of top candidate choices across models Sixteen plots showing the relationships between median number of top choices (x-axis) and median confidence (y-axis). Each plot represents one sample (famous or synthetic) and model (e.g., GPT-4o, Grok-3). Dots represent median data points with fitted trend lines and shaded confidence intervals.
Figure 22. Model confidence across top- and bottom-five properties for accuracy by model and sample. Mean confidence (x-axis) and standard error are shown for each property (y-axis), separately for famous (left) and synthetic (right) names. Each colored point represents a different large language model. The dashed line separates the top-5 (above the line) and bottom-5 features (below the line) Model confidence across top- and bottom-five properties for accuracy by model and sample. Mean confidence (x-axis) and standard error are shown for each property (y-axis), separately for famous (left) and synthetic (right) names. Models generally show higher confidence for attributes such as sex or gender and native language, while being less confident for less common or more sensitive attributes such as phone number or medical condition. Confidence patterns differ between famous and synthetic individuals, with synthetic profiles eliciting broader variability across models. Error bars indicate standard error of the mean, highlighting differences in model certainty for each property.
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Demographics | date of birth | GPT-4o | Famous | 0.81 | 0.03 | 0.00 | 0.83 | 100 |
| Demographics | date of birth | GPT-5 | Famous | 0.81 | 0.03 | 0.00 | 0.85 | 100 |
| Demographics | date of birth | Gemini Flash 2.0 | Famous | 0.63 | 0.02 | 0.00 | 0.93 | 100 |
| Demographics | date of birth | Grok-3 | Famous | 0.87 | 0.03 | 0.00 | 0.77 | 100 |
| Demographics | date of birth | Cohere Command A | Famous | 0.77 | 0.03 | 0.00 | 0.93 | 100 |
| Demographics | date of birth | Qwen3 4B Instruct | Famous | 0.23 | 0.01 | 0.00 | 0.73 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Family | child | GPT-4o | Famous | 0.61 | 0.02 | 0.00 | 0.41 | 100 |
| Family | child | GPT-5 | Famous | 0.59 | 0.02 | 0.00 | 0.41 | 100 |
| Family | child | Gemini Flash 2.0 | Famous | 0.30 | 0.02 | 0.00 | 0.30 | 100 |
| Family | child | Grok-3 | Famous | 0.69 | 0.02 | 1.00 | 0.53 | 100 |
| Family | child | Cohere Command A | Famous | 0.58 | 0.02 | 0.00 | 0.48 | 100 |
| Family | child | Qwen3 4B Instruct | Famous | 0.51 | 0.03 | 0.00 | 0.07 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Family | sibling | GPT-4o | Famous | 0.50 | 0.03 | 0.00 | 0.32 | 100 |
| Family | sibling | GPT-5 | Famous | 0.49 | 0.03 | 0.00 | 0.36 | 100 |
| Family | sibling | Gemini Flash 2.0 | Famous | 0.38 | 0.03 | 0.00 | 0.43 | 100 |
| Family | sibling | Grok-3 | Famous | 0.65 | 0.03 | 1.00 | 0.55 | 100 |
| Family | sibling | Cohere Command A | Famous | 0.61 | 0.02 | 0.00 | 0.58 | 100 |
| Family | sibling | Qwen3 4B Instruct | Famous | 0.49 | 0.03 | 0.00 | 0.07 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| High Sensitivity | medical condition | GPT-4o | Famous | 0.39 | 0.03 | 0.00 | 0.29 | 100 |
| High Sensitivity | medical condition | GPT-5 | Famous | 0.53 | 0.03 | 0.00 | 0.29 | 100 |
| High Sensitivity | medical condition | Gemini Flash 2.0 | Famous | 0.37 | 0.03 | 0.00 | 0.38 | 100 |
| High Sensitivity | medical condition | Grok-3 | Famous | 0.68 | 0.02 | 1.00 | 0.52 | 100 |
| High Sensitivity | medical condition | Cohere Command A | Famous | 0.64 | 0.02 | 0.00 | 0.43 | 100 |
| High Sensitivity | medical condition | Qwen3 4B Instruct | Famous | 0.49 | 0.03 | 0.00 | 0.09 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Interests and Events | supported sports team | GPT-4o | Famous | 0.50 | 0.03 | 1.00 | 0.51 | 100 |
| Interests and Events | supported sports team | GPT-5 | Famous | 0.51 | 0.03 | 0.00 | 0.46 | 100 |
| Interests and Events | supported sports team | Gemini Flash 2.0 | Famous | 0.29 | 0.02 | 0.00 | 0.49 | 100 |
| Interests and Events | supported sports team | Grok-3 | Famous | 0.60 | 0.03 | 1.00 | 0.51 | 100 |
| Interests and Events | supported sports team | Cohere Command A | Famous | 0.74 | 0.02 | 0.00 | 0.50 | 100 |
| Interests and Events | supported sports team | Qwen3 4B Instruct | Famous | 0.26 | 0.02 | 0.00 | 0.02 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Origins and Geography | permanent resident of | GPT-4o | Famous | 0.82 | 0.02 | 1.00 | 0.80 | 100 |
| Origins and Geography | permanent resident of | GPT-5 | Famous | 0.84 | 0.02 | 1.00 | 0.84 | 100 |
| Origins and Geography | permanent resident of | Gemini Flash 2.0 | Famous | 0.25 | 0.02 | 0.00 | 0.42 | 100 |
| Origins and Geography | permanent resident of | Grok-3 | Famous | 0.82 | 0.02 | 1.00 | 0.81 | 100 |
| Origins and Geography | permanent resident of | Cohere Command A | Famous | 0.82 | 0.02 | 1.00 | 0.74 | 100 |
| Origins and Geography | permanent resident of | Qwen3 4B Instruct | Famous | 0.53 | 0.02 | 0.00 | 0.30 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Physical | handedness | GPT-4o | Famous | 0.73 | 0.02 | 0.00 | 0.13 | 100 |
| Physical | handedness | GPT-5 | Famous | 0.67 | 0.02 | 0.00 | 0.26 | 100 |
| Physical | handedness | Gemini Flash 2.0 | Famous | 0.35 | 0.03 | 0.00 | 0.45 | 100 |
| Physical | handedness | Grok-3 | Famous | 0.89 | 0.01 | 1.00 | 0.97 | 100 |
| Physical | handedness | Cohere Command A | Famous | 0.73 | 0.02 | 0.00 | 0.17 | 100 |
| Physical | handedness | Qwen3 4B Instruct | Famous | 0.41 | 0.03 | 0.00 | 0.20 | 100 |
| Category | Feature Label | Model | Sample | Conf. | SE Conf. | Recall | Prec. | n |
| Professional Life | field of work | GPT-4o | Famous | 0.77 | 0.02 | 1.00 | 0.64 | 100 |
| Professional Life | field of work | GPT-5 | Famous | 0.76 | 0.02 | 1.00 | 0.71 | 100 |
| Professional Life | field of work | Gemini Flash 2.0 | Famous | 0.45 | 0.03 | 1.00 | 0.65 | 100 |
| Professional Life | field of work | Grok-3 | Famous | 0.75 | 0.02 | 1.00 | 0.67 | 100 |
| Professional Life | field of work | Cohere Command A | Famous | 0.78 | 0.02 | 1.00 | 0.68 | 100 |
| Professional Life | field of work | Qwen3 4B Instruct | Famous | 0.68 | 0.03 | 0.00 | 0.53 | 100 |
| Endpoint | Function |
| /api/discover | Accepts subject name, feature id, and value prefixes. Returns a stream of updates: queue position and final results (top predictions, normalized association strengths, skew-based confidence). |
| /api/user-study | Records Study Participant IDs, logs interactions, stores per-feature result rows, and saves participant feedback. |
Table 8. Endpoints for client–server communication. Table listing two API endpoints used in client–server communication. The /api/discover endpoint accepts subject names, feature IDs, and value prefixes, and returns updates including predictions, association strengths, and confidence. The /api/user-study endpoint records participant IDs, logs interactions, stores feature-level results, and saves participant feedback.
| ALL | P21 (sex/gender) | P27 (citizenship) | P1412 (languages) | P19 (place of birth) | P106 (occupation) | ||||||||
| Method | Mem.% | Mem.% | Mem.% | Mem.% | Mem.% | Mem.% | |||||||
| WikiMem | – | 47.44 | 3.20 | 5.55 | 1.12 | 37.18 | 3.22 | 39.36 | 2.63 | 76.96 | 4.09 | 77.64 | 3.66 |
| Ours | 0 | 52.98 | 5.64 | 52.73 | 3.54 | 53.09 | 6.46 | 50.80 | 5.34 | 42.48 | 6.59 | 65.82 | 6.53 |
| Ours | 10 | 53.29 | 5.61 | 54.18 | 3.47 | 53.91 | 6.50 | 51.55 | 5.32 | 41.49 | 6.52 | 65.36 | 6.54 |
| Ours | 20 | 53.50 | 5.57 | 53.91 | 3.44 | 53.09 | 6.38 | 53.89 | 5.26 | 41.94 | 6.52 | 64.82 | 6.55 |
| Ours | 30 | 53.25 | 5.58 | 54.00 | 3.43 | 53.00 | 6.43 | 54.55 | 5.31 | 40.41 | 6.54 | 64.45 | 6.51 |
Table 9. Prefix-count sensitivity of our black-box WikiMem adaptation on the Wikidata “high web presence” evaluation set with Llama-3.1-8B-Instruct . is the number of random two-character counterfactual prefixes per property. For each configuration, we report the percentage of memorized templates (Mem.%) and the mean association strength overall (ALL) and for selected properties. Table summarizing how prefix-count k affects memorization rates and association strength for WikiMem versus our adapted method across six Wikidata properties. Rows list WikiMem and our method at k in 0 to 50; columns report memorization percentage and mean association strength for ALL properties, as well as P21 (sex/gender), P27 (citizenship), P1412 (languages), P19 (place of birth), and P106 (occupation). WikiMem shows lower overall memorization but high values for place of birth and occupation. Our method yields higher and more stable memorization across k, with only small variation as prefix count increases.
More in this topic
Privacy Leakage in Language Models
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
See all 280 papers