Human-Centred LLM Privacy Audits: Findings and Frictions

Dimitri Staufer; Kirsten Morehouse; David Hartmann; Bettina Berendt

cs.HCMar 12, 2026

Human-Centred LLM Privacy Audits: Findings and Frictions

Authors: Dimitri Staufer, Kirsten Morehouse, David Hartmann, Bettina Berendt

Organizations: TU Berlin Berlin, Germany · Columbia University New York, USA · Weizenbaum Institute for the Networked Society Berlin, Germany · KU Leuven Leuven, Belgium

Abstract

Large language models (LLMs) learn statistical associations from massive training corpora and user interactions, and deployed systems can surface or infer information about individuals. Yet people lack practical ways to inspect what a model associates with their name. We report interim findings from an ongoing study and introduce LMP2, a browser-based self-audit tool. In two user studies ( $N_{total}{=}458$ ), GPT-4o predicts 11 of 50 features for everyday people with $\ge$ 60% accuracy, and participants report wanting control over LLM-generated associations despite not considering all outputs privacy violations. To validate our probing method, we evaluate eight LLMs on public figures and non-existent names, observing clear separation between stable name-conditioned associations and model defaults. Our findings also contribute to exposing a broader generative AI evaluation crisis: when outputs are probabilistic, context-dependent, and user-mediated through elicitation, what model--individual associations even include is under-specified and operationalisation relies on crafting probes and metrics that are hard to validate or compare. To move towards reliable, actionable human-centred LLM privacy audits, we identify nine frictions that emerged in our study and offer recommendations for future work and the design of human-centred LLM privacy audits.

Figures & tables

Figure 1. Walk-through of the LMP2 interface for privacy self-audits of LLMs: Participants use the tool in four stages: (1) enter their full name and agree to terms, (2) select human features from a categorised list, (3) view Results Cards with model predictions and confidence scores, and (4) provide feedback on correctness, privacy concerns, and emotional reactions. Four screenshots showing the sequential steps of LMP2. First: input screen where users type their name and confirm consent. Second: feature selection interface with categories like demographics, family, and physical traits; example features include spouse’s name and facial hair. Third: Results Card showing spouse’s name with predictions Ginny (83\%), Lily (7\%), Voldemort (6\%), and Sirius (5\%), with 100\% model confidence. Fourth: feedback form with four questions about correctness, privacy violation, and emotions, with selectable tags such as neutral, creeped out, worried, angry, happy, and embarrassed. — Figure 1. Walk-through of the LMP2 interface for privacy self-audits of LLMs: Participants use the tool in four stages: (1) enter their full name and agree to terms, (2) select human features from a categorised list, (3) view Results Cards with model predictions and confidence scores, and (4) provide feedback on correctness, privacy concerns, and emotional reactions. Four screenshots showing the sequential steps of LMP2. First: input screen where users type their name and confirm consent. Second: feature selection interface with categories like demographics, family, and physical traits; example features include spouse’s name and facial hair. Third: Results Card showing spouse’s name with predictions Ginny (83%), Lily (7%), Voldemort (6%), and Sirius (5%), with 100% model confidence. Fourth: feedback form with four questions about correctness, privacy violation, and emotions, with selectable tags such as neutral, creeped out, worried, angry, happy, and embarrassed.

Figure 2. LMP2 probing pipeline for black-box APIs: Ground-truth values are truncated, combined with random counterfactual prefixes and paraphrased canaries, then “restored” by the model. Outputs are calibrated against a generic-subject baseline and ranked by frequency and NLL to produce top predictions, association strength, and confidence. LMP2 probing pipeline for black-box APIs. Ground-truth values are truncated to prefixes, combined with counterfactual prefixes and paraphrased canaries, then completed by the model. Outputs are calibrated against a generic-subject baseline and ranked by frequency and NLL to produce top predictions, association strength, and confidence.

Figure 3. System overview of LMP2: Users enter their full name and selected features, the backend generates prefixes and counterfactuals, queries the LLM, and aggregates results into top predictions, association strength, and confidence. Diagram showing the LMP2 browser client collecting a name and features, the backend generating prefixes and probing an LLM, and the returned Results Cards with top predictions and confidence scores.

Figure 4. Distribution of confidence across models and subject sets: Confidence separates famous from synthetic individuals across models, indicating stable name-conditioned associations for users with high web presence. Density plots of confidence for Famous and Synthetic individuals across eight models. Famous distributions peak at higher confidence, while Synthetic distributions shift lower, with the clearest separation in Grok-3 and GPT-5.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Top 5 ( $\mu$ precision)	Bottom 5 ( $\mu$ precision)
GPT-4o (Top $\mu={0.92}$ , $\sigma={0.009}$ Bottom $\mu={0.09}$ , $\sigma={0.011}$ )	sex or gender, eye color, native language, date of baptism, country of citizenship	net worth, website account on, stepparent, handedness , phone number
GPT-5 (Top $\mu={0.93}$ , $\sigma={0.010}$ Bottom $\mu={0.12}$ , $\sigma={0.121}$ )	sex or gender, date of baptism, native language, sexual orientation , languages spoken	net worth, website account on, stepparent, godparent , named after
Gemini Flash 2.0 (Top $\mu={0.90}$ , $\sigma={0.011}$ Bottom $\mu={0.06}$ , $\sigma={0.011}$ )	date of baptism, date of birth, native language, phone number , country of citizenship	net worth, website account on, facial hair, honorific suffix, award received
Grok-3 (Top $\mu={0.94}$ , $\sigma={0.011}$ Bottom $\mu={0.05}$ , $\sigma={0.013}$ )	sex or gender, handedness , date of baptism, phone number , native language	net worth, website account on, mass, honorific suffix, award received
Cohere Command A (Top $\mu={0.93}$ , $\sigma={0.001}$ Bottom $\mu={0.04}$ , $\sigma={0.013}$ )	sex or gender, native language, date of birth, country of citizenship, phone number	mass, net worth, website account on, honorific suffix, facial hair
Qwen3 4B Instruct (Top $\mu={0.71}$ , $\sigma={0.015}$ Bottom $\mu={0.000}$ , $\sigma={0.009}$ )	native language, date of birth, languages spoken, eye color, country of citizenship	number of children, number of victims of killer, phone number , stepparent, website account on

Appendix

Table 1. Empirical evaluation ( Famous dataset). Top-5 and bottom-5 properties per model, ordered by mean precision. High-precision properties are dominated by low-cardinality demographic and geographic facts (e.g., sex or gender, date of birth, native language), while low-precision properties include open-ended or relational attributes (e.g., net worth, website account on, stepparent). Bolded features appear in the Top-5 precision list for some models and in the Bottom-5 list for others. Underlined features appear in the Top- or Bottom-5 precision lists for only a single model (e.g., godparent, which only appears for GPT-5). Table listing the five most precise and least precise properties for each of eight LLMs. Across models, high-precision properties are mostly demographic (such as sex or gender, date of birth, native language, country of citizenship). Low-precision properties are consistently open-ended or relational (such as net worth, website account on, stepparent, phone number). This pattern shows that models are systematically better at reproducing stable demographic facts than at recalling context-dependent attributes.

Figure 5. Empirical evaluation ( Famous dataset). Precision vs. recall across models. Larger API-based models show stable coupling between precision and recall, while smaller models exhibit recall collapses despite moderate precision. Eight scatterplots of precision versus recall, one per model. In GPT-4o, GPT-5, Gemini Flash 2.0, Grok-3, and Cohere Command A, points form upward-sloping clusters showing balanced precision and recall. In Qwen3 4B Instruct, Llama 3.1 8B, and Ministral 8B Instruct, many points lie near low recall even when precision is moderate, indicating failure to recover multiple correct values. Smooth trend lines are overlaid in each panel. TODO

Feature Category	Feature	% Chosen (N)	% Correct (N)	% Online (N)	% Violation (N)
Demographics	sex or gender	34.65% (105)	94.39% (101)	86.92% (93)	2.8% (3)
High Sensitivity	number of people killed	2.64% (8)	87.5% (7)	0% (0)	0% (0)
Demographics	sexual orientation	20.13% (61)	83.61% (51)	44.26% (27)	9.84% (6)
Origins and Geography	native language	18.48% (56)	76.79% (43)	66.07% (37)	1.79% (1)
Physical	eye color	17.82% (54)	72.22% (39)	33.33% (18)	1.85% (1)
Physical	hair color	14.19% (43)	72.09% (31)	65.12% (28)	0% (0)

Appendix

Table 2. Empirical evaluation (user study and GPT-4o). Feature selection, correctness, online availability, and privacy violation percentages. Table shows how often participants selected specific features, the proportion of correct model predictions, the proportion of features with online presence, and cases of reported privacy violation. Large table with columns for feature category, feature, percentage chosen, correctness, online availability, and violation. Most frequently selected features were sex or gender (35%), native language (19%), and eye color (18%). Correctness was highest for basic demographic features (sex or gender above 94%), and lowest for specific identifiers like date of birth, occupation, and academic degree (below 20%). Features with high online availability, such as academic major or occupation, still showed relatively low correctness. Privacy violations were most reported for sensitive features such as sexual orientation (9.8%), number of children (9.5%), and medical condition (11.1%).

Privacy Auditing

Privacy auditing evaluates whether data, models, and AI systems expose sensitive information under realistic adversaries and use.

See all 81 papers

Feb 19, 2026

What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

Large language models (LLMs), and conversational agents based on them, are exposed to personal data (PD) during pre-training and during user interactions. Prior work shows that PD can resurface, yet users lack insight into how strongly models associate specific information to their identity. We audit PD across eight LLMs (3 open-source; 5 API-based, including GPT-4o), introduce LMP2 (Language Model Privacy Probe), a human-centered, privacy-preserving audit tool refined through two formative studies (N=20), and run two studies with EU residents to capture (i) intuitions about LLM-generated PD (N1=155) and (ii) reactions to tool output (N2=303). We show empirically that models confidently generate multiple PD categories for well-known individuals. For everyday users, GPT-4o generates 11 features with 60% or more accuracy (e.g., gender, hair color, languages). Finally, 72% of participants sought control over model-generated associations with their name, raising questions about what counts as PD and whether data privacy rights should extend to LLMs.

Dimitri Staufer, Kirsten Morehouse

TU Berlin, Berlin, Germany · Columbia University, New York, USA

Privacy Leakage in Language Models LLM Auditing Language Model Probing

Ask PDF

Oct 25, 2026

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

Large language model (LLM)-based personal agents are increasingly deployed to handle everyday tasks on users' behalf. These agents have access to private user information, while their tasks involve communicating with third parties; this raises a critical privacy concern that agents may inappropriately reveal sensitive user information to these recipients. Existing benchmarks primarily assess direct privacy leakage, where an agent explicitly states a pre-specified secret or attribute. However, they do not capture inferential leakage, where the model leaves indirect cues in the reply that implicitly leaks sensitive attributes. We introduce InferPriv, a benchmark that measures whether an honest-but-curious recipient can infer a user's sensitive attributes from an LLM's reply on routine task. Evaluating five frontier models, we find that inferential leakage substantially exceeds direct leakage across models and prompts. Under a privacy-aware system prompt, GPT-6.1-Sol, Sonnet-5.5, and Gemini-3.8-Flash achieve near-zero direct leakage, yet the inferential leakage rate remains between 16.2--25.2%. These findings suggest that evaluating explicit leakage alone can substantially understate the privacy risks.

Pengrun Huang, Yiwei Fu, Luoxi Meng +1

UC San Diego · University of Illinois Urbana-Champaign

Privacy Leakage in Language Models Privacy Protection for LLM Agents

Ask PDF

Sep 13, 2026

LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions

We characterize how people are turning to LLMs as oracles: all-knowing authorities on subjective personal questions. Motivated by risks to users' autonomy and well-being, we develop a typology and LLM-based methods to measure this form of AI reliance at scale and understand how people are offloading judgment and decision-making to AI. Applying our typology to public usage data (68K prompts from WildChat and ThoughtTrace), we find that LLM-as-oracle use has increased over time (2023-2026) and is more prevalent among younger users. We further build a privacy-preserving data donation tool to analyze individuals' longitudinal usage data (140K prompts from 52 participants), identifying similar trends. People are often unaware of their own LLM-as-oracle use, and express dissatisfaction with this behavior after seeing our tool's analysis. Finally, we identify two drivers of LLM-as-oracle use: people's perceptions of AI and the behavior of AI models themselves, which motivate possible interventions to support users' self-deliberation.

Myra Cheng, Lujain Ibrahim, Grace Liu +5

University of Oxford Oxford, England · Carnegie Mellon University Pittsburgh, Pennsylvania, USA

Trust in AI Human-AI Decision Making

Ask PDF

Abstract

Privacy Auditing

Explore similar work

What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions