NIST's Privacy Risk Assessment Methodology (PRAM) provides a structured framework for privacy experts to assess privacy risks. However, its complexity and reliance on expert knowledge make it difficult for novice developers to use effectively. This paper explores methods to lower these barriers. We first performed an observational study with 12 participants using PRAM in real-world scenarios, and found that novice developers struggled most with articulating privacy-related design decisions. We then developed PrivacyAkinator, an interactive tool that helps developers articulate key privacy decisions by answering LLM-generated multiple-choice questions. PrivacyAkinator introduces three innovations: a universal privacy representation that abstracts privacy-related design decisions into data flows and stakeholder interactions; a domain-aware design space mined from 10K privacy-related news articles; and a dynamic question-generation workflow to prioritize relevant questions. Our user study with 24 participants suggests that developers using PrivacyAkinator identified 47% more key decisions in 73% less time compared to PRAM.
Figures & tables
Figure 1. PrivacyAkinator helps developers articulate key privacy design decisions by answering LLM-generated multiple-choice questions. After (1) developers provide a high-level description of their system, (2) PrivacyAkinator expands it into detailed, editable functional requirements with adaptable design choices highlighted. (3) PrivacyAkinator guides developers through key design decisions with contextualized, specific questions. As developers make choices, PrivacyAkinator organizes and documents these decisions within a structured privacy representation. (4) PrivacyAkinator maps these design decisions to NIST PRAM worksheets ( NIST, 2020 ) to further identify potential issues and prioritize privacy risks. Note that we only enumerate data actions and relevant design decisions; analysts still need to manually evaluate risk likelihood and impact.
#ID
Scenario
Description
Participants
#1
Zoom Attendee Attention Tracking ( Li et al., 2024b ; Amatulli, 2020 )
A feature that monitored attendees’ attention during video conferences by tracking whether Zoom was the active application on a participant’s computer. It showed a clock icon next to inattentive participants and generated a post-meeting report with attention percentage scores.
P1, P2, P3, P8
#2
Alexa’s Smart Home Voice Assistant ( Matt Day and Bloomberg, 2019 )
A cloud-based voice service for smart home devices that allows users to control their home environment through voice commands. It captures audio when activated by a wake word, processes requests in the cloud, and executes actions through connected devices.
P4, P7, P11
#3
Target Retail Recommendation System ( Hill, 2024 )
An analytics system that identified pregnant customers based on purchase patterns of approximately 25 indicator products, allowing Target to predict pregnancy and estimate due dates. This enabled targeted marketing of pregnancy and baby-related products.
A service that collects and analyzes genetic information from customer saliva samples for ancestry, health traits, and disease risks. The service stores comprehensive genetic profiles that can reveal sensitive information about individuals and their biological relatives.
P6, P10
Table 1. Task scenarios used in our study. We selected four task scenarios based on participant familiarity, scenario complexity, and domain diversity. For each scenario, we provided participants with a high-level description of the system. A table listing four scenarios used to study how developers conduct privacy risk assessments. The columns are ID, Scenario (e.g., "Zoom Attendee Attention Tracking"), Description, and Participants (showing which study participants were assigned to each scenario).
Figure 2. Data Flow Diagram defines data actions that occur across the data lifecycle, indicates the key entities involved in the system, and connects the data flows between them. However, the level of detail is insufficient to capture privacy nuances, and it omits interaction with different stakeholders, and may lead to deferred responsibility for privacy. A flowchart illustrating the data lifecycle for a Zoom attention tracking feature. An "Attendee" icon is connected to a "Zoom" icon, which then connects to a "Cloud Storage Provider", and finally to a "Host" icon. Numbered data keys indicate what information is transferred at each step: (1) Attendee Account Information, (2) Active Windows, (3) Aggregated Attention Percentage Score, and (4) Attendance Report. A legend explains the icons for Individual, Data Storage, Computer, Active Windows, Zoom, and Third-Party.
Figure 3. Privacy Storyboard illustrates data actions across the data lifecycle with stakeholder roles. While more aligned with developer workflow, it can overwhelm users with too much information and often conflates multiple design decisions. A flowchart showing four main stages: Collect, Process, Store, and Access, with arrows indicating the flow from "Meeting Participants" to a "Host". Each stage has a box with a brief description. For example, the "Collect" stage "Collects clients' screen activities continuously during host's screen-sharing meeting session." The storyboard illustrates data actions and stakeholder roles.
Figure 4. Our final multi-layer graphical representation adopts a three-layer representation with data flow, stakeholder interaction, and individual design decision. The combination of both a high-level overview and a detailed view supports quick updates and maintaining awareness of the big picture. A diagram showing two views of a data flow. (a) High-level view: A sequence of colored rectangular nodes representing data actions like "Collect application focus status" and stakeholder interactions like "Control attention tracking participation." (b) Detailed view: A pop-up box for the "Collect application focus status" node, showing its properties as key-value pairs, such as DataType: Application focus status and Frequency: Continuously. This demonstrates the tool's ability to zoom in from an overview to specific details.
Figure 5. We organize our domain-aware privacy design space by data actions and stakeholder interactions, categorizing privacy design decisions based on the applicability of their keys and values across different domains. A large, complex diagram showing a taxonomy of privacy design decisions organized by data actions (Collect, Process, Share, Notify) and stakeholder interactions (Consent, Control, Request). Within each category, decisions are classified as "Universal Keys with Global Values," "Universal Keys with Domain-specific Values," or "Domain-specific Keys and Values," with color-coded domain labels like Health, Social Media, and Behavior.
Coverage
GPT-4o
Claude-3.7
Gemini-2.5 Flash
PrivacyAkinator
Decision
Choice
Decision
Choice
Decision
Choice
Decision
Choice
Collect
59.74 %
29.47 %
60.90 %
30.77 %
65.38 %
55.13 %
98.08 %
80.77 %
Process
38.38 %
18.84 %
55.10 %
28.57 %
54.42 %
40.14 %
89.80 %
75.51 %
Store
55.64 %
35.16 %
55.26 %
30.70 %
75.27 %
56.76 %
100.00 %
92.11 %
Share
43.54 %
18.62 %
42.67 %
22.67 %
48.50 %
36.44 %
92.00 %
80.00 %
Consent
75.72 %
44.06 %
63.28 %
41.13 %
78.44 %
62.28 %
96.15 %
88.46 %
Table 2. PrivacyAkinator achieved broader coverage of key privacy design decisions and their corresponding values than three baseline models (i.e., GPT-4o, Claude 3.7, and Gemini 2.5 Flash) across all categories. A table comparing the performance of PrivacyAkinator against three baseline LLM models (GPT-4o, Claude-3.7, Gemini 2.5 Flash). For 10 categories of privacy decisions (e.g., Collect, Consent), the table shows the percentage of key Decisions and actual Choices that each model successfully identified.
Task ID
PRAM Worksheet
PrivacyAkinator
Time (min)
# Total Decisions
Key Decision Coverage
Time (min)
# Total Decisions
Key Decision Coverage
Task #1
36.4 ± 10.9
12.4 ± 3.5
36.9 ± 13.8 %
14.2 ± 9.6
33.5 ± 5.9
79.1 ± 3.8 %
Task #2
32.6 ± 11.1
11.2 ± 3.6
40.0 ± 16.9 %
7.0 ± 1.9
29.8 ± 6.2
93.3 ± 6.2 %
Task #3
31.5 ± 5.2
11.4 ± 3.2
38.2 ± 7.6 %
7.5 ± 1.8
26.3 ± 4.3
93.9 ± 4.1 %
Task #4
41.8 ± 9.3
13.3 ± 6.0
48.6 ± 9.8 %
11.2 ± 2.8
24.8 ± 1.7
87.5 ± 4.8 %
Average
35.8 ± 10.7
12.9 ± 4.3
42.6 ± 12.0 %
9.7 ± 6.4
28.9 ± 5.5
89.3 ± 7.0 %
Table 3. In our user study, participants spent less time and covered more key design decisions with PrivacyAkinator compared to PRAM Worksheet. Format: mean ± standard deviation. We also highlight the higher value between the two conditions. A table showing quantitative results from a user study. For four different tasks, it compares the performance of participants using the standard PRAM worksheet versus \nameon three metrics: Time (min), # Total Decisions identified, and Key Decision Coverage percentage. A final row provides the average for each metric.
Condition
Mental ↓
Physical ↓
Temporal ↓
Performance ↑
Effort ↓
Frustration ↓
PRAM
6 (5.4 ± 1.4)
2 (2.5 ± 1.8)
3 (3.6 ± 1.6)
5 (4.3 ± 1.1)
5 (5.3 ± 1.0)
3 (3.4 ± 2.0)
PrivacyAkinator
3 (3.3 ± 1.3) ***
1 (1.7 ± 1.1) *
2 (2.1 ± 0.9) ***
6 (5.7 ± 0.6) ***
3 (3.1 ± 1.5) ***
1 (1.7 ± 1.0) **
Table 4. Participants perceived significantly lower cognitive load using PrivacyAkinator compared to PRAM worksheet. We present NASA TLX results (scale of 1 to 7) as “median (mean ± standard deviation)”. We annotate statistical significance based on the Wilcoxon Signed-Rank test (∗:p<0.05,∗∗:p<0.01,∗∗∗:p<0.001) . ↓ denotes that the lower is better. A table presenting results from the NASA Task Load Index (TLX) survey, comparing the perceived cognitive load of using PRAM versus PrivacyAkinator. The columns represent six dimensions of workload: Mental, Physical, Temporal, Performance, Effort, and Frustration. Arrows indicate whether a lower or higher score is better. Asterisks denote statistically significant differences.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. PrivacyAkinator Design Goal Panel A screenshot of the initial user interface for \name. It features a title, "\name," and a large text input box labeled "Input your design goal." The box contains example text: "Design an attendee attention tracking feature for a video conferencing application." There are buttons for "Load Example" and "Continue."
Figure 7. PrivacyAkinator functional Requirement Panel A screenshot of the functional requirements editor in \name. It shows a list of requirements organized under headings like "Data Collection" and "Data Processing." Specific requirements, such as "Calculate attention status when application loses focus for more than 30 seconds," are highlighted, indicating that these are editable design choices.
Figure 8. PrivacyAkinator Workflow & Questions Panel A composite screenshot showing the main interaction panel of \name. The top part displays a question for the user ("Would users be notified…?") with multiple-choice answers. To the right is the "Related data action," which provides context. The bottom part shows the multi-layer graphical representation of the system design, which updates as the user answers questions.
Figure 9. PrivacyAkinator Access System Design Panel A screenshot of the system design assessment table in \name. The table has columns for Data Action, Data, Specific Context, and Summary Issues. Rows detail specific actions like "Collect application focus status" and list potential issues, such as "Continuous monitoring may create privacy concerns for users," with options for the user to validate or edit these issues.
Figure 10. PrivacyAkinator Generated Worksheet A screenshot of a worksheet exported from \name, formatted for NIST PRAM. It's a detailed table with columns for Data Action, Data, Specific Context, and Summary Issues, summarizing the privacy analysis of the user's design. This shows the final output of the tool.
Category
Operation
Definition
Data Action
Collect
Collect users’ data or sensor inputs.
Process
Process data to derive new information.
Store
Keep data in a persistent storage system.
Share
Share data to different parties.
Stakeholder Interaction
Consent
A data subject gives permission for specific data actions.
Notice
A data observer informs data subjects about the data action or its results.
Appendix
Table 5. Data Actions and Stakeholder Interactions A table defining the common operations used in PrivacyAkinator's representation. The table has three columns: Category (Data Action or Stakeholder Interaction), Operation (e.g., Collect, Consent, Control), and Definition (a brief explanation of the operation).
#ID
Scenario
Description
#1
Zoom Attention Tracking
An attendee attention tracking feature for a video conferencing application
#2
Facebook Cambridge Analytica
A data platform that allows third-party apps to collect users’ data through a social media platform
#3
AirTag
A real-time location tracking device for personal items
#4
Target Pregnancy Prediction
A retail analytics system that processes customer purchase histories to automatically generate personalized coupons and recommendations
#5
Facebook Emotional Contagion
A research experiment to study how exposure to emotional content affects users’ own emotional expressions on a social media platform
#6
Google Buzz
A system that integrates email services with social networking functionality
Appendix
Table 6. A summary of 30 data practices collected to assess the coverage of key privacy design decisions. A table listing 30 real-world data practices used to evaluate the coverage of PrivacyAkinator's design space. The columns are ID and Scenario (e.g., "Zoom Attention Tracking," "Target Pregnancy Prediction," "Social Credit Score"), with a brief Description for each.
Phase / Section
Prompts
Introduction
Thank you for participating. In this study, we are interested in understanding how developers perform privacy risk assessments. We will ask you to work through a task using a standard framework called PRAM. There are no right or wrong answers; we are interested in your natural thought process.
Instruction
As you work on the task, please try to think out loud. Tell us what you’re thinking. This will help us understand your perspective.
Consent Script
Before we begin the study, I need to obtain your informed consent. This session will be recorded for the sole purpose of accurate transcription. The recording will be permanently deleted by the research team immediately after the transcript is verified. All data collected from the interview will be fully anonymized to protect your privacy. Your participation is completely voluntary. You have the right to skip any question you are not comfortable with, or to withdraw from the study at any time without penalty.
Post-study Questions
Overall
How would you describe your overall experience of using the PRAM framework? Which of the PRAM worksheets or tasks did you find the most challenging, and why? Which part of the framework, if any, did you find helpful or easy to use? Why? If you could change one thing about this framework to make it easier for developers, what would it be?
WS1: Framing Business Objectives
How are you thinking about the main purpose of this system? What factors are you considering when describing the functional requirements?
Appendix
Table 7. The interview protocol used during the observational study. A table outlining the questions asked to participants during the observational study. It is organized by Phase / Section (e.g., Introduction, Post-study Questions, WS2: Assessing System Design) and lists the specific Prompts or questions for each part of the interview.
Theme
Post-Study Interview Questions
Overall Experiences
How would you describe your overall experience using the PRAM worksheet versus PrivacyAkinator? Which approach did you find more intuitive or easier to use, and why?
Strengths & Weaknesses
What aspects of the PRAM worksheet did you find most helpful or effective? What aspects of PRAM worksheet were confusing, frustrating, or difficult to use? What challenges or limitations did you encounter with the PRAM worksheet? What aspects of PrivacyAkinator did you find most helpful or effective? What aspects of PrivacyAkinator were confusing, frustrating, or difficult to use? What challenges or limitations did you encounter with PrivacyAkinator?
Reflection & Preference
If you need to use one approach in a real project, which would you choose, and why? How would you improve either approach based on your experience?
Appendix
Table 8. The interview questions used during the evaluation study. A table listing the post-study interview questions for the evaluation study comparing PRAM and \name. The columns are Theme (e.g., Overall Experiences, Strengths & Weaknesses) and the Post-Study Interview Questions corresponding to each theme.
Theme
Code
Sub-code
Description
Attention Allocation
Information Overload
Cognitive Fatigue
Participants experienced heavy mental load from juggling numerous factors simultaneously. This led to growing fatigue as the tasks progressed.
Context-Switching Overhead
Participants experienced additional cognitive load when transferring information across worksheets or repeatedly referring back and forth between different parts of the task.
Ineffective Prioritization
Fixation on Familiar Concepts
Participants focused excessively on design aspects that aligned with their technical background, while neglecting less familiar but critical privacy risks.
Fixation on Early Tasks
Participants spent a disproportionate amount of time and mental energy on the initial stages of a task, leading them to rush through or neglect later stages.
Inaccessible Privacy Knowledge
Knowledge Blindspot
Unaware of Key Concepts
Participants overlooked important privacy issues because they did not know these aspects needed to be considered.
Knowledge Recall Failure
Missed Known Concepts
Participants knew about a relevant privacy concept but forgot to apply or include it during the assessment due to a slip of the mind.
Appendix
Table 9. The codebook developed during our qualitative analysis of the observational study A table summarizing the themes and codes developed from analyzing user study data. The columns are Theme (e.g., Attention Allocation), Code (e.g., Information Overload), Sub-code (e.g., Cognitive Fatigue), and a Description of the observed challenge.
Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.
Recent work on protecting privacy during user-LLM interactions often focuses on direct, explicit identifiers: the personally-identifiable information (PII) captured by standard detectors. One such approach is Privacy-Conscious Delegation (PCD), where a local LLM acts as an intermediary. However, privacy risk does not stem solely from explicit identifiers but also PII-free self-disclosures, leaving users identifiable through combinations of quasi-identifying traits. We investigate a probabilistic variant of PCD, where we augment its objectives with an LLM-driven probabilistic estimation of k-anonymity. To facilitate this, we first create the PUPA-SD dataset, which contains naturalistic user queries with self-disclosure. Our preliminary results indicate that optimizing PAPILLON on PUPA-SD improves quality on unseen conversations across a variety of local models and produces the best privacy-utility balance for Llama-3.2-3B, while smaller models struggle to jointly optimize quality and privacy. We propose k-anonymity as a useful auxiliary metric for tackling PCD.
Li Siyan, Zhou Yu, Julia Hirschberg
Department of Computer Science Columbia University
While prior work has documented privacy failures in LLM agents, it remains unclear how the presentation of privacy guidance influences their choice of information sources. We introduce PrivacySkills, a controlled framework for evaluating how agents choose among acquisition pathways that provide the same task-relevant value: consulting publicly available personal information, accessing confidential sources, or interacting with the user. The evaluation framework comprises 55 synthetic tasks spanning 11 categories of personal information, with 169 associated skills that describe the available acquisition pathways. We consider privacy guidance through system-level instructions, skill-level metadata labels, or both. Separately, we vary user availability and urgency framing. With users available and no privacy guidance, agents access confidential sources in 30% of valid runs on average across five open-weight models, despite sufficient alternatives. This rate increases to 45% when users are unavailable, whereas urgency framing has no detectable effect. System-level privacy instructions alone have limited effects on confidential access, while skill-level intrusiveness labels produce a modest reduction (24% on average), but combining the two roughly halves confidential access. Our findings motivate incorporating privacy annotations into skill specifications and evaluating their effectiveness alongside system-level instructions.
Lucas Biechy, Cédric Eichler, Héber H. Arcolezi +1