cs.HCApr 8, 2026
SavePrivacyAkinator: Articulating Key Privacy Design Decisions by Answering LLM-Generated Multiple-choice Questions
Organizations: University of California San Diego, La Jolla, California, USA · University of California Riverside, Riverside, California, USA
Abstract
NIST's Privacy Risk Assessment Methodology (PRAM) provides a structured framework for privacy experts to assess privacy risks. However, its complexity and reliance on expert knowledge make it difficult for novice developers to use effectively. This paper explores methods to lower these barriers. We first performed an observational study with 12 participants using PRAM in real-world scenarios, and found that novice developers struggled most with articulating privacy-related design decisions. We then developed PrivacyAkinator, an interactive tool that helps developers articulate key privacy decisions by answering LLM-generated multiple-choice questions. PrivacyAkinator introduces three innovations: a universal privacy representation that abstracts privacy-related design decisions into data flows and stakeholder interactions; a domain-aware design space mined from 10K privacy-related news articles; and a dynamic question-generation workflow to prioritize relevant questions. Our user study with 24 participants suggests that developers using PrivacyAkinator identified 47% more key decisions in 73% less time compared to PRAM.
Figures & tables
Figure 1. PrivacyAkinator helps developers articulate key privacy design decisions by answering LLM-generated multiple-choice questions. After (1) developers provide a high-level description of their system, (2) PrivacyAkinator expands it into detailed, editable functional requirements with adaptable design choices highlighted. (3) PrivacyAkinator guides developers through key design decisions with contextualized, specific questions. As developers make choices, PrivacyAkinator organizes and documents these decisions within a structured privacy representation. (4) PrivacyAkinator maps these design decisions to NIST PRAM worksheets ( NIST, 2020 ) to further identify potential issues and prioritize privacy risks. Note that we only enumerate data actions and relevant design decisions; analysts still need to manually evaluate risk likelihood and impact.
| #ID | Scenario | Description | Participants |
|---|---|---|---|
| #1 | Zoom Attendee Attention Tracking ( Li et al., 2024b ; Amatulli, 2020 ) | A feature that monitored attendees’ attention during video conferences by tracking whether Zoom was the active application on a participant’s computer. It showed a clock icon next to inattentive participants and generated a post-meeting report with attention percentage scores. | P1, P2, P3, P8 |
| #2 | Alexa’s Smart Home Voice Assistant ( Matt Day and Bloomberg, 2019 ) | A cloud-based voice service for smart home devices that allows users to control their home environment through voice commands. It captures audio when activated by a wake word, processes requests in the cloud, and executes actions through connected devices. | P4, P7, P11 |
| #3 | Target Retail Recommendation System ( Hill, 2024 ) | An analytics system that identified pregnant customers based on purchase patterns of approximately 25 indicator products, allowing Target to predict pregnancy and estimate due dates. This enabled targeted marketing of pregnancy and baby-related products. | P5, P9, P12 |
| #4 | 23andMe Direct-to-consumer Genetic Testing ( Mineo, 2025 ) | A service that collects and analyzes genetic information from customer saliva samples for ancestry, health traits, and disease risks. The service stores comprehensive genetic profiles that can reveal sensitive information about individuals and their biological relatives. | P6, P10 |
Table 1. Task scenarios used in our study. We selected four task scenarios based on participant familiarity, scenario complexity, and domain diversity. For each scenario, we provided participants with a high-level description of the system. A table listing four scenarios used to study how developers conduct privacy risk assessments. The columns are ID, Scenario (e.g., "Zoom Attendee Attention Tracking"), Description, and Participants (showing which study participants were assigned to each scenario).
Figure 2. Data Flow Diagram defines data actions that occur across the data lifecycle, indicates the key entities involved in the system, and connects the data flows between them. However, the level of detail is insufficient to capture privacy nuances, and it omits interaction with different stakeholders, and may lead to deferred responsibility for privacy. A flowchart illustrating the data lifecycle for a Zoom attention tracking feature. An "Attendee" icon is connected to a "Zoom" icon, which then connects to a "Cloud Storage Provider", and finally to a "Host" icon. Numbered data keys indicate what information is transferred at each step: (1) Attendee Account Information, (2) Active Windows, (3) Aggregated Attention Percentage Score, and (4) Attendance Report. A legend explains the icons for Individual, Data Storage, Computer, Active Windows, Zoom, and Third-Party.
Figure 3. Privacy Storyboard illustrates data actions across the data lifecycle with stakeholder roles. While more aligned with developer workflow, it can overwhelm users with too much information and often conflates multiple design decisions. A flowchart showing four main stages: Collect, Process, Store, and Access, with arrows indicating the flow from "Meeting Participants" to a "Host". Each stage has a box with a brief description. For example, the "Collect" stage "Collects clients' screen activities continuously during host's screen-sharing meeting session." The storyboard illustrates data actions and stakeholder roles.
Figure 4. Our final multi-layer graphical representation adopts a three-layer representation with data flow, stakeholder interaction, and individual design decision. The combination of both a high-level overview and a detailed view supports quick updates and maintaining awareness of the big picture. A diagram showing two views of a data flow. (a) High-level view: A sequence of colored rectangular nodes representing data actions like "Collect application focus status" and stakeholder interactions like "Control attention tracking participation." (b) Detailed view: A pop-up box for the "Collect application focus status" node, showing its properties as key-value pairs, such as DataType: Application focus status and Frequency: Continuously. This demonstrates the tool's ability to zoom in from an overview to specific details.
Figure 5. We organize our domain-aware privacy design space by data actions and stakeholder interactions, categorizing privacy design decisions based on the applicability of their keys and values across different domains. A large, complex diagram showing a taxonomy of privacy design decisions organized by data actions (Collect, Process, Share, Notify) and stakeholder interactions (Consent, Control, Request). Within each category, decisions are classified as "Universal Keys with Global Values," "Universal Keys with Domain-specific Values," or "Domain-specific Keys and Values," with color-coded domain labels like Health, Social Media, and Behavior.
| Coverage | GPT-4o | Claude-3.7 | Gemini-2.5 Flash | PrivacyAkinator | ||||
| Decision | Choice | Decision | Choice | Decision | Choice | Decision | Choice | |
| Collect | 59.74 % | 29.47 % | 60.90 % | 30.77 % | 65.38 % | 55.13 % | 98.08 % | 80.77 % |
| Process | 38.38 % | 18.84 % | 55.10 % | 28.57 % | 54.42 % | 40.14 % | 89.80 % | 75.51 % |
| Store | 55.64 % | 35.16 % | 55.26 % | 30.70 % | 75.27 % | 56.76 % | 100.00 % | 92.11 % |
| Share | 43.54 % | 18.62 % | 42.67 % | 22.67 % | 48.50 % | 36.44 % | 92.00 % | 80.00 % |
| Consent | 75.72 % | 44.06 % | 63.28 % | 41.13 % | 78.44 % | 62.28 % | 96.15 % | 88.46 % |
Table 2. PrivacyAkinator achieved broader coverage of key privacy design decisions and their corresponding values than three baseline models (i.e., GPT-4o, Claude 3.7, and Gemini 2.5 Flash) across all categories. A table comparing the performance of PrivacyAkinator against three baseline LLM models (GPT-4o, Claude-3.7, Gemini 2.5 Flash). For 10 categories of privacy decisions (e.g., Collect, Consent), the table shows the percentage of key Decisions and actual Choices that each model successfully identified.
| Task ID | PRAM Worksheet | PrivacyAkinator | ||||
|---|---|---|---|---|---|---|
| Time (min) | # Total Decisions | Key Decision Coverage | Time (min) | # Total Decisions | Key Decision Coverage | |
| Task #1 | 36.4 10.9 | 12.4 3.5 | 36.9 13.8 % | 14.2 9.6 | 33.5 5.9 | 79.1 3.8 % |
| Task #2 | 32.6 11.1 | 11.2 3.6 | 40.0 16.9 % | 7.0 1.9 | 29.8 6.2 | 93.3 6.2 % |
| Task #3 | 31.5 5.2 | 11.4 3.2 | 38.2 7.6 % | 7.5 1.8 | 26.3 4.3 | 93.9 4.1 % |
| Task #4 | 41.8 9.3 | 13.3 6.0 | 48.6 9.8 % | 11.2 2.8 | 24.8 1.7 | 87.5 4.8 % |
| Average | 35.8 10.7 | 12.9 4.3 | 42.6 12.0 % | 9.7 6.4 | 28.9 5.5 | 89.3 7.0 % |
Table 3. In our user study, participants spent less time and covered more key design decisions with PrivacyAkinator compared to PRAM Worksheet. Format: mean standard deviation. We also highlight the higher value between the two conditions. A table showing quantitative results from a user study. For four different tasks, it compares the performance of participants using the standard PRAM worksheet versus \nameon three metrics: Time (min), # Total Decisions identified, and Key Decision Coverage percentage. A final row provides the average for each metric.
| Condition | Mental | Physical | Temporal | Performance | Effort | Frustration |
|---|---|---|---|---|---|---|
| PRAM | 6 (5.4 1.4) | 2 (2.5 1.8) | 3 (3.6 1.6) | 5 (4.3 1.1) | 5 (5.3 1.0) | 3 (3.4 2.0) |
| PrivacyAkinator | 3 (3.3 1.3) *** | 1 (1.7 1.1) * | 2 (2.1 0.9) *** | 6 (5.7 0.6) *** | 3 (3.1 1.5) *** | 1 (1.7 1.0) ** |
Table 4. Participants perceived significantly lower cognitive load using PrivacyAkinator compared to PRAM worksheet. We present NASA TLX results (scale of 1 to 7) as “median (mean standard deviation)”. We annotate statistical significance based on the Wilcoxon Signed-Rank test . denotes that the lower is better. A table presenting results from the NASA Task Load Index (TLX) survey, comparing the perceived cognitive load of using PRAM versus PrivacyAkinator. The columns represent six dimensions of workload: Mental, Physical, Temporal, Performance, Effort, and Frustration. Arrows indicate whether a lower or higher score is better. Asterisks denote statistically significant differences.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. PrivacyAkinator Design Goal Panel A screenshot of the initial user interface for \name. It features a title, "\name," and a large text input box labeled "Input your design goal." The box contains example text: "Design an attendee attention tracking feature for a video conferencing application." There are buttons for "Load Example" and "Continue."
Figure 7. PrivacyAkinator functional Requirement Panel A screenshot of the functional requirements editor in \name. It shows a list of requirements organized under headings like "Data Collection" and "Data Processing." Specific requirements, such as "Calculate attention status when application loses focus for more than 30 seconds," are highlighted, indicating that these are editable design choices.
Figure 8. PrivacyAkinator Workflow & Questions Panel A composite screenshot showing the main interaction panel of \name. The top part displays a question for the user ("Would users be notified…?") with multiple-choice answers. To the right is the "Related data action," which provides context. The bottom part shows the multi-layer graphical representation of the system design, which updates as the user answers questions.
Figure 9. PrivacyAkinator Access System Design Panel A screenshot of the system design assessment table in \name. The table has columns for Data Action, Data, Specific Context, and Summary Issues. Rows detail specific actions like "Collect application focus status" and list potential issues, such as "Continuous monitoring may create privacy concerns for users," with options for the user to validate or edit these issues.
Figure 10. PrivacyAkinator Generated Worksheet A screenshot of a worksheet exported from \name, formatted for NIST PRAM. It's a detailed table with columns for Data Action, Data, Specific Context, and Summary Issues, summarizing the privacy analysis of the user's design. This shows the final output of the tool.
| Category | Operation | Definition |
| Data Action | Collect | Collect users’ data or sensor inputs. |
| Process | Process data to derive new information. | |
| Store | Keep data in a persistent storage system. | |
| Share | Share data to different parties. | |
| Stakeholder Interaction | Consent | A data subject gives permission for specific data actions. |
| Notice | A data observer informs data subjects about the data action or its results. |
Table 5. Data Actions and Stakeholder Interactions A table defining the common operations used in PrivacyAkinator's representation. The table has three columns: Category (Data Action or Stakeholder Interaction), Operation (e.g., Collect, Consent, Control), and Definition (a brief explanation of the operation).
| #ID | Scenario | Description |
|---|---|---|
| #1 | Zoom Attention Tracking | An attendee attention tracking feature for a video conferencing application |
| #2 | Facebook Cambridge Analytica | A data platform that allows third-party apps to collect users’ data through a social media platform |
| #3 | AirTag | A real-time location tracking device for personal items |
| #4 | Target Pregnancy Prediction | A retail analytics system that processes customer purchase histories to automatically generate personalized coupons and recommendations |
| #5 | Facebook Emotional Contagion | A research experiment to study how exposure to emotional content affects users’ own emotional expressions on a social media platform |
| #6 | Google Buzz | A system that integrates email services with social networking functionality |
Table 6. A summary of 30 data practices collected to assess the coverage of key privacy design decisions. A table listing 30 real-world data practices used to evaluate the coverage of PrivacyAkinator's design space. The columns are ID and Scenario (e.g., "Zoom Attention Tracking," "Target Pregnancy Prediction," "Social Credit Score"), with a brief Description for each.
| Phase / Section | Prompts |
|---|---|
| Introduction | Thank you for participating. In this study, we are interested in understanding how developers perform privacy risk assessments. We will ask you to work through a task using a standard framework called PRAM. There are no right or wrong answers; we are interested in your natural thought process. |
| Instruction | As you work on the task, please try to think out loud. Tell us what you’re thinking. This will help us understand your perspective. |
| Consent Script | Before we begin the study, I need to obtain your informed consent. This session will be recorded for the sole purpose of accurate transcription. The recording will be permanently deleted by the research team immediately after the transcript is verified. All data collected from the interview will be fully anonymized to protect your privacy. Your participation is completely voluntary. You have the right to skip any question you are not comfortable with, or to withdraw from the study at any time without penalty. |
| Post-study Questions | |
| Overall | How would you describe your overall experience of using the PRAM framework? Which of the PRAM worksheets or tasks did you find the most challenging, and why? Which part of the framework, if any, did you find helpful or easy to use? Why? If you could change one thing about this framework to make it easier for developers, what would it be? |
| WS1: Framing Business Objectives | How are you thinking about the main purpose of this system? What factors are you considering when describing the functional requirements? |
Table 7. The interview protocol used during the observational study. A table outlining the questions asked to participants during the observational study. It is organized by Phase / Section (e.g., Introduction, Post-study Questions, WS2: Assessing System Design) and lists the specific Prompts or questions for each part of the interview.
| Theme | Post-Study Interview Questions |
|---|---|
| Overall Experiences | How would you describe your overall experience using the PRAM worksheet versus PrivacyAkinator? Which approach did you find more intuitive or easier to use, and why? |
| Strengths & Weaknesses | What aspects of the PRAM worksheet did you find most helpful or effective? What aspects of PRAM worksheet were confusing, frustrating, or difficult to use? What challenges or limitations did you encounter with the PRAM worksheet? What aspects of PrivacyAkinator did you find most helpful or effective? What aspects of PrivacyAkinator were confusing, frustrating, or difficult to use? What challenges or limitations did you encounter with PrivacyAkinator? |
| Reflection & Preference | If you need to use one approach in a real project, which would you choose, and why? How would you improve either approach based on your experience? |
Table 8. The interview questions used during the evaluation study. A table listing the post-study interview questions for the evaluation study comparing PRAM and \name. The columns are Theme (e.g., Overall Experiences, Strengths & Weaknesses) and the Post-Study Interview Questions corresponding to each theme.
| Theme | Code | Sub-code | Description |
| Attention Allocation | Information Overload | Cognitive Fatigue | Participants experienced heavy mental load from juggling numerous factors simultaneously. This led to growing fatigue as the tasks progressed. |
| Context-Switching Overhead | Participants experienced additional cognitive load when transferring information across worksheets or repeatedly referring back and forth between different parts of the task. | ||
| Ineffective Prioritization | Fixation on Familiar Concepts | Participants focused excessively on design aspects that aligned with their technical background, while neglecting less familiar but critical privacy risks. | |
| Fixation on Early Tasks | Participants spent a disproportionate amount of time and mental energy on the initial stages of a task, leading them to rush through or neglect later stages. | ||
| Inaccessible Privacy Knowledge | Knowledge Blindspot | Unaware of Key Concepts | Participants overlooked important privacy issues because they did not know these aspects needed to be considered. |
| Knowledge Recall Failure | Missed Known Concepts | Participants knew about a relevant privacy concept but forgot to apply or include it during the assessment due to a slip of the mind. |
Table 9. The codebook developed during our qualitative analysis of the observational study A table summarizing the themes and codes developed from analyzing user study data. The columns are Theme (e.g., Attention Allocation), Code (e.g., Information Overload), Sub-code (e.g., Cognitive Fatigue), and a Description of the observed challenge.