End-users seldom read verbose privacy policies, leading app stores like Google Play to mandate simplified data safety declarations as a user-friendly alternative. However, these self-declared disclosures often contradict the full privacy policies, deceiving users about actual data practices and violating regulatory requirements for consistency. To address this, we introduce PrivPRISM, a robust framework that combines encoder and decoder language models to systematically extract and compare fine-grained data practices from privacy policies and to compare against data safety declarations, enabling scalable detection of non-compliance. Evaluating 7,770 popular mobile games uncovers discrepancies in nearly 53% of cases, rising to 61% among 1,711 widely used generic apps. Additionally, static code analysis reveals possible under-disclosures, with privacy policies disclosing just 66.8% of potential accesses to sensitive data like location and financial information, versus only 36.4% in data safety declarations of mobile games. Our findings expose systemic issues, including widespread reuse of generic privacy policies, vague / contradictory statements, and hidden risks in high-profile apps with 100M+ downloads, underscoring the urgent need for automated enforcement to protect platform integrity and for end-users to be vigilant about sensitive data they disclose via popular apps.
Figures & tables
Figure 1: Top: Data safety declarations of three games by Spider Solitaire Card Games and bottom: corresponding privacy policy highlighting inconsistencies. Data safety labels only report collecting and sharing device identifiers for analytics and ads, yet their privacy policy omit ad-related details and instead disclose access to more sensitive data such as GPS location, user accounts, and app activity. Identified via PrivPRISM framework deployment to mobile games.
Figure 2: Terminology used in PrivPRISM
Figure 3: End-to-end pipeline of our framework (top) and a walkthrough example (bottom)
Ours (EDPC)
F.T. Llama3.1
Z.S.GPT4o
Class
Pr
Re
Pr
Re
Pr
Re
c0
0.94
0.74
0.90
0.21
0.88
0.69
c1
0.96
0.66
0.88
0.25
0.85
0.68
c4
0.73
0.41
0.51
0.19
0.87
0.54
μcˉ
0.91
0.56
0.41
0.24
0.85
0.54
c0 -first party collection, c1 -third party sharing
Table 1: Benchmarked test-set results for OPP-115 dataset conducted for our Explained Data Practice Classification (EDPC) module compared against fine-tuned Llama3.1 and zero-shot prompted GPT4o; both performing as explainable classifiers.
Figure 4: Data item mapping verifier embedding alignment and transferability
Downloads:
∼5M+
∼1−5M
Collect
Share
Collect
Share
PP Compliance
81.23
68.08
83.28
69.17
DS Compliance
25.44
22.58
21.29
18.47
Table 2: Overall Data Practice Compliance (%) Landscape
Figure 5: Structural composition of a privacy policy - A column represents the data practice category distribution of the policy dataset at a given normalised length.
p[0]
p[1]
p[2]
p[3]
p[4]
p[5]
p[6]
d[0]
0.71
4.00
0.78
1.40
7.50
33.06
20.41
d[1]
2.63
17.09
6.76
1.96
1.55
50.00
11.81
d[2]
20.00
7.49
14.66
5.71
12.88
43.48
35.10
d[4]
0.00
7.69
0.00
0.00
0.00
40.00
7.14
d[8]
16.55
1.45
34.39
6.57
6.37
1.76
25.76
d[9]
26.46
2.05
7.11
7.83
9.33
1.95
17.19
Table 3: Data purpose compliance (%): Each row represents kth purpose compliance score for a given valid data item category j .
Figure 6: Four types of moving average filtered data item compliance scores based on the app popularity. Moving average window = 50, Shade = 0.5*STD
Figure 7: Number of data item declarations mentioned in PP, DS and within APK evidence for 4,538 game apps.
Figure 8: Data item collection or sharing frequency for game and non-game apps if mentioned in privacy policy (PP) or data safety (DS). X axis is normalised for easier comparison; i.e., the x-axis is normalised to show how many data items are observed per 1,000 privacy policies.
Figure 9: APK evidence analysis for non-game apps. Note: the x-axes are normalised for easier observations, therefore are in the same scale.
Figure 10: Privacy Policy and Data Safety collection and sharing compliance scores with respect to the corresponding non-game app’s category. Note that several categories contained less than 50 apps when we selected the most popular non-game app corpus. Data points: mean compliance per category, Shade: 0.5*STD
Figure 11: (a): A case study of ‘FARM STUDIO’, (b,c): Existing policy links but invalid contents. Validated 2025.12
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Combinations of Google Data Safety declaration labels, compared with data practice, data item and data purpose categories we utilise in PrivPRISM framework.
PrivPRISM’s EDPC
F.T. Llama3.1
Z.S. GPT4o
Class
Pr
Re
F1
Pr
Re
F1
Pr
Re
F1
Support
c0 First Party Collection / Use
0.94
0.74
0.83
0.90
0.21
0.34
0.88
0.69
0.78
289
c1 Third Party Sharing / Collection
0.96
0.66
0.78
0.88
0.25
0.38
0.85
0.68
0.76
204
c2 User Choice / Control
0.95
0.46
0.62
0.65
0.19
0.30
0.87
0.54
0.67
115
c3 User Access, Edit and Deletion
0.94
0.37
0.53
0.60
0.15
0.24
0.72
0.63
0.68
41
c4 Introductory / Generic
0.73
0.41
0.52
0.51
0.25
0.33
0.94
0.25
0.39
118
Appendix
Table 4: Data Practice Classification Results. EDPC: Explained Data Practice Classifier, F.T.:Fine-tuned, Z.S.:Zero Shot
Trained on: GPT
Trained on: GPT
Trained on: Llama
Trained on: Llama
Tested on: GPT
Tested on: Llama
Tested on: Llama
Tested on: GPT
Class
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Support
Name
0.97
0.92
0.94
0.98
0.95
0.96
0.97
0.93
0.95
0.96
0.92
0.94
199 149*
Email
0.99
0.98
0.99
0.99
0.99
0.99
0.97
0.98
0.97
0.98
0.99
0.98
347 295*
User Acct
0.95
0.80
0.87
0.96
0.67
0.79
0.84
0.69
0.75
0.93
0.83
0.88
475 436*
Financial
0.95
0.98
0.96
0.94
0.91
0.92
0.94
0.85
0.89
0.94
0.99
0.97
270 205*
Appendix
Table 5: Transferability of self-supervised mapping verifier training
Class
Pr
Re
F1
Support
0 - Analytics
0.96
0.85
0.91
627
1 - Dev. Commu.
0.80
0.88
0.84
320
2 - Fraud Prevention
0.95
0.95
0.95
1530
3 - Advertising
0.97
0.95
0.96
590
4 - Personalisation
0.86
0.95
0.90
796
5 - Account Manage
0.96
0.88
0.92
847
Appendix
Table 6: Alignment of 110M encoder model against the ground truth assumption from a 8B decoder model
Figure 14: An example output from our framework for a given PP, DS pair with 0% overall compliance score. Note: For output visualisation only - a more readable version is discussed in the main text, Fig. 1 .
Existing research typically treats privacy policies as flat, uniform text, extracting information without regard for the document's logical hierarchy. Disregard for structural cues of section headings designed to guide the reader, often leads automated methods to entangle distinct data practices, particularly when linking sensitive data items to their specific purposes. To address this, we introduce PrivSTRUCT, a novel and systematic encoder and decoder combined framework that to untangle complex privacy disclosures. Benchmarking against the state-of-the-art tool PoliGrapher reveals that PrivSTRUCT robustly extracts more than x2 the number of data item and purpose excerpts while retaining developer-defined structural cues. By applying PrivSTRUCT to a large-scale dataset of 3,756 Android apps, we uncover a critical transparency gap: the probability of developers overstating a data purpose is 20.4% higher for first-party collection and 9.7% higher for third-party sharing when they rely on globally defined purposes rather than specific, locally scoped disclosures. Alarmingly, we find that sensitive third-party data flows such as sharing financial data for analytics are frequently diluted and entangled into generic or unrelated categories, highlighting a persistent failure in the current purpose disclosure landscape.
Automated analyses of privacy policies enable large-scale assessments of transparency in digital ecosystems, yet existing auditing pipelines remain predominantly English-centric. This limits their ability to systematically evaluate multilingual environments, as in the European Union, where many services disclose privacy practices only in local languages. This paper examines whether large language models (LLMs) can extend privacy policy analysis beyond English without requiring language-specific adaptation, thus empowering large-scale auditing in linguistically diverse app ecosystems. We assemble an evaluation corpus spanning all 24 official EU languages from translated versions of two established expert-annotated datasets (OPP-115 and MAPP) and assess translation fidelity through automated metrics and targeted legal-expert review. Our LLM-based classifier for identifying categories of personal data collection achieves stable cross-lingual performance, with macro-F1 scores ranging between 0.91 and 0.94. We then leverage this capability in a large-scale audit of 2,611 Android applications from the Spanish Google Play Store. Combining multilingual privacy policy analysis with the evaluation of corresponding privacy labels and runtime network traffic exposes an important linguistic barrier: public-sector apps predominantly provide privacy policies in Spanish, whereas popular commercial apps mostly provide them in English. We reveal systematic discrepancies between declared and observed practices, especially in public-sector apps. Overall, our results indicate how English-only privacy audits can systematically obfuscate transparency gaps in multilingual environments.
Marcos Moran, David Rodriguez, Luka Nenadic +2
1, 2, 5Universidad Politécnica de Madrid · 2, 5Information Processing and Telecommunications Center · 3ETH Zurich +1
Privacy policies may contain internal contradictions in which commitments are undermined by practices documented elsewhere in the same policy. We operationalize this phenomenon, privacy washing, through a four-stage pipeline: statement extraction, compatibility filtering and natural language inference screening, multi-model judge verification, and thematic analysis, with contradictions confirmed by majority vote of a three-model LLM panel. Applied to two corpora of website privacy policies, 123 collected in 2026 (OPPT) and 115 collected in 2015 (OPP-115), the pipeline finds the same category patterns recurring across the 11-year gap, with third-party sharing contradictions the majority of confirmed cases in each primary run, consistent with structural factors in policy composition rather than necessarily intentional deception. At least one panel-confirmed contradiction appears in 12.2% of OPPT companies (15/123; 9.8% excluding legacy pairs) and 36.5% of OPP-115 companies (42/115). A stability re-run seven months later, with a fully separated configuration (new extraction models, judges from three Chinese providers absent from both corpora, matched filters, no judge-submission similarity threshold), reproduces the OPPT prevalence under the original protocol (13.0% vs. 12.2%), finds sub-threshold pairs confirm at rates of the same order as those above (raising prevalence to 20.3% and 40.9%), and shows the third-party majority is panel-sensitive while the recurrence of the same category pairs is not. Two caveats govern all figures: panel verdicts are not validated against human expert judgment, so precision is unknown and prevalence figures are lower bounds; and the two primary runs used different filter configurations, so their prevalence difference is not interpretable as a corpus or era effect (the matched re-run reduces the gap to roughly twofold but does not eliminate it).