PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review
Organizations: North Carolina State University, Raleigh, NC, USA · Independent Researcher
Abstract
Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024. Each PR includes its commits, diffs, review discussion, and LLM-generated labels for privacy relevance, fix type, and GDPR principles. PrivacyPR provides 1,169 high-confidence privacy fixes and 1,643 non-fix PRs, together with Verified, a manually checked split of 250 privacy fixes and 250 non-fix PRs, and Small, a 100 + 100 subset for evaluation. On Verified, both annotators endorsed 72% of the LLM's GDPR-principle assignments. Beyond supporting empirical study of how privacy defects arise and are fixed, PrivacyPR provides a benchmark for privacy-aware code review on real, developer-accepted fixes. Its evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detect only 42-51% of defects under a generic review prompt, but 88-91% when explicitly asked to check privacy, while flagging 11-15% of non-fix PRs. These results show that privacy defects can be detected by coding agents, but that detection depends strongly on review intent and comes with false alarms. Data, code, reviews, judge verdicts, and an extensible evaluation harness are publicly available at https://doi.org/10.5281/zenodo.23744765.
Figures & tables
| Field | Values and meaning |
|---|---|
| is_privacy_related | true if the PR’s primary goal is to fix a privacy issue, e.g., faulty handling of personal data or missing privacy functionality |
| fix_type | one or more of 8: data removal, data masking, access control, data exposure restriction, encryption or secure storage, logging fix, consent or policy enforcement, other |
| gdpr_principles | one or more of the six principles of GDPR Art. 5(1) |
| gdpr_articles | free list of GDPR articles, e.g., Art. 32 |
| confidence_level | 1 (low) to 5 (high); tracks privacy relevance (Section II-E ) |
| reasoning , evidence | free text justifying the label |
| PRs (%) | Lines | Files | Rev.% | Dur.% | Com. | Days | |
| Fix type | |||||||
| Exposure restriction | 807 (69.0) | 106 | 5 | 57.9 | 5.0 | 10.3 | 2.2 |
| Consent/policy | 405 (34.6) | 87 | 4 | 57.5 | 4.2 | 8.8 | 2.0 |
| Access control | 242 (20.7) | 402 | 12 | 71.5 | 5.8 | 17.9 | 4.4 |
| Other | 195 (16.7) | 51 | 4 | 47.7 | 3.1 | 10.0 | 1.2 |
| Data removal | 74 (6.3) | 333 | 10 | 64.9 | 4.1 | 17.8 | 5.5 |
| Field | PRs | A % | B % | Agr. % | AC1 | |
|---|---|---|---|---|---|---|
| is_privacy_related , fix | 300 | 94.0 | 88.7 | 89.0 | .88 | .31 |
| is_privacy_related , non-fix | 350 | 96.3 | 81.4 | 81.7 | .80 | .14 |
| fix_type | 300 | 92.3 | 51.0 | 56.0 | .26 | .10 |
| gdpr_principles | 300 | 96.7 | 70.7 | 70.0 | .59 | .02 |
| gdpr_articles | 72 | 91.7 | 69.4 | 72.2 | .60 | .18 |
| fixed_commit | 300 | 93.3 | 84.0 | 80.7 | .76 | .06 |
Automated Code Review
Automated code review uses software analysis and AI to find defects, assess changes, and support pull-request decisions.
See all 57 papers