PrivacyPR benchmarks privacy-aware code review on real fixes
The point
PrivacyPR collects 4,131 merged GitHub pull requests from 1,469 repositories, created between June 2018 and January 2024. Each includes commits, diffs, and review discussion, plus labels for privacy relevance, fix type, and GDPR principles.
Why it matters
For builders, privacy review improves sharply when agents are told to check privacy.
Key details
- Verified split: 250 privacy fixes and 250 non-fix PRs, manually checked
- On Small, two agents detected 42 to 51 percent generically, but 88 to 91 percent when asked to check privacy
- Agents also flagged 11 to 15 percent of non-fix PRs as defects
Original ~6 min → 25 sec here Source: arXiv cs.AI, Nov 16, 2026