cs.CLMar 9, 2026
SaveThe Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques
Organizations: Trustworthy Human Language Technologies www.trusthlt.org · Technical University of Darmstadt, Germany · Research Center for Trustworthy Data Science and Security, Ruhr University Bochum, Germany
Abstract
Removing personally identifiable information (PII) from texts is necessary to comply with various data protection regulations and to enable data sharing without compromising privacy. However, recent works show that documents sanitized by PII removal techniques are vulnerable to reconstruction attacks. Yet, we suspect that the reported success of these attacks is largely overestimated. We critically analyze the evaluation of existing attacks and find that data leakage and data contamination are not properly mitigated, leaving the question whether or not PII removal techniques truly protect privacy in real-world scenarios unaddressed. We investigate possible data sources and attack setups that avoid data leakage and conclude that only truly private data can allow us to objectively evaluate vulnerabilities in PII removal techniques. However, access to private data is heavily restricted - and for good reasons - which also means that the public research community cannot address this problem in a transparent, reproducible, and trustworthy manner.
Figures & tables
| Technique | Example |
|---|---|
| Original | Hi, I’m Jane Doe, and I live at 123 Maple Street, Springfield. You can reach me at (555) 123-4567 or email me at [email protected]. |
| Redaction | Hi, I’m , and I live at 123 , . You can reach me at or email me at . |
| Masking | Hi, I’m ********, and I live at 123 ************, ***********. You can reach me at ************** or email me at ***************om. |
| Replacement (Tag) | Hi, I’m <PERSON>, and I live at 123 <LOCATION>, <LOCATION>. You can reach me at <PHONE_NUMBER> or email me at <EMAIL_ADDRESS>. |
| Pseudonymization | Hi, my name is Alex Carter, and I live at 123 Oak Drive, Rivertown. You can reach me at (206) 482-7743 or via email at [email protected]. |
Table 1: We generated a message from a ficticious person with ChatGPT 4o mini ( https://chatgpt.com/share/681de483-181c-800e-ae40-88cc2fefb0f9 ) and applied common PII removal techniques to the same input sentence with Microsoft Presidio ( https://huggingface.co/spaces/presidio/presidio_demo ).
| System | Find which celebrity matches the following description and match the missing information |
|---|---|
| User | PERSON_FIRSTNAME_1 PERSON_LASTNAME_2 is a NUMERIC_4 year old actress, born in LOCATION_2 but raised in LOCATION_1. PRONOUN is most famously known for playing PERSON_FIRSTNAME_2 PERSON_LASTNAME_1 in the OTHER_IDENTIFYING_ATTRIBUTE_2 franchise. […] PRONOUN is very involved in activism and is a very outspoken feminist. |
| Assistant | Emma Watson is a 31 year old actress, born in Paris but raised in the UK . She is most famously known for playing Hermione Granger in the Harry Potter franchise. […] she is very involved in activism and is a very outspoken feminist. |
Table 2: Shortened example from Table 5, Patsakis and Lykousas (2023) of GPT-4 recognizing Emma Watson from a PII removed text ( https://chatgpt.com/share/6825cf22-06e4-800e-9de1-edf923c9dc23 ).
Figure 1: Solving the task by Patsakis and Lykousas (2023) by querying ChatGPT-4o with a few keywords only (cf. Table 2 ).
| Gold standard | Emma Watson is a 31 year old actress, born in Paris but raised in the UK . She is most famously known for playing Hermione Granger in the Harry Potter franchise. […] she is very involved in activism and is a very outspoken feminist. |
|---|---|
| Alternative solution | Angelina Jolie is a 49 year old actress, born in Los Angeles but raised in the Palisades, New York . She is most famously known for playing Lara Croft in the Lara Croft franchise. […] she is very involved in activism and is a very outspoken feminist. |
Table 3: Both Emma Watson and Angelina Jolie would be a valid solution of the PII reconstruction attack by Patsakis and Lykousas (2023) .
| Loc. | Org. | Person | NRP | Time | URL | Tel. | Misc. | Total | ||
|---|---|---|---|---|---|---|---|---|---|---|
| YT | 747 | 151 | 110 | 102 | 4 | 4 | 0 | 1 | 1 | 1120 |
| CZ | 294 | 190 | 243 | 2 | 23 | 13 | 16 | 3 | 21 | 805 |
Table 4: # of PII spans by entity type in our two datasets (YT = YouTube transcripts; CZ = Czech court announcements). Abbreviations: Loc. = location, Org. = organization, NRP = nationality/religious/political group, Time = date/time, URL = web address, Tel. = telephone number, Misc. = other numerical PII, such as bank account numbers.
| Model prompt |
|---|
| You are a privacy risk evaluator. Input: |
| A text containing masked spans like [MASK_0] , [MASK_1] , … (any number; indices may be non-contiguous). Task A — Hint audit (leakage check): |
| Scan the text for unmasked information that could help infer the masked spans. Hints include: unmasked proper nouns, partial names, unique locations/landmarks, organization names, routes, flight/train identifiers, exact dates, distinctive numbers, usernames, emails, phone numbers, or any unusually specific phrase. Quote each hint exactly as it appears in the text (verbatim substring). If there are no clear leaks, still list weaker contextual hints, but label them as "contextual" rather than "direct leak" . Task B — Reconstruction: |
| Identify every unique mask token in the text. For each mask, silently derive constraints from local context + the hints you listed. Produce EXACTLY 3 candidate strings per mask (ranked: most likely first). Rules: |
| Do NOT output your reasoning steps. Do NOT add any keys beyond the required JSON schema. Guesses must be only the missing span text (no extra words). Match capitalization/punctuation implied by context. Use outside knowledge only if strongly implied by hints in the text. Output JSON only, with this exact schema: |
| { |
Table 5: Prompt used for the PII reconstruction attack.
| Total | MaxDoc | Loc. | Org. | Person | NRP | Time | |
|---|---|---|---|---|---|---|---|
| YT | 0.191 | 0.515 | 0.194 | 0.139 | 0.155 | 0.294 | 0.250 |
| CZ | 0.055 | 0.364 | 0.119 | 0.037 | 0.008 | 0.000 | 0.000 |
Table 6: Top-3 exact match rate (EM@3) by PII category and dataset. Total displays EM@3 over all masked PII within a dataset. MaxDoc denotes the maximum document-level EM@3 observed in each dataset. We omit four categories (URL, Tel., Email, Misc.), as they have EM@3 in both datasets.
| Reconstruction example from YT |
|---|
| Hi everyone! [MASK_0] here Today, we’re heading to [MASK_1] for a 3-day trip with our friend visiting from [MASK_2] The hotel is spacious with a kitchen, and the kids are so excited At [MASK_3] , we even spotted an ad for the Demon Slayer movie - so exciting! We’ll also visit some iconic [MASK_4] spots - [MASK_5] , the I Love NY gift shop, and more! And now, our Day 1 adventure begins! I’m three years old Wow, beautiful! We are here, mama! We are in [MASK_6] ! We are not in [MASK_7] That’s a huge bed! No more monkeys jumping on the bed Don’t do that. [MASK_8] is going to be sad This is a great location near [MASK_9] Mama, there is a phone! [MASK_10] likes that Yes, it’s wonderful Be gentle please Good night Good morning! After a short rest, we walked to [MASK_11] - about 20 minutes from our hotel Mail man! Look! (The bird) had a bath I’m a birdie What’s that? [MASK_12] ’s getting a piggyback ride Be careful, pigeon It’s a place that truly represents [MASK_13] - filled with giant billboards and crowds of people everywhere Look! That’s mama! Is that me?! There are amazing people They look cool, huh She said hi to me That’s wonderful Next stop - the [MASK_14] store! It smells sweet here like chocolate One two, one two Do you want to do this? Which one do you want? Do you like this one? Not too much, OK? Don’t eat it yet It’s a little bit rainy! That’s right Then we stopped by the I Love NY gift shop Are they friends? Yeah! There there We stopped by [MASK_15] to grab dinner You are tired, aren’t you? No, I’m not sleepy Really? Beautiful! Thank you for watching until the end! If you enjoyed this video, don’t forget to like and subscribe! And stay tuned for Day 2 of our [MASK_16] adventure Vlog! Thank you for watching! |
| Hints: I Love NY gift shop; Demon Slayer movie; giant billboards and crowds of people everywhere; sweet here like chocolate |
Table 7: Example masked document and extracted hints (top) and gold spans with the attack model’s top-3 reconstruction candidates (bottom). Person names were manually replaced with italic placeholders.
| Reconstruction example from CZ |
|---|
| Posted on: October 30, 2018 Deadline for removal: November 30, 2018 Removed on: file number 1 Notice for posting on the court’s official notice board pursuant to Section 49(4) of the Code of Civil Procedure. Addressee: [MASK_0] , born [MASK_1] Address to which the document is to be delivered: [MASK_2] number , number [MASK_3] number - location Court that submitted the document for delivery: [MASK_4] for [MASK_5] 3 Document to be delivered: file number - decision no. number , lh. [MASK_6] As the addressee was not present at the time of delivery of the court document and as it was not possible to deliver the court document to a person authorized to accept it, the court document was deposited. As the addressee did not collect the document within 10 days, the tenth day of this period is considered the date of delivery according to the law. Since after the expiry of this period it was not possible to place the document in the addressee’s mailbox or other mailbox used by the addressee, it was returned to the sending court. You can collect the stored shipment at the information center of [MASK_7] , [MASK_8] , [MASK_9] , during the following hours: Mon from 8:00 a.m. to 11:00 a.m. and from 12:00 p.m. to 4:30 p.m. Tue from 8:00 a.m. to 11:00 a.m. and from 12:00 p.m. to 4:00 p.m. Wed from 8:00 a.m. to 11:00 a.m. and from 12:00 p.m. to 5:00 p.m. Thu from 8:00 a.m. to 11:00 a.m. and from 12:00 p.m. to 4:00 p.m. Friday from 8:00 a.m. to 11:00 a.m. and from 12:00 p.m. to 2:30 p.m. Please bring your ID card or other proof of identity with you. Name and surname of the court delivery person: [MASK_10] , [MASK_11] Date: October 30, 2018 Signature of court delivery agent |
| Hints: Posted on: October 30, 2018; Deadline for removal: November 30, 2018; File number; location ; information center of; Mon from 8:00 a.m. |
Table 8: Example masked court announcement (top), extracted hints, and gold spans with the attack model’s top-3 reconstruction candidates (bottom). Italic text was manually removed to protect the privacy of individuals before publication.
More in this topic
Data Leakage
Data leakage is unintended information flow into model training, evaluation, or outputs that inflates results or exposes sensitive data.
See all 125 papers