cs.CLOct 25, 2026

Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

Authors: Vladimir Beskorovainyi

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Professionals send working documents to external large language models. A common safeguard is reversible pseudonymisation: personal data are replaced with placeholders before the request and restored locally in the answer. We present an open benchmark of five existing open tools and a regular-expression baseline on Russian and UK English, and an open engine built on the two strongest tools. The benchmark excludes each tool's own authors' dataset from its verdict, compares the engine with each tool on identical datasets for detection, reports masking precision, and tests on a held-out set of synthetic documents whose files, hashes, decision rules, run plan and stopping rule were published before any run. Two local LLMs test whether placeholders survive. All evaluation data are synthetic, and the benchmark was run by the author of the engine; code, data, predictions and raw answers are released. On external datasets no existing tool meets all predefined thresholds: in UK English the identifier and leak thresholds are not met, in Russian the identifier threshold is borderline. On the last held-out set of 1,000 documents per language, the best existing tools leave a whole value visible in 61% (Russian) and 58% (UK) of documents. On these documents, from the generator it was developed on, the engine does so in 0.3% [0.1; 0.9] and 0.2% [0.1; 0.7], leaves any part of a value visible in 0.8% and 0.2%, and meets all detection thresholds in both languages. In UK English it also restores at least 99% of its placeholders after 760 answers. In Russian it restores only 94.6% [93.6; 95.6] of name placeholders, against a 99% threshold, mainly because the inflection it borrows from pii-guard produces wrong surname forms. The engine was developed over six earlier held-out sets from the same generator; real documents were not tested.

Figures & tables

ToolVersionConfiguration
pii-guardcommit 24230ab; model redmadrobot-rnd/rubert-base-pii-ner, rev. c802e8crules + ruBERT, Russian only; no organisation type
GLiNER Guard Omnihivetrace/gliner-guard-omni, rev. 763f8aa; gliner2 2.1.032 PII labels, threshold 0.4, 1,200-character windows with 200-character overlap
OpenAI privacy-filteropf commit f7f00caViterbi decoding; no organisation type
Presidio2.2.364spaCy large models, score threshold 0.3, added Russian identifier recognisers
Prompt Anonymizer0.3.3spaCy large model, IBAN recogniser enabled; English only; no NHS, NI or organisation type
Regex baselinethis workchecksum-validated patterns
Table 1: Tools and configurations. The runs used one environment, installed on 12 October 2026; from release 0.3.1 the public lock file pins the same package versions.
SetDocuments per languageEngineOrder of events attested byOutcome and what followed
1, 21000.1private logdevelopment data
31,0000.2 (cand.)private logUK leak threshold failed; errors analysed
41,0000.2private loground trip showed placeholders with a dropped bracket not restored; restorer changed (0.3)
51,0000.3public commit 8b8e933partly masked addresses and organisations, real Russian name failures; engine changed (0.4)
61,0000.4public commit 444c4bdfive thresholds met (pre-registered evaluator; Russian restoration overstated, Appendix A ); wrongly stored Russian surnames found after the run; engine changed (0.5)
71,0000.5public commit 13c42b8last set, published whatever the result (this paper)
Table 2: Held-out sets. Sets 5 to 7 were published with their files, hashes, decision rules and run plans before any run.
ThresholdRU: pii-guard (hivetrace)UK: GLiNER Guard (Nemotron)
Identifiers ≥\geq 99%97.7% [96.1; 99.0] undet. b96.4% [95.7; 97.1] not met
Names, addr., org. a ≥\geq 95%99.5% [98.7; 100.0] met98.7% [98.2; 99.1] met
Texts with a leak ≤\leq 2%0.7% [0.4; 1.2] met4.0% [3.4; 4.8] not met
Precision, fragments / chars96.5% / 98.0%89.3% / 88.7%
Table 3: Best existing tools on external datasets, excluding home datasets; “undet.” means undetermined. Recall and precision: document-level cluster bootstrap; leaks: Wilson. a The Russian external datasets contain no organisations. b Borderline: the upper end is 98.94–99.03% across bootstrap seeds 0 to 5.
ToolDocs with a leakDocs with a value not fully masked cIdentifiersNames, addr., org.Precision (fragments)
Russian, 1,000 documents
pii-roundtrip0.3% [0.1; 0.9]0.8% [0.4; 1.6]100.0% (17,245/17,245)99.93% [99.88; 99.98]98.4% [98.3; 98.5]
pii-guard a60.8% [57.7; 63.8]99.9%92.4% [91.7; 93.1]73.4% [72.8; 74.0]87.6% [87.5; 87.7]
GLiNER Guard Omni99.6% [99.0; 99.8]99.9%72.9% [72.2; 73.6]78.4% [77.5; 79.3]77.0% [76.7; 77.2]
Presidio87.1% [84.9; 89.0]100.0%91.1% [90.4; 91.9]60.8% [60.3; 61.3]76.1% [75.8; 76.3]
OpenAI privacy-filter a97.0% [95.7; 97.9]99.6%95.8% [95.4; 96.2]74.2% [73.3; 75.0]81.2% [81.1; 81.4]
Table 4: Held-out set 7 (engine version 0.5). Leaks: Wilson intervals over documents; recall and precision: document-level cluster bootstrap. a No organisation type. b No NHS number, NI number or organisation type. c Pre-registered for sets 6 and 7 (added after review on set 5): a document counts if any value of a threshold type is not fully masked (the recall criterion), for example a street or town left visible after the postcode was masked; Wilson intervals in the released results/strict_leaks.json .
ToolAll documentsWithout noiseNoisyWith hard formats
Russian (592 without noise, 408 noisy, 397 with hard formats)
pii-roundtrip0.3% [0.1; 0.9]0.3% [0.1; 1.2]0.2% [0.0; 1.4]0.3% [0.0; 1.4]
pii-guard60.8% [57.7; 63.8]41.2% [37.3; 45.2]89.2% [85.8; 91.9]89.9% [86.6; 92.5]
GLiNER Guard Omni99.6% [99.0; 99.8]99.3% [98.3; 99.7]100.0% [99.1; 100.0]100.0% [99.0; 100.0]
Presidio87.1% [84.9; 89.0]78.4% [74.9; 81.5]99.8% [98.6; 100.0]100.0% [99.0; 100.0]
OpenAI privacy-filter97.0% [95.7; 97.9]95.8% [93.8; 97.1]98.8% [97.2; 99.5]98.7% [97.1; 99.5]
Table 5: Documents of held-out set 7 with a leak, by noise and hard formats (Wilson intervals). Restricted to the types each tool supports, pii-guard leaks in 31.5% [28.7; 34.4] of documents, OpenAI privacy-filter in 87.1% [84.9; 89.0] of Russian and 33.0% [30.2; 36.0] of UK documents (both without organisations), and Prompt Anonymizer in 100% (without organisations, NHS and NI numbers). The breakdown was added after the run. Hard formats are inferred from the values (numbers in words, with dots, or with a failed checksum), and leaked types are assigned by matching value strings. Leaks by entity type are in the released results/leak_breakdown.holdout7.json .
Language, modelOcc. / failedAll placeholders ≥\geq 99%Name occ.Names ≥\geq 99%Answers with a failure
RU qwen3.5:9b12,547 / 14398.86% [98.64; 99.08]2,63894.58% [93.56; 95.65]99/380 [21.7; 30.8]
RU gemma4:12b14,045 / 14598.97% [98.77; 99.16]2,75694.74% [93.76; 95.74]100/380 [22.0; 31.0]
UK qwen3.5:9b12,232 / 0100% ( ≥\geq 99.03%)2,543100% ( ≥\geq 99.03%)0/380 [0.0; 0.97]
UK gemma4:12b13,876 / 199.99% [99.98; 100.0]3,16599.97% [99.90; 100.0]1/380 [0.01; 1.5]
Verdict, RU undetermined not met 
Verdict, UK met met 
Table 6: Round trip of pii-roundtrip 0.5 on held-out set 7 (190 documents per language, summary and reply, 380 answers per model), evaluator version 3. Restoration intervals: bootstrap over answers; with no failure the pre-registered bound 0.0251/n0.025^{1/n} over answers. Answers with a failure: Clopper–Pearson, in percent. Verdicts by the worse model.
Dataset (version)Identifiers ≥\geq 99%Names, addr., org. ≥\geq 95%Leaks ≤\leq 2%Strict ≤\leq 2% cRestoration ≥\geq 99% dPrecision
RU hivetrace a427/427, ≥\geq 99.1%404/404, ≥\geq 98.9%0.0% [0.0; 0.2]0.0% [0.0; 0.2]–89.2%
RU redmadrobot a98.7% [98.0; 99.3]98.3% [97.5; 99.0]1.1% [0.7; 1.5]1.3% [1.0; 1.8]–76.4%
UK Nemotron b99.97% [99.9; 100.0] f98.3% [97.9; 98.8] met0.9% [0.6; 1.3] met2.1% [1.7; 2.7]–93.5%
original IDs only99.8% [98.996; 99.995] undet.–––––
RU set 4 (0.2)all, ≥\geq 99.63%99.8% [99.7; 99.9]0.2% [0.1; 0.7]2.5% [1.7; 3.7]0 failures, 40 answers, ≥\geq 91.1%98.5%
UK set 4 (0.2)all, ≥\geq 99.63%98.7% [98.5; 98.9]0.9% [0.5; 1.7]15.5% [13.4; 17.9]99.6% [99.2; 99.9]99.8%
Table 7: pii-roundtrip on every dataset; precision is the share of masked fragments that touch personal data. Counts with ≥\geq are zero-failure results with exact one-sided 97.5% lower bounds over the documents (or texts) that contain such values, rounded down; on held-out sets 4 to 7 no document has a missed identifier (17,489, 12,566, 17,298, 12,636, 17,291, 12,646, 17,245 and 12,636 identifiers). Verdicts are given only for set 7, the last pre-registered test, and for the UK external data; for set 7 the restoration cell gives all placeholders and name placeholders, each against 99%. a Home dataset of a component of the engine (GLiNER Guard and pii-guard respectively); reference only. b Both subsets, 3,248 of 3,249 identifiers; in-distribution for GLiNER Guard. c Pre-registered for sets 6 and 7 and applied only there. d Model with the lower rate; with zero failures in both models, the bound for the number of answers. e Overstated: the evaluator of that set could not see wrongly stored or wrongly inflected Russian names (Appendix A ). f No verdict: 83% of these identifiers were injected by the generator that shares the engine’s checksum code; the verdict is given on the original identifiers.
Date (release)Data seenChangeEffect on the engine
19 Oct (0.2.1)set 4Each issued placeholder occurrence scored onceup (fewer double counts)
19 Oct (0.2.2)set 4Bracket-rewritten tags counted as issued and alterednone
19 Oct (0.2.2)set 4Recall and precision by cluster bootstrap; “borderline” rule over seedswider intervals
19 Oct (0.2.2)set 4Restoration threshold judged on clean answersdown
19 Oct (0.3.0, before set 5)set 4Back to occurrences, bootstrap over answers; 380 answers for the engineup
22 Oct (0.3.1)sets 4, 5Leak metric counts dotted numbersnone (other tools worse)
Table 8: Changes to the rules of evaluation made after data had been seen (dates in October 2026). Two changes that affect only the other tools (privacy-filter listed without an organisation type; the restoration verdict of pii-guard on set 5 removed) and smaller presentational changes are in results/rule_changes.md . The table starts with release 0.2.1; earlier corrections of the leak metric (Appendix C ) lowered the leak rates of every tool.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

 qwen3.5:9bgemma4:12b
Configurationans. / occ. / fail.restorationans. / occ. / fail.restoration
Russian
pii-guard tags, own15/40 / 542 / 199.8% [99.4; 100.0]40/40 / 1,564 / 0100% ( ≥\geq 91.1%)
re-parsed tolerantly15/40 / 542 / 0100% ( ≥\geq 78.1%)40/40 / 1,564 / 0100% ( ≥\geq 91.1%)
pii-guard tags, generic26/40 / 1,060 / 399.7% [99.2; 100.0]40/40 / 1,703 / 2998.3% [94.6; 100.0]
re-parsed tolerantly26/40 / 1,060 / 0100% ( ≥\geq 86.7%)40/40 / 1,703 / 0100% ( ≥\geq 91.1%)
Appendix
Table 9: Round trip on held-out set 5, and on set 6 for engine version 0.4 (summary and reply; 40 answers per cell; a 380 answers; set 5 scored with evaluator version 1, set 6 with version 2). b Overstated: some Russian names were stored in a wrong form, which the evaluator cannot see (Appendix A ); this also applies to the Russian rows of engine 0.3 on set 5, whose name placeholders score 89.1–89.3% with evaluator version 3. “Own”, “wrapper” and “generic” name the system prompt. “Ans.” counts answers with placeholders, “occ.” tool-issued placeholder occurrences in the answers and “fail.” those that failed. Intervals: bootstrap over answers; an upper bound of 100.0 is the bootstrap’s ceiling. Values in parentheses are the pre-registered lower bounds for zero failures, 0.0251/n0.025^{1/n} with nn the answers with placeholders.
More in this topic

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

Explore similar work

CardsList
Oct 16, 2026

Veil-Then-Reveal: Protecting PII Privacy via Reversible Pseudo-Anonymization in Zero-Trust User--LLM Interaction

Large language models (LLMs) increasingly process sensitive documents and personal information, yet sending prompts containing personally identifiable information (PII) to remote providers exposes data subjects and organizations to risks from logging, retention, model training, and unintended disclosure. Existing anonymization methods often mask or remove PII, while conventional encryption eliminates the lexical and structural cues needed for effective LLM processing. We present Veil-then-Reveal (VtR), a user-side PII pseudo-anonymization framework for privacy-preserving LLM interaction under a zero-trust threat model. VtR treats remote LLM providers as untrusted and sends only pseudo-anonymized prompts, keeping plaintext PII on the user. VtR first detects PII and replaces it with realistic, same-type, in-vocabulary surrogates before prompts leave the user's device, preserving their linguistic and structural properties. It combines format-preserving encryption (FPE) for structured PII, such as phone and credit-card numbers, with a keyed-hash dictionary and secure vault for semantic PII, such as names, locations, and organizations. Session-specific mappings provide cross-session unlinkability, while user-side restoration reversibly replaces surrogates with their original values in LLM responses. We further introduce a rank-class FPE strategy for dates that preserves format and ambiguity while reducing cipher evaluations. Evaluation on three datasets containing 11,322 annotated PII instances achieves 97.9% detection recall, 95.2% protection of detected values, and 99.9% round trip; 100% of echoed surrogates end-to-end round-trip restoration.

Mohamed Elmahallawy, Mohamed Shaaban, Thomas E. Carroll

School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA · Pacific Northwest National Laboratory, Richland, WA 99354, USA

Privacy-Preserving Language Models
AskPDF
Jan 30, 2026

AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs

Modern LLMs are increasingly accessed via black-box APIs, requiring users to transmit sensitive prompts, outputs, and fine-tuning data to external providers, creating a critical privacy risk at the API boundary. We introduce AlienLM, a deployable API-only \cradd{exposure-reduction layer that reduces plaintext exposure} by translating text into an Alien Language via a vocabulary-scale bijection, enabling lossless recovery on the client side. Using only standard fine-tuning APIs, Alien Adaptation Training (AAT) adapts target models to operate directly on alienized inputs. Across four LLM backbones and seven benchmarks, AlienLM retains over 81% of plaintext-oracle performance on average, substantially outperforming random-bijection and character-level baselines. Under adversaries with access to model weights, corpus statistics, and learning-based inverse translation, recovery attacks reconstruct fewer than 0.22% of alienized tokens. Our results demonstrate a practical pathway for \cradd{privacy-aware} LLM deployment under API-only access, substantially reducing plaintext exposure while maintaining task performance. Code and data are available at https://github.com/KimJaehee0725/AlienLM.

Jaehee Kim, Pilsung Kang

Department of Industrial Engineering, Seoul National University, South Korea.

Privacy-Preserving Language ModelsPrivacy Leakage in Language Models
AskPDF
Aug 18, 2026

Redakto - The Incognito Tab for LLMs

Large Language Models (LLMs) are being increasingly used in everyday applications. A major challenge in the context of LLMs or Artificial Intelligence (AI) in general is to ensure privacy when using them, meaning that personally identifiable information (PII) is removed from any text that enters an LLM. These challenges have become more urgent with novel EU legislation. Uncertainty around LLM usage with respect to privacy concerns in EU countries can be a major blocker for the speed of innovation and transfer from research to applications. Here we present Redakto, a tool that can be used for anonymizing text prior to feeding it to an LLM or other downstream text processing. We provide state-of-the-art functionalities for both redaction of PII but also when used for pseudonymization. These functionalities are exposed such that they can easily be used by end-users, through the Redakto web application, and by developers and researchers, via REST APIs and model context protocol (MCP) hooks. The implementation is fully open source, requires modest compute resources, and can be readily deployed on local hardware. In contrast to prior work and in order to better assess the quality of the anonymized texts, we conduct extensive empirical evaluations on textual data from legal and medical domain with respect to both privacy and utility of the redacted texts. Our empirical results demonstrate that the texts anonymized with different redaction strategies achieve utility scores on par with the original texts, suggesting that anonymization with Redakto can be used for LLM tasks without substantial negative impact for the tasks we explored.

Saurav Kumar Saha, Tom Röhr, Felix Bießmann

Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany · Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany

Privacy Protection for LLM AgentsPrivacy-Preserving Language Models
AskPDF