Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

Vladimir Beskorovainyi

cs.CLOct 25, 2026

Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

Authors: Vladimir Beskorovainyi

Abstract

Professionals send working documents to external large language models. A common safeguard is reversible pseudonymisation: personal data are replaced with placeholders before the request and restored locally in the answer. We present an open benchmark of five existing open tools and a regular-expression baseline on Russian and UK English, and an open engine built on the two strongest tools. The benchmark excludes each tool's own authors' dataset from its verdict, compares the engine with each tool on identical datasets for detection, reports masking precision, and tests on a held-out set of synthetic documents whose files, hashes, decision rules, run plan and stopping rule were published before any run. Two local LLMs test whether placeholders survive. All evaluation data are synthetic, and the benchmark was run by the author of the engine; code, data, predictions and raw answers are released. On external datasets no existing tool meets all predefined thresholds: in UK English the identifier and leak thresholds are not met, in Russian the identifier threshold is borderline. On the last held-out set of 1,000 documents per language, the best existing tools leave a whole value visible in 61% (Russian) and 58% (UK) of documents. On these documents, from the generator it was developed on, the engine does so in 0.3% [0.1; 0.9] and 0.2% [0.1; 0.7], leaves any part of a value visible in 0.8% and 0.2%, and meets all detection thresholds in both languages. In UK English it also restores at least 99% of its placeholders after 760 answers. In Russian it restores only 94.6% [93.6; 95.6] of name placeholders, against a 99% threshold, mainly because the inflection it borrows from pii-guard produces wrong surname forms. The engine was developed over six earlier held-out sets from the same generator; real documents were not tested.

Figures & tables

Tool	Version	Configuration
pii-guard	commit 24230ab; model redmadrobot-rnd/rubert-base-pii-ner, rev. c802e8c	rules + ruBERT, Russian only; no organisation type
GLiNER Guard Omni	hivetrace/gliner-guard-omni, rev. 763f8aa; gliner2 2.1.0	32 PII labels, threshold 0.4, 1,200-character windows with 200-character overlap
OpenAI privacy-filter	opf commit f7f00ca	Viterbi decoding; no organisation type
Presidio	2.2.364	spaCy large models, score threshold 0.3, added Russian identifier recognisers
Prompt Anonymizer	0.3.3	spaCy large model, IBAN recogniser enabled; English only; no NHS, NI or organisation type
Regex baseline	this work	checksum-validated patterns

Table 1: Tools and configurations. The runs used one environment, installed on 12 October 2026; from release 0.3.1 the public lock file pins the same package versions.

Set	Documents per language	Engine	Order of events attested by	Outcome and what followed
1, 2	100	0.1	private log	development data
3	1,000	0.2 (cand.)	private log	UK leak threshold failed; errors analysed
4	1,000	0.2	private log	round trip showed placeholders with a dropped bracket not restored; restorer changed (0.3)
5	1,000	0.3	public commit 8b8e933	partly masked addresses and organisations, real Russian name failures; engine changed (0.4)
6	1,000	0.4	public commit 444c4bd	five thresholds met (pre-registered evaluator; Russian restoration overstated, Appendix A ); wrongly stored Russian surnames found after the run; engine changed (0.5)
7	1,000	0.5	public commit 13c42b8	last set, published whatever the result (this paper)

Table 2: Held-out sets. Sets 5 to 7 were published with their files, hashes, decision rules and run plans before any run.

Threshold	RU: pii-guard (hivetrace)	UK: GLiNER Guard (Nemotron)
Identifiers $\geq$ 99%	97.7% [96.1; 99.0] undet. b	96.4% [95.7; 97.1] not met
Names, addr., org. a $\geq$ 95%	99.5% [98.7; 100.0] met	98.7% [98.2; 99.1] met
Texts with a leak $\leq$ 2%	0.7% [0.4; 1.2] met	4.0% [3.4; 4.8] not met
Precision, fragments / chars	96.5% / 98.0%	89.3% / 88.7%

Table 3: Best existing tools on external datasets, excluding home datasets; “undet.” means undetermined. Recall and precision: document-level cluster bootstrap; leaks: Wilson. a The Russian external datasets contain no organisations. b Borderline: the upper end is 98.94–99.03% across bootstrap seeds 0 to 5.

Tool	Docs with a leak	Docs with a value not fully masked c	Identifiers	Names, addr., org.	Precision (fragments)
Russian, 1,000 documents
pii-roundtrip	0.3% [0.1; 0.9]	0.8% [0.4; 1.6]	100.0% (17,245/17,245)	99.93% [99.88; 99.98]	98.4% [98.3; 98.5]
pii-guard a	60.8% [57.7; 63.8]	99.9%	92.4% [91.7; 93.1]	73.4% [72.8; 74.0]	87.6% [87.5; 87.7]
GLiNER Guard Omni	99.6% [99.0; 99.8]	99.9%	72.9% [72.2; 73.6]	78.4% [77.5; 79.3]	77.0% [76.7; 77.2]
Presidio	87.1% [84.9; 89.0]	100.0%	91.1% [90.4; 91.9]	60.8% [60.3; 61.3]	76.1% [75.8; 76.3]
OpenAI privacy-filter a	97.0% [95.7; 97.9]	99.6%	95.8% [95.4; 96.2]	74.2% [73.3; 75.0]	81.2% [81.1; 81.4]

Table 4: Held-out set 7 (engine version 0.5). Leaks: Wilson intervals over documents; recall and precision: document-level cluster bootstrap. a No organisation type. b No NHS number, NI number or organisation type. c Pre-registered for sets 6 and 7 (added after review on set 5): a document counts if any value of a threshold type is not fully masked (the recall criterion), for example a street or town left visible after the postcode was masked; Wilson intervals in the released results/strict_leaks.json .

Tool	All documents	Without noise	Noisy	With hard formats
Russian (592 without noise, 408 noisy, 397 with hard formats)
pii-roundtrip	0.3% [0.1; 0.9]	0.3% [0.1; 1.2]	0.2% [0.0; 1.4]	0.3% [0.0; 1.4]
pii-guard	60.8% [57.7; 63.8]	41.2% [37.3; 45.2]	89.2% [85.8; 91.9]	89.9% [86.6; 92.5]
GLiNER Guard Omni	99.6% [99.0; 99.8]	99.3% [98.3; 99.7]	100.0% [99.1; 100.0]	100.0% [99.0; 100.0]
Presidio	87.1% [84.9; 89.0]	78.4% [74.9; 81.5]	99.8% [98.6; 100.0]	100.0% [99.0; 100.0]
OpenAI privacy-filter	97.0% [95.7; 97.9]	95.8% [93.8; 97.1]	98.8% [97.2; 99.5]	98.7% [97.1; 99.5]

Table 5: Documents of held-out set 7 with a leak, by noise and hard formats (Wilson intervals). Restricted to the types each tool supports, pii-guard leaks in 31.5% [28.7; 34.4] of documents, OpenAI privacy-filter in 87.1% [84.9; 89.0] of Russian and 33.0% [30.2; 36.0] of UK documents (both without organisations), and Prompt Anonymizer in 100% (without organisations, NHS and NI numbers). The breakdown was added after the run. Hard formats are inferred from the values (numbers in words, with dots, or with a failed checksum), and leaked types are assigned by matching value strings. Leaks by entity type are in the released results/leak_breakdown.holdout7.json .

Language, model	Occ. / failed	All placeholders $\geq$ 99%	Name occ.	Names $\geq$ 99%	Answers with a failure
RU qwen3.5:9b	12,547 / 143	98.86% [98.64; 99.08]	2,638	94.58% [93.56; 95.65]	99/380 [21.7; 30.8]
RU gemma4:12b	14,045 / 145	98.97% [98.77; 99.16]	2,756	94.74% [93.76; 95.74]	100/380 [22.0; 31.0]
UK qwen3.5:9b	12,232 / 0	100% ( $\geq$ 99.03%)	2,543	100% ( $\geq$ 99.03%)	0/380 [0.0; 0.97]
UK gemma4:12b	13,876 / 1	99.99% [99.98; 100.0]	3,165	99.97% [99.90; 100.0]	1/380 [0.01; 1.5]
Verdict, RU		undetermined		not met
Verdict, UK		met		met

Table 6: Round trip of pii-roundtrip 0.5 on held-out set 7 (190 documents per language, summary and reply, 380 answers per model), evaluator version 3. Restoration intervals: bootstrap over answers; with no failure the pre-registered bound

0.025^{1/n}

over answers. Answers with a failure: Clopper–Pearson, in percent. Verdicts by the worse model.

Dataset (version)	Identifiers $\geq$ 99%	Names, addr., org. $\geq$ 95%	Leaks $\leq$ 2%	Strict $\leq$ 2% c	Restoration $\geq$ 99% d	Precision
RU hivetrace a	427/427, $\geq$ 99.1%	404/404, $\geq$ 98.9%	0.0% [0.0; 0.2]	0.0% [0.0; 0.2]	–	89.2%
RU redmadrobot a	98.7% [98.0; 99.3]	98.3% [97.5; 99.0]	1.1% [0.7; 1.5]	1.3% [1.0; 1.8]	–	76.4%
UK Nemotron b	99.97% [99.9; 100.0] f	98.3% [97.9; 98.8] met	0.9% [0.6; 1.3] met	2.1% [1.7; 2.7]	–	93.5%
original IDs only	99.8% [98.996; 99.995] undet.	–	–	–	–	–
RU set 4 (0.2)	all, $\geq$ 99.63%	99.8% [99.7; 99.9]	0.2% [0.1; 0.7]	2.5% [1.7; 3.7]	0 failures, 40 answers, $\geq$ 91.1%	98.5%
UK set 4 (0.2)	all, $\geq$ 99.63%	98.7% [98.5; 98.9]	0.9% [0.5; 1.7]	15.5% [13.4; 17.9]	99.6% [99.2; 99.9]	99.8%

Table 7: pii-roundtrip on every dataset; precision is the share of masked fragments that touch personal data. Counts with

\geq

are zero-failure results with exact one-sided 97.5% lower bounds over the documents (or texts) that contain such values, rounded down; on held-out sets 4 to 7 no document has a missed identifier (17,489, 12,566, 17,298, 12,636, 17,291, 12,646, 17,245 and 12,636 identifiers). Verdicts are given only for set 7, the last pre-registered test, and for the UK external data; for set 7 the restoration cell gives all placeholders and name placeholders, each against 99%. a Home dataset of a component of the engine (GLiNER Guard and pii-guard respectively); reference only. b Both subsets, 3,248 of 3,249 identifiers; in-distribution for GLiNER Guard. c Pre-registered for sets 6 and 7 and applied only there. d Model with the lower rate; with zero failures in both models, the bound for the number of answers. e Overstated: the evaluator of that set could not see wrongly stored or wrongly inflected Russian names (Appendix A ). f No verdict: 83% of these identifiers were injected by the generator that shares the engine’s checksum code; the verdict is given on the original identifiers.

Date (release)	Data seen	Change	Effect on the engine
19 Oct (0.2.1)	set 4	Each issued placeholder occurrence scored once	up (fewer double counts)
19 Oct (0.2.2)	set 4	Bracket-rewritten tags counted as issued and altered	none
19 Oct (0.2.2)	set 4	Recall and precision by cluster bootstrap; “borderline” rule over seeds	wider intervals
19 Oct (0.2.2)	set 4	Restoration threshold judged on clean answers	down
19 Oct (0.3.0, before set 5)	set 4	Back to occurrences, bootstrap over answers; 380 answers for the engine	up
22 Oct (0.3.1)	sets 4, 5	Leak metric counts dotted numbers	none (other tools worse)

Table 8: Changes to the rules of evaluation made after data had been seen (dates in October 2026). Two changes that affect only the other tools (privacy-filter listed without an organisation type; the restoration verdict of pii-guard on set 5 removed) and smaller presentational changes are in results/rule_changes.md . The table starts with release 0.2.1; earlier corrections of the leak metric (Appendix C ) lowered the leak rates of every tool.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	qwen3.5:9b		gemma4:12b
Configuration	ans. / occ. / fail.	restoration	ans. / occ. / fail.	restoration
Russian
pii-guard tags, own	15/40 / 542 / 1	99.8% [99.4; 100.0]	40/40 / 1,564 / 0	100% ( $\geq$ 91.1%)
re-parsed tolerantly	15/40 / 542 / 0	100% ( $\geq$ 78.1%)	40/40 / 1,564 / 0	100% ( $\geq$ 91.1%)
pii-guard tags, generic	26/40 / 1,060 / 3	99.7% [99.2; 100.0]	40/40 / 1,703 / 29	98.3% [94.6; 100.0]
re-parsed tolerantly	26/40 / 1,060 / 0	100% ( $\geq$ 86.7%)	40/40 / 1,703 / 0	100% ( $\geq$ 91.1%)

Appendix

Table 9: Round trip on held-out set 5, and on set 6 for engine version 0.4 (summary and reply; 40 answers per cell; a 380 answers; set 5 scored with evaluator version 1, set 6 with version 2). b Overstated: some Russian names were stored in a wrong form, which the evaluator cannot see (Appendix A ); this also applies to the Russian rows of engine 0.3 on set 5, whose name placeholders score 89.1–89.3% with evaluator version 3. “Own”, “wrapper” and “generic” name the system prompt. “Ans.” counts answers with placeholders, “occ.” tool-issued placeholder occurrences in the answers and “fail.” those that failed. Intervals: bootstrap over answers; an upper bound of 100.0 is the bootstrap’s ceiling. Values in parentheses are the pre-registered lower bounds for zero failures,

0.025^{1/n}

with

n

the answers with placeholders.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

Veil-Then-Reveal: Protecting PII Privacy via Reversible Pseudo-Anonymization in Zero-Trust User--LLM Interaction
Oct 16, 2026Mohamed Elmahallawy, Mohamed Shaaban, Thomas E. CarrollPrivacy-Preserving Language Models
School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA · Pacific Northwest National Laboratory, Richland, WA 99354, USA
AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs
Jan 30, 2026Jaehee Kim, Pilsung KangPrivacy-Preserving Language Models Privacy Leakage in Language Models
Department of Industrial Engineering, Seoul National University, South Korea.
Redakto - The Incognito Tab for LLMs
Aug 18, 2026Saurav Kumar Saha, Tom Röhr, Felix BießmannPrivacy Protection for LLM Agents Privacy-Preserving Language Models
Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany · Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany

Oct 16, 2026J/K move · Enter open · S save

Veil-Then-Reveal: Protecting PII Privacy via Reversible Pseudo-Anonymization in Zero-Trust User--LLM Interaction

Mohamed Elmahallawy, Mohamed Shaaban, Thomas E. Carroll

School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA · Pacific Northwest National Laboratory, Richland, WA 99354, USA

Privacy-Preserving Language Models

Open paper AskPDF arXiv

Large language models (LLMs) increasingly process sensitive documents and personal information, yet sending prompts containing personally identifiable information (PII) to remote providers exposes data subjects and organizations to risks from logging, retention, model training, and unintended disclosure. Existing anonymization methods often mask or remove PII, while conventional encryption eliminates the lexical and structural cues needed for effective LLM processing. We present Veil-then-Reveal (VtR), a user-side PII pseudo-anonymization framework for privacy-preserving LLM interaction under a zero-trust threat model. VtR treats remote LLM providers as untrusted and sends only pseudo-anonymized prompts, keeping plaintext PII on the user. VtR first detects PII and replaces it with realistic, same-type, in-vocabulary surrogates before prompts leave the user's device, preserving their linguistic and structural properties. It combines format-preserving encryption (FPE) for structured PII, such as phone and credit-card numbers, with a keyed-hash dictionary and secure vault for semantic PII, such as names, locations, and organizations. Session-specific mappings provide cross-session unlinkability, while user-side restoration reversibly replaces surrogates with their original values in LLM responses. We further introduce a rank-class FPE strategy for dates that preserves format and ambiguity while reducing cipher evaluations. Evaluation on three datasets containing 11,322 annotated PII instances achieves 97.9% detection recall, 95.2% protection of detected values, and 99.9% round trip; 100% of echoed surrogates end-to-end round-trip restoration.

Abstract

Privacy Leakage in Language Models

Explore similar work

Veil-Then-Reveal: Protecting PII Privacy via Reversible Pseudo-Anonymization in Zero-Trust User--LLM Interaction

AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs

Redakto - The Incognito Tab for LLMs