Protegrity-PII Technical Report

Hardeep Bassi; Hui Ma; Luís Seabra; Koowum Joshi; Daniel Pereira; Averell Gatton

cs.CLOct 21, 2026

Protegrity-PII Technical Report

Authors: Hardeep Bassi, Hui Ma, Luís Seabra, Koowum Joshi, Daniel Pereira, Averell Gatton

Organizations: Protegrity 1 Menlo Park, CA, USA · Protegrity 2 Stamford, CT, USA · Protegrity 3 Lisbon, Portugal · Arizona State University, Tempe, AZ, USA

Abstract

We present Protegrity-PII, a new PII decision model for structured and unstructured text. Protegrity-PII achieves a typed recall of 0.915 with a precision of 0.907 on a new benchmark we propose, PII-SCOPE. Our model processes text in 0.49 seconds per million characters at $0.0005 per million characters on a single RTX 6000 using batched inference. Due in part to the expansive nature of the entity type taxonomy used in training, Protegrity-PII achieves superior out-of-distribution generalization when benchmarked against a selection of state of the art models and datasets.

Figures & tables

Figure 1: Overview of PII detection.

Figure 2: Key results of all models across the considerations we have for judging a PII classifier, including cost efficiency, detection rate, and entity coverage.

Figure 3: Historical timeline of models [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 2 , 3 , 27 , 28 , 29 ] and datasets [ 30 , 31 , 32 , 11 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ] for PII detection.

Figure 4: Composition of the PII-SCOPE dataset.

Figure 5: Illustration of overgeneration in PII datasets. The source text contains PII that is not captured by the entity mask, leading to incorrect evaluation and training of PII detectors.

Figure 6: Overgeneration across the considered benchmark datasets.

Dataset	Entities / row ( $\rho$ )	Distinct types / row ( $\delta$ )
AI4Privacy	3.86	2.84
Nemotron	8.15	5.44
Gretel	4.25	3.41
SPY	6.82	5.86
PII-SCOPE	13.17	7.60

Table 1: Richness of the considered benchmark datasets in terms of entity density

\rho

and type diversity

\delta

from Equation ( 3 ).

Figure 7: Difficulty landscape of the evaluated benchmarks, illustrating the relationship between entity density, type diversity, and observed difficulty.

Figure 8: Architecture of the Protegrity-PII model.

Model	Availability	Backbone	Parameters	Native taxonomy
Protegrity-PII	Open weights	XLM-RoBERTa	277.6M	69
OpenAI Privacy Filter	Open weights	MoE token-level classifier	$\sim$ 1.5B ( $\sim$ 50M active)	8
GLiNER2.5 Multi	Open weights	mDeBERTa-v3-base	287M	—
AWS Comprehend	Managed API	Undisclosed	Unknown	36
Microsoft Presidio	Open source	Rule-based + spaCy NER	Unknown	91 *
Rampart	Open source	Rule-based + SLM	18.5M	20

Table 2: PII detectors considered in this work.

Outcome	Typed ruling	Untyped ruling
$s^{\prime}=s,\;e^{\prime}=e,\;t^{\prime}=t$	TP	TP
$s^{\prime}=s,\;e^{\prime}=e,\;t^{\prime}\neq t$ (entity type confusion)	FN for $g$ ; FP for $(s^{\prime},e^{\prime},t^{\prime})$	TP
No $(s^{\prime},e^{\prime},t^{\prime})\in\mathcal{P}_{x}$ with $s^{\prime}=s,\;e^{\prime}=e$ (miss)	FN	FN
$(s^{\prime},e^{\prime},t^{\prime})$ with $(s^{\prime},e^{\prime})\neq(s,e)$ for every $g\in\mathcal{G}_{x}$	FP	FP

Table 3: Prediction outcomes under the typed and untyped notions of recall.

Figure 9: Maximal spanning set typed $F_{1}$ scores for all benchmark models evaluated on PII-SCOPE.

Figure 10: Strict, typed $F_{1}$ scores for all benchmark models evaluated on PII-SCOPE in our full taxonomy and a minimally spanning intersection.

	$\overbrace{\hphantom{\text{{Protegrity-PII}}\hskip 9.24994pt\text{{GLiNER2.5 Multi}}\hskip 9.24994pt\text{{OPF}}\hskip 9.24994pt\text{{Rampart}}}}^{\text{{finetunable, open weights models}}}$				$\overbrace{\hphantom{\text{{AWS Comprehend}}\hskip 9.24994pt\text{{Microsoft Presidio}}}}^{\text{{fixed, API-based services}}}$		$\overbrace{\hphantom{\text{{Claude Fable 5}}}}^{\text{{frontier LLM reference}}}$
Dataset	Protegrity-PII	GLiNER2.5 Multi	OPF	Rampart	AWS Comprehend	Microsoft Presidio	Claude Fable 5
Typed
Recall
PII-SCOPE	0.915	0.429	0.387	0.307	0.465	0.386	0.848
AI4Privacy	0.696	0.156	0.276	0.421	0.404	0.268	0.783
Nemotron	0.691	0.292	0.280	0.198	0.451	0.280	0.752

Table 4: Full benchmark results for all datasets and models under typed and untyped recall, precision, and

F_{1}

. The best non-LLM performing result in each row is bolded .

	$\overbrace{\hphantom{\text{{Protegrity-PII}}\hskip 9.24994pt\text{{GLiNER2.5 Multi}}\hskip 9.24994pt\text{{OPF}}\hskip 9.24994pt\text{{Rampart}}}}^{\text{{finetunable, open weights models}}}$				$\overbrace{\hphantom{\text{{AWS Comprehend}}\hskip 9.24994pt\text{{Microsoft Presidio}}}}^{\text{{fixed, API-based services}}}$		$\overbrace{\hphantom{\text{{Claude Fable 5}}}}^{\text{{frontier LLM reference}}}$
	Protegrity-PII	GLiNER2.5 Multi	OPF	Rampart	AWS Comprehend	Microsoft Presidio	Claude Fable 5
Typed
FNR	0.262	0.749	0.652	0.669	0.486	0.641	0.228
FPR	0.373	0.746	0.601	0.658	0.434	0.702	0.389

Untyped

Table 5: Full benchmark results for all datasets and models under typed and untyped false negative rate (FNR) and false positive rate (FPR). For each dataset we compute

\mathrm{FNR}=\mathrm{FN}/(\mathrm{TP}+\mathrm{FN})=1-R

and

\mathrm{FPR}=\mathrm{FP}/(\mathrm{TP}+\mathrm{FP})=1-P

and report the mean over the five benchmark datasets. The best non-LLM performing result in each row is bolded .

Figure 11: Cost and latency of all benchmark models excluding Rampart evaluated on 600 rows. Lower is better on both axes. Note that axes are capped at $1/1M characters and 100 seconds/1M characters. Beyond this cap, we note that AWS Comprehend costs $1.00/1M characters and Fable 5 costs $11.65/1M characters at $\approx$ 8,905 seconds/1M characters. We leave out Rampart due to there only being an ONNX CPU version available. — Figure 11: Cost and latency of all benchmark models excluding Rampart evaluated on 600 rows. Lower is better on both axes. Note that axes are capped at $1/1M characters and 100 seconds/1M characters. Beyond this cap, we note that AWS Comprehend costs$ 1.00/1M characters and Fable 5 costs $11.65/1M characters at$ \approx$ 8,905 seconds/1M characters. We leave out Rampart due to there only being an ONNX CPU version available.

Figure 12: Generalization result on training and testing with various datasets on a fixed model, measured by untyped recall.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	#1	#2	#3	#4	#5
Protegrity-PII	MEDICAL_RECORD_NUMBER $\rightarrow$ IDCARD	TRACKING_IDENTIFIER $\rightarrow$ ACCOUNTNUMBER	TRACKING_IDENTIFIER $\rightarrow$ IDCARD	IDCARD $\rightarrow$ ACCOUNTNUMBER	BIC $\rightarrow$ ACCOUNTNUMBER
GLiNER 2.5 Multi	NAME $\rightarrow$ ACCOUNTNAME	USERNAME $\rightarrow$ ACCOUNTNAME	PHONE $\rightarrow$ FAX_NUMBER	IDCARD $\rightarrow$ ACCESS_SECURITY_CODE	IDCARD $\rightarrow$ ACCOUNTNUMBER
AWS Comprehend	IP $\rightarrow$ IP_ADDRESS	TIME $\rightarrow$ DATE	COMPANYNAME $\rightarrow$ NAME	COUNTRY $\rightarrow$ STREET	CITY $\rightarrow$ STREET
OPF	IDCARD $\rightarrow$ ACCOUNTNUMBER	USERNAME $\rightarrow$ NAME	SSN $\rightarrow$ ACCOUNTNUMBER	MEDICAL_RECORD_NUMBER $\rightarrow$ ACCOUNTNUMBER	TRACKING_IDENTIFIER $\rightarrow$ ACCOUNTNUMBER
Rampart	USERNAME $\rightarrow$ NAME	IDCARD $\rightarrow$ SSN	DATE $\rightarrow$ ZIPCODE	MEDICAL_RECORD_NUMBER $\rightarrow$ PASSPORT	TRACKING_IDENTIFIER $\rightarrow$ IDCARD
Microsoft Presidio	CITY $\rightarrow$ LOCATION	IDCARD $\rightarrow$ DRIVERLICENSE	STATE $\rightarrow$ LOCATION	TIME $\rightarrow$ DATE	COUNTRY $\rightarrow$ LOCATION

Appendix

Table 6: Top five confused entities for all models across all datasets.

Figure 13: Typed recall by boundary relaxation. Here, exact refers to the metric we used throughout the main text, containment refers to the relaxed metric where a predicted span is considered correct if it contains the ground truth span, and finally overlap refers to the relaxed metric where a predicted span is considered correct if it merely overlaps with the ground truth span.

Figure 14: Common sensitive types total typed recall across all datasets for all models.

Figure 15: Common languages total untyped recall across all datasets for all models.

Model	Ground truth side	Prediction side
Protegrity-PII	Dataset label $\rightarrow$ our label in Appendix B.1	Our label in Appendix B.1
GLiNER2.5 Multi	Dataset label $\rightarrow$ our label in Appendix B.1	Our label in Appendix B.2
AWS Comprehend	Dataset label $\rightarrow$ our label in Appendix B.1	AWS label $\rightarrow$ our label in Appendix B.3
OpenAI Privacy Filter	Dataset label $\rightarrow$ our label in Appendix B.1	OPF label $\rightarrow$ our label in Appendix B.4
Microsoft Presidio	Dataset label $\rightarrow$ our label in Appendix B.1	Presidio label $\rightarrow$ our label in Appendix B.5
Rampart	Dataset label $\rightarrow$ our label in Appendix B.1	Rampart label $\rightarrow$ our label in Appendix B.6

Appendix

Table 7: Link between Appendix B.1 – B.6 in the typed-recall computation.

Model	Evaluation labels	Supported labels	Unsupported labels
Protegrity-PII	69	69	0
GLiNER2.5 Multi	69	69	0
AWS Comprehend	69	25	44
OpenAI Privacy Filter	69	8	61
Microsoft Presidio	69	17	52
Rampart	69	18	51

Appendix

Table 8: Taxonomy coverage of the considered models with respect to our native taxonomy. A label is supported if at least one of the model’s native labels maps onto it in Appendix B.1 – B.6 .

Entity labels shared by all models
ACCOUNTNUMBER	EMAIL
NAME	PHONE	URL

Appendix

Table 9: 5-label intersection of Protegrity-PII, AWS Comprehend, OpenAI Privacy Filter, Microsoft Presidio, and Rampart’s native taxonomies.

Our entity labels
ACCESS_SECURITY_CODE	ACCOUNT_BALANCE	ACCOUNTNAME
ACCOUNTNUMBER	AGE	AMOUNT
BIC	BUILDING	CARD_EXPIRATION_DATE
CARD_SERVICE_CODE	CARD_TRACK_DATA	CCN
CITY	COMPANYNAME	COUNTRY
COUNTY	CREDIT_SCORE	CREDITCARDCVV

Appendix

Table 10: Protegrity-PII supported entities.

Native dataset label	Our label
ACCOUNTNAME	ACCOUNTNAME
ACCOUNTNUMBER	ACCOUNTNUMBER
AGE	AGE
AMOUNT	AMOUNT
BIC	BIC
BUILDING	BUILDING

Appendix

Table 11: AI4Privacy label mappings.

Native dataset label	Our label
account_number	ACCOUNTNUMBER
age	AGE
api_key	LOGIN_CREDENTIAL
bank_routing_number	BIC
certificate_license_number	PROFESSIONAL_LICENSE_NUMBER
city	CITY

Appendix

Table 12: Nemotron label mappings.

Native dataset label	Our label
account_number	ACCOUNTNUMBER
address	STREET
api_key	LOGIN_CREDENTIAL
bank_routing_number	BIC
certificate_license_number	PROFESSIONAL_LICENSE_NUMBER
city	CITY

Appendix

Table 13: Gretel label mappings.

Native dataset label	Our label
ADDRESS	STREET
EMAIL	EMAIL
ID_NUM	IDCARD
NAME	NAME
PHONE_NUM	PHONE
URL	URL

Appendix

Table 14: SPY label mappings.

Native model labels	Mapping	Supported labels	Unsupported labels
All labels in our native taxonomy	Identity	69	0

Appendix

Table 15: GLiNER2.5 label mapping and taxonomy coverage.

Entity name	Description
ACCESS_SECURITY_CODE	Security code used to access an account or protected system
ACCOUNTNAME	Name assigned to a financial or user account
ACCOUNTNUMBER	Identifier for a bank, billing, or service account
ACCOUNT_BALANCE	Monetary balance held in an account
AGE	A person’s age or age range
AMOUNT	Monetary amount or payment value

Appendix

Table 16: GLiNER2.5 entity descriptions

Native model label(s)	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
BANK_ACCOUNT_NUMBER	ACCOUNTNUMBER
AGE	AGE
—	AMOUNT

Appendix

Table 17: AWS Comprehend output mappings to our evaluation taxonomy.

Native model label	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
account_number	ACCOUNTNUMBER
—	AGE
—	AMOUNT

Appendix

Table 18: OpenAI Privacy Filter output mappings to our evaluation taxonomy.

Native model label(s)	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
US_BANK_NUMBER	ACCOUNTNUMBER
—	AGE
—	AMOUNT

Appendix

Table 19: Microsoft Presidio output mappings to our evaluation taxonomy.

Native model label(s)	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
BANK_ACCOUNT	ACCOUNTNUMBER
—	AGE
—	AMOUNT

Appendix

Table 20: Rampart output mappings to our evaluation taxonomy.

PII Detection

Personally identifiable information (PII) detection finds sensitive personal details in text, conversations, images, and other data.

See all 39 papers

May 11, 2026

GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction

Reliable detection of personally identifiable information (PII) is increasingly important across modern data-processing systems, yet the task remains difficult: PII spans are heterogeneous, locale-dependent, context-sensitive, and often embedded in noisy or semi-structured documents. We present GLiNER2-PII, a small 0.3B-parameter model adapted from GLiNER2 and designed to recognize a broad taxonomy of 42 PII entity types at character-span resolution. Training such systems, however, is constrained by the scarcity of shareable annotated data and the privacy risks associated with collecting real PII at scale. To address this challenge, we construct a multilingual synthetic corpus of 4,910 annotated texts using a constraint-driven generation pipeline that produces diverse, realistic examples across languages, domains, formats, and entity distributions. On the challenging SPY benchmark, GLiNER2-PII achieves the highest span-level F1 among five compared systems, including OpenAI Privacy Filter and three GLiNER-based detectors. We publicly release the model on Hugging Face to support further research and practical deployment of open PII detection systems.

Urchade Zaratiana, Ash Lewis, George Hurn-Maloney

Fastino Labs

Named Entity Recognition Document Information Extraction PII Detection

Ask PDF

May 25, 2026

Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa

Personally identifiable information (PII) detection systems are frequently trained within narrow source or domain boundaries, limiting coverage when deployed on heterogeneous text. We study model fine-tuning on a corrected multi-source PIIBench preparation spanning 82 retained entity types across ten source datasets. We evaluate three DeBERTa-based approaches: direct token classification fine-tuning, a source-conditioned hierarchical model (SC+H), and a three-phase curriculum extension (SC+H+Curr). Against eight published comparator systems on a reproducible 5,000-record held-out subset (test_5k), direct fine-tuned DeBERTa achieves F1 0.6476, while SC+H and the curriculum variant achieve 0.5899 and 0.2772 respectively; the strongest published comparator reaches only 0.1723. Because validation initially favoured SC+H, we perform a final streamed evaluation on the complete 100,002-record held-out split. Direct fine-tuning remains superior, achieving F1 0.6455 versus 0.5894 for SC+H. Entity-level analysis shows that direct fine tuning wins 54 of 82 fine entity types and all ten coarse groups by support-weighted entity F1, while SC+H retains localised advantages on 28 types. The results indicate that diverse task-specific training data and a simple weighted cross-entropy objective contribute more to broad-coverage PII detection than the tested architectural and curriculum complexity.

Pritesh Jha

Fine-Tuning PII Detection

Ask PDF

Apr 17, 2026

PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection

We present PIIBench, a unified benchmark corpus for Personally Identifiable Information (PII) detection in natural language text. Existing resources for PII detection are fragmented across domain-specific corpora with mutually incompatible annotation schemes, preventing systematic comparison of detection systems. We consolidate ten publicly available datasets spanning synthetic PII corpora, multilingual Named Entity Recognition (NER) benchmarks, and financial domain annotated text, yielding a corpus of 2,369,883 annotated sequences and 3.35 million entity mentions across 48 canonical PII entity types. We develop a principled normalization pipeline that maps 80+ source-specific label variants to a standardized BIO tagging scheme, applies frequency-based suppression of near absent entity types, and produces stratified 80/10/10 train/validation/test splits preserving source distribution. To establish baseline difficulty, we evaluate eight published systems spanning rule-based engines (Microsoft Presidio), general purpose NER models (spaCy, BERT-base NER, XLM-RoBERTa NER, SpanMarker mBERT, SpanMarker BERT), a PII-specific model (Piiranha DeBERTa), and a financial NER specialist (XtremeDistil FiNER). All systems achieve span-level F1 below 0.14, with the best system (Presidio, F1=0.1385) still producing zero recall on most entity types. These results directly quantify the domain-silo problem and demonstrate that PIIBench presents a substantially harder and more comprehensive evaluation challenge than any existing single source PII dataset. The dataset construction pipeline and benchmark evaluation code are publicly available at https://github.com/pritesh-2711/pii-bench.

Pritesh Jha

Independent Researcher

Named Entity Recognition PII Detection

Ask PDF

Abstract

PII Detection

Explore similar work

GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction

Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa

PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection