Protegrity-PII Technical Report

Hardeep Bassi; Hui Ma; Luís Seabra; Koowum Joshi; Daniel Pereira; Averell Gatton

cs.CLOct 21, 2026

Protegrity-PII Technical Report

Authors: Hardeep Bassi, Hui Ma, Luís Seabra, Koowum Joshi, Daniel Pereira, Averell Gatton

Organizations: Protegrity 1 Menlo Park, CA, USA · Protegrity 2 Stamford, CT, USA · Protegrity 3 Lisbon, Portugal · Arizona State University, Tempe, AZ, USA

Abstract

We present Protegrity-PII, a new PII decision model for structured and unstructured text. Protegrity-PII achieves a typed recall of 0.915 with a precision of 0.907 on a new benchmark we propose, PII-SCOPE. Our model processes text in 0.49 seconds per million characters at $0.0005 per million characters on a single RTX 6000 using batched inference. Due in part to the expansive nature of the entity type taxonomy used in training, Protegrity-PII achieves superior out-of-distribution generalization when benchmarked against a selection of state of the art models and datasets.

Figures & tables

Figure 1: Overview of PII detection.

Figure 2: Key results of all models across the considerations we have for judging a PII classifier, including cost efficiency, detection rate, and entity coverage.

Figure 3: Historical timeline of models [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 2 , 3 , 27 , 28 , 29 ] and datasets [ 30 , 31 , 32 , 11 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ] for PII detection.

Figure 4: Composition of the PII-SCOPE dataset.

Figure 5: Illustration of overgeneration in PII datasets. The source text contains PII that is not captured by the entity mask, leading to incorrect evaluation and training of PII detectors.

Figure 6: Overgeneration across the considered benchmark datasets.

Dataset	Entities / row ( $\rho$ )	Distinct types / row ( $\delta$ )
AI4Privacy	3.86	2.84
Nemotron	8.15	5.44
Gretel	4.25	3.41
SPY	6.82	5.86
PII-SCOPE	13.17	7.60

Table 1: Richness of the considered benchmark datasets in terms of entity density

\rho

and type diversity

\delta

from Equation ( 3 ).

Figure 7: Difficulty landscape of the evaluated benchmarks, illustrating the relationship between entity density, type diversity, and observed difficulty.

Figure 8: Architecture of the Protegrity-PII model.

Model	Availability	Backbone	Parameters	Native taxonomy
Protegrity-PII	Open weights	XLM-RoBERTa	277.6M	69
OpenAI Privacy Filter	Open weights	MoE token-level classifier	$\sim$ 1.5B ( $\sim$ 50M active)	8
GLiNER2.5 Multi	Open weights	mDeBERTa-v3-base	287M	—
AWS Comprehend	Managed API	Undisclosed	Unknown	36
Microsoft Presidio	Open source	Rule-based + spaCy NER	Unknown	91 *
Rampart	Open source	Rule-based + SLM	18.5M	20

Table 2: PII detectors considered in this work.

Outcome	Typed ruling	Untyped ruling
$s^{\prime}=s,\;e^{\prime}=e,\;t^{\prime}=t$	TP	TP
$s^{\prime}=s,\;e^{\prime}=e,\;t^{\prime}\neq t$ (entity type confusion)	FN for $g$ ; FP for $(s^{\prime},e^{\prime},t^{\prime})$	TP
No $(s^{\prime},e^{\prime},t^{\prime})\in\mathcal{P}_{x}$ with $s^{\prime}=s,\;e^{\prime}=e$ (miss)	FN	FN
$(s^{\prime},e^{\prime},t^{\prime})$ with $(s^{\prime},e^{\prime})\neq(s,e)$ for every $g\in\mathcal{G}_{x}$	FP	FP

Table 3: Prediction outcomes under the typed and untyped notions of recall.

Figure 9: Maximal spanning set typed $F_{1}$ scores for all benchmark models evaluated on PII-SCOPE.

Figure 10: Strict, typed $F_{1}$ scores for all benchmark models evaluated on PII-SCOPE in our full taxonomy and a minimally spanning intersection.

	$\overbrace{\hphantom{\text{{Protegrity-PII}}\hskip 9.24994pt\text{{GLiNER2.5 Multi}}\hskip 9.24994pt\text{{OPF}}\hskip 9.24994pt\text{{Rampart}}}}^{\text{{finetunable, open weights models}}}$				$\overbrace{\hphantom{\text{{AWS Comprehend}}\hskip 9.24994pt\text{{Microsoft Presidio}}}}^{\text{{fixed, API-based services}}}$		$\overbrace{\hphantom{\text{{Claude Fable 5}}}}^{\text{{frontier LLM reference}}}$
Dataset	Protegrity-PII	GLiNER2.5 Multi	OPF	Rampart	AWS Comprehend	Microsoft Presidio	Claude Fable 5
Typed
Recall
PII-SCOPE	0.915	0.429	0.387	0.307	0.465	0.386	0.848
AI4Privacy	0.696	0.156	0.276	0.421	0.404	0.268	0.783
Nemotron	0.691	0.292	0.280	0.198	0.451	0.280	0.752

Table 4: Full benchmark results for all datasets and models under typed and untyped recall, precision, and

F_{1}

. The best non-LLM performing result in each row is bolded .

	$\overbrace{\hphantom{\text{{Protegrity-PII}}\hskip 9.24994pt\text{{GLiNER2.5 Multi}}\hskip 9.24994pt\text{{OPF}}\hskip 9.24994pt\text{{Rampart}}}}^{\text{{finetunable, open weights models}}}$				$\overbrace{\hphantom{\text{{AWS Comprehend}}\hskip 9.24994pt\text{{Microsoft Presidio}}}}^{\text{{fixed, API-based services}}}$		$\overbrace{\hphantom{\text{{Claude Fable 5}}}}^{\text{{frontier LLM reference}}}$
	Protegrity-PII	GLiNER2.5 Multi	OPF	Rampart	AWS Comprehend	Microsoft Presidio	Claude Fable 5
Typed
FNR	0.262	0.749	0.652	0.669	0.486	0.641	0.228
FPR	0.373	0.746	0.601	0.658	0.434	0.702	0.389

Untyped

Table 5: Full benchmark results for all datasets and models under typed and untyped false negative rate (FNR) and false positive rate (FPR). For each dataset we compute

\mathrm{FNR}=\mathrm{FN}/(\mathrm{TP}+\mathrm{FN})=1-R

and

\mathrm{FPR}=\mathrm{FP}/(\mathrm{TP}+\mathrm{FP})=1-P

and report the mean over the five benchmark datasets. The best non-LLM performing result in each row is bolded .

Figure 11: Cost and latency of all benchmark models excluding Rampart evaluated on 600 rows. Lower is better on both axes. Note that axes are capped at $1/1M characters and 100 seconds/1M characters. Beyond this cap, we note that AWS Comprehend costs $1.00/1M characters and Fable 5 costs $11.65/1M characters at $\approx$ 8,905 seconds/1M characters. We leave out Rampart due to there only being an ONNX CPU version available. — Figure 11: Cost and latency of all benchmark models excluding Rampart evaluated on 600 rows. Lower is better on both axes. Note that axes are capped at $1/1M characters and 100 seconds/1M characters. Beyond this cap, we note that AWS Comprehend costs$ 1.00/1M characters and Fable 5 costs $11.65/1M characters at$ \approx$ 8,905 seconds/1M characters. We leave out Rampart due to there only being an ONNX CPU version available.

Figure 12: Generalization result on training and testing with various datasets on a fixed model, measured by untyped recall.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	#1	#2	#3	#4	#5
Protegrity-PII	MEDICAL_RECORD_NUMBER $\rightarrow$ IDCARD	TRACKING_IDENTIFIER $\rightarrow$ ACCOUNTNUMBER	TRACKING_IDENTIFIER $\rightarrow$ IDCARD	IDCARD $\rightarrow$ ACCOUNTNUMBER	BIC $\rightarrow$ ACCOUNTNUMBER
GLiNER 2.5 Multi	NAME $\rightarrow$ ACCOUNTNAME	USERNAME $\rightarrow$ ACCOUNTNAME	PHONE $\rightarrow$ FAX_NUMBER	IDCARD $\rightarrow$ ACCESS_SECURITY_CODE	IDCARD $\rightarrow$ ACCOUNTNUMBER
AWS Comprehend	IP $\rightarrow$ IP_ADDRESS	TIME $\rightarrow$ DATE	COMPANYNAME $\rightarrow$ NAME	COUNTRY $\rightarrow$ STREET	CITY $\rightarrow$ STREET
OPF	IDCARD $\rightarrow$ ACCOUNTNUMBER	USERNAME $\rightarrow$ NAME	SSN $\rightarrow$ ACCOUNTNUMBER	MEDICAL_RECORD_NUMBER $\rightarrow$ ACCOUNTNUMBER	TRACKING_IDENTIFIER $\rightarrow$ ACCOUNTNUMBER
Rampart	USERNAME $\rightarrow$ NAME	IDCARD $\rightarrow$ SSN	DATE $\rightarrow$ ZIPCODE	MEDICAL_RECORD_NUMBER $\rightarrow$ PASSPORT	TRACKING_IDENTIFIER $\rightarrow$ IDCARD
Microsoft Presidio	CITY $\rightarrow$ LOCATION	IDCARD $\rightarrow$ DRIVERLICENSE	STATE $\rightarrow$ LOCATION	TIME $\rightarrow$ DATE	COUNTRY $\rightarrow$ LOCATION

Appendix

Table 6: Top five confused entities for all models across all datasets.

Figure 13: Typed recall by boundary relaxation. Here, exact refers to the metric we used throughout the main text, containment refers to the relaxed metric where a predicted span is considered correct if it contains the ground truth span, and finally overlap refers to the relaxed metric where a predicted span is considered correct if it merely overlaps with the ground truth span.

Figure 14: Common sensitive types total typed recall across all datasets for all models.

Figure 15: Common languages total untyped recall across all datasets for all models.

Model	Ground truth side	Prediction side
Protegrity-PII	Dataset label $\rightarrow$ our label in Appendix B.1	Our label in Appendix B.1
GLiNER2.5 Multi	Dataset label $\rightarrow$ our label in Appendix B.1	Our label in Appendix B.2
AWS Comprehend	Dataset label $\rightarrow$ our label in Appendix B.1	AWS label $\rightarrow$ our label in Appendix B.3
OpenAI Privacy Filter	Dataset label $\rightarrow$ our label in Appendix B.1	OPF label $\rightarrow$ our label in Appendix B.4
Microsoft Presidio	Dataset label $\rightarrow$ our label in Appendix B.1	Presidio label $\rightarrow$ our label in Appendix B.5
Rampart	Dataset label $\rightarrow$ our label in Appendix B.1	Rampart label $\rightarrow$ our label in Appendix B.6

Appendix

Table 7: Link between Appendix B.1 – B.6 in the typed-recall computation.

Model	Evaluation labels	Supported labels	Unsupported labels
Protegrity-PII	69	69	0
GLiNER2.5 Multi	69	69	0
AWS Comprehend	69	25	44
OpenAI Privacy Filter	69	8	61
Microsoft Presidio	69	17	52
Rampart	69	18	51

Appendix

Table 8: Taxonomy coverage of the considered models with respect to our native taxonomy. A label is supported if at least one of the model’s native labels maps onto it in Appendix B.1 – B.6 .

Entity labels shared by all models
ACCOUNTNUMBER	EMAIL
NAME	PHONE	URL

Appendix

Table 9: 5-label intersection of Protegrity-PII, AWS Comprehend, OpenAI Privacy Filter, Microsoft Presidio, and Rampart’s native taxonomies.

Our entity labels
ACCESS_SECURITY_CODE	ACCOUNT_BALANCE	ACCOUNTNAME
ACCOUNTNUMBER	AGE	AMOUNT
BIC	BUILDING	CARD_EXPIRATION_DATE
CARD_SERVICE_CODE	CARD_TRACK_DATA	CCN
CITY	COMPANYNAME	COUNTRY
COUNTY	CREDIT_SCORE	CREDITCARDCVV

Appendix

Table 10: Protegrity-PII supported entities.

Native dataset label	Our label
ACCOUNTNAME	ACCOUNTNAME
ACCOUNTNUMBER	ACCOUNTNUMBER
AGE	AGE
AMOUNT	AMOUNT
BIC	BIC
BUILDING	BUILDING

Appendix

Table 11: AI4Privacy label mappings.

Native dataset label	Our label
account_number	ACCOUNTNUMBER
age	AGE
api_key	LOGIN_CREDENTIAL
bank_routing_number	BIC
certificate_license_number	PROFESSIONAL_LICENSE_NUMBER
city	CITY

Appendix

Table 12: Nemotron label mappings.

Native dataset label	Our label
account_number	ACCOUNTNUMBER
address	STREET
api_key	LOGIN_CREDENTIAL
bank_routing_number	BIC
certificate_license_number	PROFESSIONAL_LICENSE_NUMBER
city	CITY

Appendix

Table 13: Gretel label mappings.

Native dataset label	Our label
ADDRESS	STREET
EMAIL	EMAIL
ID_NUM	IDCARD
NAME	NAME
PHONE_NUM	PHONE
URL	URL

Appendix

Table 14: SPY label mappings.

Native model labels	Mapping	Supported labels	Unsupported labels
All labels in our native taxonomy	Identity	69	0

Appendix

Table 15: GLiNER2.5 label mapping and taxonomy coverage.

Entity name	Description
ACCESS_SECURITY_CODE	Security code used to access an account or protected system
ACCOUNTNAME	Name assigned to a financial or user account
ACCOUNTNUMBER	Identifier for a bank, billing, or service account
ACCOUNT_BALANCE	Monetary balance held in an account
AGE	A person’s age or age range
AMOUNT	Monetary amount or payment value

Appendix

Table 16: GLiNER2.5 entity descriptions

Native model label(s)	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
BANK_ACCOUNT_NUMBER	ACCOUNTNUMBER
AGE	AGE
—	AMOUNT

Appendix

Table 17: AWS Comprehend output mappings to our evaluation taxonomy.

Native model label	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
account_number	ACCOUNTNUMBER
—	AGE
—	AMOUNT

Appendix

Table 18: OpenAI Privacy Filter output mappings to our evaluation taxonomy.

Native model label(s)	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
US_BANK_NUMBER	ACCOUNTNUMBER
—	AGE
—	AMOUNT

Appendix

Table 19: Microsoft Presidio output mappings to our evaluation taxonomy.

Native model label(s)	Our label
—	ACCESS_SECURITY_CODE
—	ACCOUNT_BALANCE
—	ACCOUNTNAME
BANK_ACCOUNT	ACCOUNTNUMBER
—	AGE
—	AMOUNT

Appendix

Table 20: Rampart output mappings to our evaluation taxonomy.

PII Detection

Personally identifiable information (PII) detection finds sensitive personal details in text, conversations, images, and other data.

See all 39 papers

GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction
May 11, 2026Urchade Zaratiana, Ash Lewis, George Hurn-MaloneyNamed Entity Recognition Document Information Extraction
Fastino Labs
Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa
May 25, 2026Pritesh JhaFine-Tuning PII Detection
PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection
Apr 17, 2026Pritesh JhaNamed Entity Recognition PII Detection
Independent Researcher

May 11, 2026J/K move · Enter open · S save

GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction

Urchade Zaratiana, Ash Lewis, George Hurn-Maloney

Fastino Labs

Named Entity Recognition Document Information Extraction PII Detection

Open paper AskPDF arXiv

Reliable detection of personally identifiable information (PII) is increasingly important across modern data-processing systems, yet the task remains difficult: PII spans are heterogeneous, locale-dependent, context-sensitive, and often embedded in noisy or semi-structured documents. We present GLiNER2-PII, a small 0.3B-parameter model adapted from GLiNER2 and designed to recognize a broad taxonomy of 42 PII entity types at character-span resolution. Training such systems, however, is constrained by the scarcity of shareable annotated data and the privacy risks associated with collecting real PII at scale. To address this challenge, we construct a multilingual synthetic corpus of 4,910 annotated texts using a constraint-driven generation pipeline that produces diverse, realistic examples across languages, domains, formats, and entity distributions. On the challenging SPY benchmark, GLiNER2-PII achieves the highest span-level F1 among five compared systems, including OpenAI Privacy Filter and three GLiNER-based detectors. We publicly release the model on Hugging Face to support further research and practical deployment of open PII detection systems.

Abstract

PII Detection

Explore similar work

GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction

Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa

PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection