Strong Multilingual Privacy Tagging at Encoder Speed

Jonathan Graehl

cs.CLSep 29, 2026

Strong Multilingual Privacy Tagging at Encoder Speed

Authors: Jonathan Graehl

Organizations: RWS Language Weaver

Abstract

Privacy redaction must remove personal information while preserving relationships expressed in text. We develop a multilingual named-entity tagger with fine-grained distinctions supporting varied redaction policies and methods for cheaply learning additional distinctions. We fine-tune a multilingual encoder with an affine span-tagging head on frontier-model annotations in 35 languages, replay mapped human gold with coverage-aware masking so unannotated types are not treated as negatives, and repair subword boundaries with a learned +/-1-character adjustment. On 1,283 human-gold test segments in seven languages, best measured redaction F1 is 88.8, against 69.1 for published GLiNER2 with 11 unrepresentable types excluded from its task (68.8 without that exemption), 67.8 for GLiNER2 adapted to the new training data, 57.3 for Microsoft Presidio and 35.8 for the best published OpenAI Privacy Filter fine-tune. Adding about 50,000 annotated training sentences and increasing human-gold replay improves exact typed-span F1 from 74.5 to 76.3 on Ont3, our 31-type frontier-annotated NER evaluation of 1,201 development segments. Mapped-gold replay alone raises human-gold F1 by ten points without loss on frontier-annotated text; boundary adjustment adds 1.7 exact typed-span F1 points on Ont3. Local LLMs fitting on a single 96-GB GPU underperformed as prompted annotators and frozen encoders, with encoding 30-95 times slower than XLM-R inference and prompted annotation roughly 180-1,100 times slower in the evaluated configurations. The encoder architecture delivers 4.9 times GLiNER2's CPU throughput. We release code, prompts and training recipes, with data-acquisition scripts and source links.

Figures & tables

Figure 1: Precision–recall on 1,283 human-gold segments in seven languages (top three rows) and on Ont3 (bottom row): the 302-segment seven-language subset at left and all 1,201 segments over 35 languages at right. Untyped redaction regions match at 80% overlap of both. Auxiliary refinements such as name components are not scored; models are not penalized for lacking them. GL and GL4: published and new-data-adapted GLiNER2; O2–O4: XLM - R variants; PR: multilingual Presidio, a fixed point; PF: OpenMed multilingual-v2 Privacy Filter. GL has an unequal advantage: gold spans outside its label coverage are unscored for GL alone; Figure 13 compares GL and O4 on the same restricted task. Across all five Privacy Filter variants, the best F1 stays below 40 on pooled human gold (35.8) and full Ont3 (39.4). Both axes start at 0.4, with off-scale Presidio points marked at the axis edge. O4’s best measured F1 is 88.8 on human gold and 82.1 on Ont3. Qualitatively similar exact span-boundary-required results appear in Section A.7 .

Figure 2: Components locate subspans of a name; a role labels the whole name span (illustrative).

Figure 3: Primary labels shared by Ont2 and Ont3, grouped by semantic family. Dashed amber circles mark Ont3’s two optional reference labels; there are no Ont2-only primary labels. Identifier-kind and name-component refinements are in Figure 9 .

Figure 4: Prompt assembly: fixed English instructions plus a demonstration in the target language (French shown). Ellipses omit other rules and annotations.

Recipe	Unique non-human texts	Labeled tokens	Human-gold draws	Input
O3	38,700	477,685	40%	Neighboring sentences
O4	88,512	856,116	50%	Isolated sentence

Table 1: Both models use Ont3 in 35 languages and replay the same 80,099 unique human-gold texts. Counts deduplicate language and text before sampling; labeled tokens include optional references. O2 uses a twenty-language mapped-gold recipe (Table 12 ).

System	Hardware	Input tokens/s
XLM - R	CPU	675
GLiNER2	CPU	138
XLM - R	L40S	2,744
GLiNER2	L40S	1,405
Qwen3.8-Flash - Next	RTX PRO 6000	29
Gemma - 4-31B base	RTX PRO 6000	93

Table 2: Throughput per labeled input token, excluding instructions and context. CPU is EPYC 7R13 with 16 threads. XLM - R and GLiNER2 use shared workloads and document batches on GPU (Section A.18 ). The three LLM rows measure encoding for simple BIOES heads, not prompted annotation output generation (Section A.5 ).

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Human-gold component	Inputs	Source labels used by the mapping	Accepted primary types	Gold spans with multiple accepted types / annotated spans
MAPA	189	6	13	59 / 126
AQMAR	134	3	8	82 / 173
OpenNER collection	904	3	8	350 / 999
Wojood subset	56	11	14	52 / 99

Appendix

Table 3: Annotation coverage of the frozen human-gold test subset. Counts refer to original annotated spans, before merging redaction regions. Source-label counts describe the evaluated schema, not every label in the original corpus.

Figure 5: Broad source labels can supervise several target types: MAPA’s ADDRESS permits locality, postal code and street address, among others. Filled squares mark fallback labels; open circles mark other acceptable labels. Shown: sixteen selected mappings from five corpora to the Ont2 label inventory.

Resource	Terms and access
Nemotron-PII , OpenPII 1M and 1.5M	CC BY 4.0; public downloads.
Ai4Privacy 200k	Custom terms; for-profit organizations with more than three staff need a corporate license. Redistribution restrictions apply.
Ai4Privacy health-PHI 400k diagnostic sample	Gated access under custom commercial terms.
TAB , idner-news-2k and Wojood public sample	MIT; public repositories. Full Wojood is separate and was not used.
MEDDOCAN , SPY , MultiGraSCCo and MAPA EUR-Lex	CC BY 4.0; public distributions. Other MAPA packages may differ.
KLUE NER and HiNER	CC BY-SA 4.0; public distributions.

Appendix

Table 4: Data sources and reuse conditions for the acquired versions.

Training scope	Updates	GPU / AWS instance	Hours	Estimated $
Ont2-equivalent exposure	33,300	L40S / g6e.4xlarge	6.0*	8.5–18.0*
Ont3 fit from pretrained XLM - R	16,000	RTX PRO 6000 Blackwell Server / g7e.2xlarge	2.3	4.3–5.4

Appendix

Table 5: Training-only estimates at AWS Linux Spot prices observed on 22 September 2026:

1.41–3.00/hour in us-west-2 for Ont2 and

1.87–2.37/hour in ap-south-1 for Ont3.

Model	Failed format (%)	Character F1	With XLM - R fallback*	Rescued subset F1*
Gemma - 4-12B	5.01	83.84	86.38*	84.51*
Gemma - 4-31B-FP8	6.22	83.98	90.11*	85.13*
Qwen3.8-Flash - Next-NVFP4	8.19	76.03	87.02*	90.77*
O4, sentence only	—	91.53	—	—

Appendix

Table 6: Annotation quality on the 659-segment Ont3 subset. Asterisks mark XLM - R substitution for invalid annotations; without fallback, invalid outputs count as empty predictions. Rescued subsets contain 33, 41 and 54 segments, respectively. O4 uses zero O bias, isolated input and boundary refinement.

System	Params (B)	Active (B)	Labeled input tokens/s
Gemma - 4-12B	12	—	2
Gemma - 4-31B-IT, bfloat16	30.7	—	2.5
Qwen3.8-Flash - Next-NVFP4	176	6	15
XLM - R encoder	0.55	—	2,744

Appendix

Table 7: Gemma - 4-31B-IT serial generation measured on a 96-GB RTX PRO 6000: 35 inputs in 35 languages, repeated twice, 591 labeled input tokens per pass. Other prompted rates divide labeled input tokens by summed request durations, including retries. The encoder rate is measured throughput (Table 2 ). Parameter counts describe nominal backbones; the active count is shown for Flash-Next’s sparse main model, per token (Table 8 ).

(Prompted) LLM / encoder	Params (B)	Head	Input state offsets	Typed span F1 (%)	Redaction-character F1 (%)
Flash-Next unprompted	176	Affine	0	39.14	78.59
Qwen3.8-Flash-Next	176	Affine	0	44.56	80.64
Qwen3.8-Flash-Next	176	Affine	−1, 0, +1	53.00	82.96
Qwen3.8-Flash-Next	176	512-unit GELU	0	53.80	85.25
Qwen3.8-Flash-Next	176	512-unit GELU	−1, 0, +1	59.54	86.61
Gemma - 4-31B base	30.7	512-unit GELU	−1, 0, +1	62.41	87.07

Appendix

Table 8: Flash - Next, Gemma and XLM - R heads on 659 Ont3 development segments from 468 source documents. Scores pool counts with optional references and zero O bias. Offsets are relative to the target token; 0 uses only its own state. LLM rows use the annotation prompt unless marked unprompted; XLM - R uses no prompt. Parameter counts are nominal backbone sizes in billions, excluding the BIOES head. Flash - Next has 125B main-model parameters (6B active per token) plus 51B n-gram embeddings; its unused 4B multi-token prediction module is excluded. Flash and Gemma heads use the same 32,821 training targets, 279-target selector and annotation prompt; the fit ceiling is ten epochs of 250 batches of 4,096 tokens. LLM heads train in about 15 minutes or less on frozen, already-computed encodings. The first three XLM - R heads receive 1,000 steps on a 40,303-window pool with the encoder frozen and the same selector. O4 is the fully fine-tuned serving model.

Figure 7: Precision–recall comparison on the same 659 Ont3 development inputs across 35 languages; axes start at 20% recall and 40% precision. Curves vary the O bias before constrained BIOES decoding; thin plus signs mark zero bias. Standalone points mark prompted annotation, whose format failures count as empty predictions without encoder fallback. Current corrected references and optional-reference scoring apply throughout; counts are pooled across inputs. Gemma - 4-12B uses layer-32 representations conditioned by a short task prompt. Flash - Next and both Gemma - 4-31B variants use matched 512-unit GELU heads over final-layer states at token offsets [−1, 0, +1], with the same training targets, annotation prompt and fitting schedule. O4 uses isolated input, human-gold replay and character-boundary refinement. Name-kind subspans are not scored.

Figure 8: Ont1’s union inventory. Branches show the recorded parent relations, not mutually exclusive subclasses. Under person_name, teal name components and orange person roles mix different annotation axes. The government_id branch likewise retains the finer distinctions contributed by source corpora.

Figure 9: Refinement heads read the same encoder states as the BIOES head. Token categories identify name components; independent binary predicates allow overlapping attributes. A whole-span category, such as identifier kind, is predicted from the mean token logits. Refinements do not affect the reported model comparisons.

Figure 10: The human-gold and Ont3 panels of Figure 1 , requiring identical redaction-region boundaries. GL has an unfair advantage: categories its labels cannot express are excluded from its scoring obligations, and it may omit job titles. Honorifics are optional; titles may be inside name spans or separate. All models receive corpus-coverage masks (Section A.9.1 ); Figure 13 compares O4 and GL on identical supported categories.

Figure 11: Ont3 comparison on 1,201 segments, including the O1 predecessor omitted from the main figure. Regions match at 80% symmetric overlap. GL has an unfair advantage: categories its labels cannot express are excluded from its scoring obligations, and it may omit job titles. Honorifics are optional; titles may be inside name spans or separate. All models receive corpus-coverage masks (Section A.9.1 ); Figure 13 compares O4 and GL on identical supported categories.

Coverage	Importance	Language codes
Original 20	×4	en
	×2	ar, de, es, fr, ko, pt, vi, zh
	×1	cs, hi, id, it, ja, nl, pl, ru, sv, tr, uk
+15 → 35	×1	bn, da, el, fa, fi, fil, he, hr, ms, no, ro, ta, te, th, ur

Appendix

Table 9: Language weights in historical evaluation sets. The nine priority languages have weight ×2 or ×4. Main Ont3 and human-gold comparisons use pooled counts without language weights.

Figure 12: O4 and the original OpenAI Privacy Filter (PF) on a reduced task that scores only the entity categories covered by PF’s eight labels. Organizations, generic locations and every other category outside PF’s labels are removed from the task, gold and predictions alike, preserving many-to-many alternatives admitted by coarse human-gold labels, without retraining either model; characters inside removed gold regions are not scored. Curves sweep each model’s decision threshold; labels give each curve’s best character F1. Columns are human gold and Ont3; the top row uses all languages (1,283 and 1,201 segments), the bottom English only (176 and 56), the primary language named by PF’s model card. Both models use the same subset in each panel.

Figure 13: O4 and published GLiNER2 (GL) with identical category obligations. Both models are scored only on categories expressible by GL’s 42 requested labels plus organization, with the same title allowances and corpus-coverage masks. O4 predictions outside that inventory are removed as well, preserving many-to-many alternatives admitted by coarse human-gold labels. Unlike the main comparison, GL receives no exclusive category exemption. Untyped regions match at 80% symmetric overlap; labels give best measured region F1. Columns are human gold and Ont3. The Ont3 panels use only the 542 segments not used to select O4 or GL4. The top row uses all languages (1,283 and 542 segments); the bottom uses GL’s advertised English, French, Spanish, German, Italian, Portuguese and Dutch (898 and 171). Both models use the same subset in each panel.

Updates	Human gold exact regions	Ont3 exact regions
0 (transferred initialization)	57.44	50.11
100	63.73	54.84
200	64.72	56.96
300	64.52	56.24
400	62.98	57.10
600 (selected)	67.24	54.79

Appendix

Table 10: Fixed-confidence F1 (%) on the paper’s 1,283 human-gold segments and the 659-segment Ont3 selection subset. With the same full-inventory scoring obligations, selected GL4 improves over the transferred initialization by 9.8 human-gold and 4.7 Ont3 F1 points. Stock GL’s category exemptions in the main graph obscure this adaptation gain. These are descriptive trajectory values; the smaller human-gold view does not choose the checkpoint.

Baseline configuration	Languages	Documents	Our model’s F1 gain over baseline (points)	95% interval (points)
GLiNER2	7	421	+21.78	[+19.14, +24.42]
OpenAI Privacy Filter	1	61	+47.31	[+42.37, +52.25]
Multilingual OpenMed	14	907	+40.55	[+37.63, +43.46]
OpenMed with additional task fine-tuning	20	1,267	+48.70	[+46.08, +51.31]
OpenMed Nemotron	9	541	+43.76	[+40.18, +47.35]

Appendix

Table 11: Ont2 encoder gains in untyped span-overlap F1 at fixed operating points on Fresh20. Each row uses only the baseline’s supported languages; differing subsets prevent ranking the baselines by these gains. Intervals use paired document resampling.

Figure 14: English. GLiNER2’s financial span extends from the name through the insurance number, including intervening prose. The encoder matches the gold boundaries; its finer identifier type differs.

Figure 15: Spanish. In this short clinical contact block, GLiNER2 labels only the surname; the encoder includes the given name as well. Both recover the address fields and email.

Method	O2	O3	O4
Starting encoder	Previously task-trained encoder, transferred to Ont2	Upstream pretrained XLM - R-large	Ont3 encoder’s isolated-input branch
Language coverage	20 languages	35 languages	35 languages
Supervision	Mapped annotations, translated data and Ont2 complete-label calibration mixture	Ont3 mixture plus 40% draws from four mapped human-gold corpora	49,812 additional frontier-annotated texts; earlier labels cleaned; prompts clarify occupation/job titles as demographic_attribute , outside names. 50% mapped human-gold draws.
Primary label inventory	29 types	Same 29 types plus two optional reference types	Same as O3
Additional training objectives	Binary entity-presence loss, weight 2	Subclass and type-conditioned predicate losses, each weight 1	Same as O3
O loss weight	1 on complete annotations; 0 on partial annotations	0.75 on supervised O tokens; corpus-unannotated types masked	Same as O3

Appendix

Table 12: Active methods in O2, O3 and O4. O2 uses Ont2; O3 and O4 use Ont3. O4’s additional annotations, cleanup and sampling are detailed in Section A.16 . Boundary refinement is described in Section A.14 ; the main curves score primary redaction spans and ignore name-kind subspans.

Figure 16: BIOES labels mark a span’s beginning, interior, end or single token; O marks text outside spans. In this constructed example, independent winners switch fine type inside “St Mary Hospital.” Requiring one fine type loses the name; class-aware decoding recovers one organization span. Bold numbers are the independent winners. Scores are invented logits, not probabilities or measured model outputs; words stand in for tokenizer tokens.

Decoder	Typed exact F1 (%)	Typed overlap F1 (%)	CPU decode time (s)
Matching-type Viterbi	73.77	78.65	2.41
Coarse-compatible greedy; first label	70.51	75.61	0.32
Coarse-class Viterbi; max scores	74.00	78.91	1.54
Coarse-class Viterbi; soft scores, T = 0.5	74.19	79.18	2.33
Score-adjusted Viterbi; α = 0.95, T = 0.5	74.19	79.17	2.53

Appendix

Table 13: Decoder contrast on the 487 Fresh7 development documents in seven languages, with the same XLM - R checkpoint logits and zero O bias. Decode times exclude encoder inference.

Figure 17: Exact fine-type Viterbi approaches the soft coarse-class decoder as its scores back off toward the normalized sibling aggregate. Points are the nine tested interpolation weights; dashed lines show the coarse-class decoder. F1 uses 20 shared reporting classes on the same 487 reused development documents. Fine-label quality is not measured by these curves.

Token features and head	Head params	Initial screen F1 (%)	Repeat screen F1 (%)	Interpretation
final layer [24], affine	357,725	82.20	82.14	stable reference
middle + final [12,24], affine	715,101	83.82	81.86	initial gain does not repeat
layers [23,24], affine	715,101	—	73.73 step 1,250	early-screen loss
layers [8,24], affine	715,101	—	71.67 step 1,250	early-screen loss
embedding + final [0,24], affine	715,101	—	69.16 step 1,250	early-screen loss
final layer, 1,024-unit GELU	1,409,373	69.21 step 1,250	—	early-screen loss

Appendix

Table 14: XLM - R head and layer-feature screens on a fixed 192,000-window training subset. Entries report peak validation F1, except where an early stopping step is shown. The initial middle-layer gain did not repeat.

Encoder and task head	Fresh6 F1 (%) [95% interval]	MultiGraSCCo F1 (%) [95% interval]	Disposition
XLM - R-large, final affine	92.23 [91.41, 92.98]	75.07 [74.27, 75.90]	quality base selected
LaBSE, final affine	87.88 [86.68, 88.94]	72.53 [71.69, 73.38]	common-head comparison
mmBERT-base, stock head	89.25 [88.27, 90.17]	70.61 [69.77, 71.49]	faster challenger; different head

Appendix

Table 15: Encoder systems trained on the same 1,854 MEDDOCAN/TAB documents and evaluated on Fresh6 and MultiGraSCCo. XLM - R and LaBSE each use an affine head; mmBERT uses its stock nonlinear head. Brackets give 95% intervals.

Model and head	Human gold regions	Ont3 regions	Ont3 typed
mmBERT-base, stock GELU	84.89	76.86	72.32
mmBERT-base, affine	87.03	77.02	73.90
XLM - R-large O4, affine	87.80	80.20	76.30

Appendix

Table 16: Exact-span F1 (%) on 1,283 human-gold examples and 1,201 Ont3 development and held-out examples. The mmBERT head comparison uses O4’s full training pool and serving policy; O4 retains its own training lineage.

Seed	Name repair: macro span Δ (points) [95% interval]	Name repair: macro fine Δ (points) [95% interval]
154	+1.57 [+0.79, +2.38]	+2.47 [+1.78, +3.20]
155	+1.44 [+0.64, +2.23]	+1.96 [+1.30, +2.62]
156	+1.88 [+0.92, +2.84]	+2.64 [+1.86, +3.42]

Appendix

Table 17: Name-order repair versus its matched control on 487 Fresh7 documents. Deltas are language-macro overlap-F1 points, with paired 95% intervals.

Seed	Date repair: Korean span Δ (points) [95% interval]	Date repair: macro span Δ (points) [95% interval]
154	+5.08 [+2.64, +7.63]	+1.62 [+0.92, +2.33]
155	+7.53 [+4.61, +10.63]	+2.12 [+1.29, +2.96]
156	+4.38 [+1.06, +7.58]	+1.16 [+0.27, +2.04]

Appendix

Table 18: Korean date-format repair versus its matched control on Fresh7. Deltas are span-overlap-F1 points for Korean and the seven-language macro average, with paired 95% intervals.

Step	Text
Source field	First Name: Kevin
Replace the name with a placeholder	First Name: [GIVEN_NAME_1]
TranslateGemma output	Nombre: [GIVEN_NAME_1]

Appendix

Table 19: Placeholder translation preserves the given-name annotation through its stable slot identifier. The translated marker is subsequently filled with an entity value.

Surface	View	XLM - R Δ (F1 points) [95% interval]	mmBERT Δ (F1 points) [95% interval]
Fresh20	span overlap	+0.59 [+0.13, +1.06]	+0.44 [+0.18, +0.71]
Fresh20	fine overlap	+0.20 [-0.08, +0.49]	+0.30 [+0.08, +0.53]
MultiGraSCCo	span overlap	+0.15 [+0.04, +0.26]	+0.38 [+0.22, +0.53]
MultiGraSCCo	fine overlap	+0.05 [-0.04, +0.13]	+0.44 [+0.28, +0.61]

Appendix

Table 20: Checked translations versus unchanged training data, at matched exposure within each encoder. Fresh20 has 1,267 documents; MultiGraSCCo has 630 document-language instances. Deltas are F1 points with paired 95% intervals.

Evaluation	Untyped spans	Fine labels
Fresh20 (twenty-language development set)	+1.36 [0.90, 1.82]	−0.45 [−1.02, 0.10]
MultiGraSCCo	+0.46 [0.24, 0.68]	−0.90 [−1.19, −0.61]

Appendix

Table 21: Routed translations, identifier-label repairs and locale changes evaluated with the same deployment decoder. Entries are language-macro overlap-F1 changes in points, with paired 95% intervals; untyped gains accompany fine-label losses.

Stage	Accepted at this stage	Cumulative acceptance
TranslateGemma-27B, after marker repair	68,309 / 69,480 (98.31%)	98.31%
Gemma - 4-31B retry of the 1,171 failures	1,007 / 1,171 (86.00%)	99.76%

Appendix

Table 22: Translation-route acceptance after marker restoration and automated checks across eight target languages. Acceptance measures structural and quality-filter passes, not human-reviewed annotation accuracy.

Figure 19: Literal paired Spanish outputs from a nine-sentence, three-document pilot covering Spanish, Indonesian and Japanese. TranslateGemma-12B loses the surface delimiters; prompted Gemma - 4-12B preserves them and supplies replacement names.

Model	Format passed	Format and replacement type passed
TranslateGemma-12B	0% (0/5)	0% (0/5)
Prompted Gemma - 4-12B	100% (5/5)	60% (3/5)

Appendix

Table 23: Joint translation and entity replacement on five entity-bearing sentences. Format requires intact markers and value delimiters; replacement type also requires a value of the requested entity type.

Figure 20: Recovery boundaries in saved transport output. The Turkish clinical example means “normal blood count”: its translated label name retains numeric ID 43 and an adjacent value. A Unicode-aware validator retrospectively recovers 19 Turkish rows in a 124-output, four-language TranslateGemma frame control. Detached values remain rejected. The county/city example summarizes the separate Gemma - 4 pilot’s semantic failure.

Model	Before repair	After numeric-ID recovery
TranslateGemma-12B	25.0% (31/124)	40.3% (50/124)

Appendix

Table 24: Structural acceptance in a 124-output, four-language translation test. Recovering translated marker names by numeric ID rescues 19 Turkish outputs; semantic correctness was not measured.

Weighted view	Mean delta (points)	Paired 95% interval (points)	Adjusted status
span overlap	+1.00	[+0.73, +1.40]	win
character precision	-0.26	[-0.38, -0.17]	loss
character recall	+2.80	[+2.40, +3.20]	win
character F1	+1.50	[+1.20, +1.70]	win
fine overlap	+0.11	[-0.20, +0.42]	unresolved

Appendix

Table 25: Ignoring unmarked Gemma teacher tokens versus treating them as background. The contrast averages the two positive-only seeds 155 and 156 and subtracts the existing complete-supervision seed-155 baseline. Results are on Fresh20, using 10,000 paired document resamples. Intervals are per metric; win/loss means the interval excludes zero after simultaneous adjustment across 276 comparisons, and unresolved means it includes zero.

Annotator	Span density relative to Luna	Without _reference tags
Gemma - 4-31B	82%	90%
Qwen3.8-27B	61%	75%

Appendix

Table 26: First-pass span counts relative to Luna on the same 1,113 Ont3 development segments in 35 languages. Ratios measure annotation density, not recall; span splitting and false positives also affect them.

Figure 21: Ont2 development accuracy and fresh legal evaluation. Exact typed span F1 is measured separately on each population. The clinical-heavy MEDDOCAN, MultiGraSCCo and TAB development set covers 16 types; the older Final20 retention set covers 29. Fresh English HUDOC legal text reveals a large gap despite high development scores.

Model	Languages	Sentences	Exact typed span F1 (%)	Character-redaction F1 (%)
Ont2	Shared 20	374	52.52	76.63
Ont3	Shared 20	374	73.21	91.58
Ont2	35	659	51.33	75.09
Ont3	35	659	72.89	91.17

Appendix

Table 27: Ont2 and the pre-replay, sentence-only Ont3 model on identical development text, with optional references. The shared-language difference is +20.68 span-F1 points (95% paired interval [+16.24, +25.20]).

Population	Exact-match task	Without model	With model	ΔF1 (95% interval)
Human gold, 1,283 segments	Redaction regions	87.87	87.80	−0.07 [−0.23, 0.00]
Ont3, 1,201 segments	Redaction regions	77.71	80.20	+2.49 [+1.58, +3.50]
Ont3, 1,201 segments	Fine typed spans	74.62	76.30	+1.68 [+1.01, +2.41]

Appendix

Table 28: O4 without and with the ±1-character boundary-refinement model. F1 and differences are percentage points; intervals use 10,000 paired document resamples on reused development populations. The CPU replay and scoring took 18 seconds.

Figure 22: Effect of annotated training-pool size on 659 development rows from 468 source documents, with non-bare references retained in the full view. Quarter and half support are nested document subsets; language weights are adjusted to meet the same coverage floors. Every arm uses the same optimization ceiling and selector rule. Lines connect the three observed pool sizes. The intervals concern paired + _reference differences; they measure resampling uncertainty on reused development data.

Figure 23: Gold sampling share at 4,000 updates on the expanded annotation pool. Dots are individual seeds and lines connect their means; no fitted trend or uncertainty band is implied. The selection criterion changes little above 40%, while higher shares favor human gold over fine Ont3. The 60% setting has only one seed.

Prompt	F1 (%)	ΔF1 (points)	Paired 95% CI	Time (s)
No prompt	72.53	0	—	2.64
10 shared tokens	71.65	−0.880	[−2.133, +0.394]	2.59
8 shared + 2 language tokens	71.94	−0.587	[−2.012, +0.848]	2.59
58 learned tokens, 29 at inference	73.16	+0.63	—	2.91
Gold status: 58 active in train, 29 at inference	73.34	+0.807	[−0.514, +2.210]	2.95
O4: no prompt	75.74	0	—	—

Appendix

Table 29: Exact typed-span soft-prompt controls. The first five rows use 545 development sentences, with _reference supervision and scoring disabled. The O4 rows average four paired training seeds on all 1,201 Ont3 inputs, with optional references and O4’s serving chain. Compare ΔF1 within each block; absolute scores have different populations and training conditions. ΔF1 and paired 95% confidence intervals are relative to no prompt, in F1 points. During training, each entity type receives a ? token and a gold-derived + or − token; at inference, only the ? tokens are visible to the encoder. The 58-token control uses learned tokens independent of gold status, with only the first 29 visible at inference. For the 545-input control, its interval versus no prompt is not reported here. Times cover forward computation and decoding of all 545 sentences; O4 timing was not measured under those conditions. O4 intervals resample both documents and the four paired seeds, 10,000 times.

Explore similar work

Jun 17, 2026cs.CL

RedactionBench

Large Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII). While redacting PII is a data cleaning prerequisite, existing benchmarks conflate extraction mechanics with privacy semantics. A public phone number is not equivalent to a phone number in a medical record. Whether information constitutes a violation depends heavily on who holds it, why, and in what context, fundamentally differentiating redaction from simple entity recognition. Grounded in contextual integrity, we introduce RedactionBench, a manually annotated benchmark comprising 200 diverse documents across 11 domains, mostly seeded from real-world sources. We also introduce R-Score, a novel character-level metric that treats semantically similar redactions equally and nullifies shallow formatting choices, such as varying masking styles for phone numbers. Evaluations across Named Entity Recognition models, entity extraction Small Language Models, and frontier models equipped with agentic tools demonstrate that contextual redaction remains an unsolved problem. A human evaluation with over 80 users on RedactionBench reveals a stark dichotomy in privacy perceptions. Annotators show consensus with target labels for mandatory redactions (89.4 percent) and safe text preservations (94.1 percent), but fail to agree on contextual redactions (47.7 percent). This variance demonstrates the subjective nature of contextual privacy and motivates R-Score, which decouples contextual ambiguity from strict precision. We compare 35 models across families and report their performance in redacting PII. Finally, we release RedactionBench to establish a baseline for future privacy-preserving systems, hoping to inspire efficient model design and standardized evaluations.

Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali +2

A10 Networks, Inc.

Named-Entity Recognition

Ask PDF

May 11, 2026cs.CL

GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction

Reliable detection of personally identifiable information (PII) is increasingly important across modern data-processing systems, yet the task remains difficult: PII spans are heterogeneous, locale-dependent, context-sensitive, and often embedded in noisy or semi-structured documents. We present GLiNER2-PII, a small 0.3B-parameter model adapted from GLiNER2 and designed to recognize a broad taxonomy of 42 PII entity types at character-span resolution. Training such systems, however, is constrained by the scarcity of shareable annotated data and the privacy risks associated with collecting real PII at scale. To address this challenge, we construct a multilingual synthetic corpus of 4,910 annotated texts using a constraint-driven generation pipeline that produces diverse, realistic examples across languages, domains, formats, and entity distributions. On the challenging SPY benchmark, GLiNER2-PII achieves the highest span-level F1 among five compared systems, including OpenAI Privacy Filter and three GLiNER-based detectors. We publicly release the model on Hugging Face to support further research and practical deployment of open PII detection systems.

Urchade Zaratiana, Ash Lewis, George Hurn-Maloney

Fastino Labs

Named-Entity Recognition Machine-Generated Text Detection Text Corpora Multilingual Language Models Distributions

Ask PDF

May 13, 2026cs.CL

Locale-Conditioned Few-Shot Prompting Mitigates Demonstration Regurgitation in On-Device PII Substitution with Small Language Models

Personally Identifiable Information (PII) redaction usually replaces detected entities with placeholder tokens such as [PERSON], destroying the downstream utility of the redacted text for retrieval and Named Entity Recognition (NER) training. We propose a fully on-device pipeline that substitutes PII with consistent, type-preserving fake values: a 1.5 B mixture-of-experts token classifier (openai/privacy-filter) detects spans, a 1-bit Bonsai-1.7B Small Language Model (SLM) proposes contextual surrogates for names, addresses, and dates, and a rule-based generator (faker) handles patterned fields. We report a prompting finding more important than the quantization choice: with naive fixed three-shot demonstrations, the 1-bit SLM regurgitates demonstration outputs verbatim regardless of input; 1.58-bit Ternary-Bonsai-1.7B reproduces byte-identical failures, ruling out quantization as the cause. We fix this with locale-conditioned rotating few-shot demonstrations: a character-range heuristic picks a locale-pure pool and a per-input MD5 hash samples three demonstrations. With the fix, 482/482 unique Bonsai-1.7B calls succeed (no echoes) and produce locale-correct surrogates, although the SLM still copies from a small same-locale demonstration pool - a residual narrowness we quantify. On a 2000-document multilingual corpus, hybrid perplexity (PPL) beats faker in all six locales under a multilingual evaluator (XGLM-564M); length preservation is best-of-three in 4 of 6 locales. On downstream NER (400 train / 100 test, English), redact yields F1=0.000, faker 0.656, original 0.960; on a matched 160/40 subset including hybrid, faker (0.506) outperforms hybrid (0.346) at p < 0.001. We report this as an honest negative finding: SLM surrogates produce more natural text but a less varied training distribution, and downstream NER benefits more from variety than from naturalness.

Anuj Sadani, Deepak Kumar

Substitute Small Large Language Models Surrogates Fake News Mitigation

Ask PDF