cs.CLSep 29, 2026
SaveStrong Multilingual Privacy Tagging at Encoder Speed
Organizations: RWS Language Weaver
Abstract
Privacy redaction must remove personal information while preserving relationships expressed in text. We develop a multilingual named-entity tagger with fine-grained distinctions supporting varied redaction policies and methods for cheaply learning additional distinctions. We fine-tune a multilingual encoder with an affine span-tagging head on frontier-model annotations in 35 languages, replay mapped human gold with coverage-aware masking so unannotated types are not treated as negatives, and repair subword boundaries with a learned +/-1-character adjustment. On 1,283 human-gold test segments in seven languages, best measured redaction F1 is 88.8, against 69.1 for published GLiNER2 with 11 unrepresentable types excluded from its task (68.8 without that exemption), 67.8 for GLiNER2 adapted to the new training data, 57.3 for Microsoft Presidio and 35.8 for the best published OpenAI Privacy Filter fine-tune. Adding about 50,000 annotated training sentences and increasing human-gold replay improves exact typed-span F1 from 74.5 to 76.3 on Ont3, our 31-type frontier-annotated NER evaluation of 1,201 development segments. Mapped-gold replay alone raises human-gold F1 by ten points without loss on frontier-annotated text; boundary adjustment adds 1.7 exact typed-span F1 points on Ont3. Local LLMs fitting on a single 96-GB GPU underperformed as prompted annotators and frozen encoders, with encoding 30-95 times slower than XLM-R inference and prompted annotation roughly 180-1,100 times slower in the evaluated configurations. The encoder architecture delivers 4.9 times GLiNER2's CPU throughput. We release code, prompts and training recipes, with data-acquisition scripts and source links.
Figures & tables
Figure 1: Precision–recall on 1,283 human-gold segments in seven languages (top three rows) and on Ont3 (bottom row): the 302-segment seven-language subset at left and all 1,201 segments over 35 languages at right. Untyped redaction regions match at 80% overlap of both. Auxiliary refinements such as name components are not scored; models are not penalized for lacking them. GL and GL4: published and new-data-adapted GLiNER2; O2–O4: XLM - R variants; PR: multilingual Presidio, a fixed point; PF: OpenMed multilingual-v2 Privacy Filter. GL has an unequal advantage: gold spans outside its label coverage are unscored for GL alone; Figure 13 compares GL and O4 on the same restricted task. Across all five Privacy Filter variants, the best F1 stays below 40 on pooled human gold (35.8) and full Ont3 (39.4). Both axes start at 0.4, with off-scale Presidio points marked at the axis edge. O4’s best measured F1 is 88.8 on human gold and 82.1 on Ont3. Qualitatively similar exact span-boundary-required results appear in Section A.7 .
Figure 2: Components locate subspans of a name; a role labels the whole name span (illustrative).
Figure 3: Primary labels shared by Ont2 and Ont3, grouped by semantic family. Dashed amber circles mark Ont3’s two optional reference labels; there are no Ont2-only primary labels. Identifier-kind and name-component refinements are in Figure 9 .
Figure 4: Prompt assembly: fixed English instructions plus a demonstration in the target language (French shown). Ellipses omit other rules and annotations.
| Recipe | Unique non-human texts | Labeled tokens | Human-gold draws | Input |
|---|---|---|---|---|
| O3 | 38,700 | 477,685 | 40% | Neighboring sentences |
| O4 | 88,512 | 856,116 | 50% | Isolated sentence |
Table 1: Both models use Ont3 in 35 languages and replay the same 80,099 unique human-gold texts. Counts deduplicate language and text before sampling; labeled tokens include optional references. O2 uses a twenty-language mapped-gold recipe (Table 12 ).
| System | Hardware | Input tokens/s |
|---|---|---|
| XLM - R | CPU | 675 |
| GLiNER2 | CPU | 138 |
| XLM - R | L40S | 2,744 |
| GLiNER2 | L40S | 1,405 |
| Qwen3.8-Flash - Next | RTX PRO 6000 | 29 |
| Gemma - 4-31B base | RTX PRO 6000 | 93 |
Table 2: Throughput per labeled input token, excluding instructions and context. CPU is EPYC 7R13 with 16 threads. XLM - R and GLiNER2 use shared workloads and document batches on GPU (Section A.18 ). The three LLM rows measure encoding for simple BIOES heads, not prompted annotation output generation (Section A.5 ).
Appendix figures & tables44 assets
Supplementary material from the paper’s appendix.
Appendix
| Human-gold component | Inputs | Source labels used by the mapping | Accepted primary types | Gold spans with multiple accepted types / annotated spans |
|---|---|---|---|---|
| MAPA | 189 | 6 | 13 | 59 / 126 |
| AQMAR | 134 | 3 | 8 | 82 / 173 |
| OpenNER collection | 904 | 3 | 8 | 350 / 999 |
| Wojood subset | 56 | 11 | 14 | 52 / 99 |
Table 3: Annotation coverage of the frozen human-gold test subset. Counts refer to original annotated spans, before merging redaction regions. Source-label counts describe the evaluated schema, not every label in the original corpus.
Figure 5: Broad source labels can supervise several target types: MAPA’s ADDRESS permits locality, postal code and street address, among others. Filled squares mark fallback labels; open circles mark other acceptable labels. Shown: sixteen selected mappings from five corpora to the Ont2 label inventory.
| Resource | Terms and access |
|---|---|
| Nemotron-PII , OpenPII 1M and 1.5M | CC BY 4.0; public downloads. |
| Ai4Privacy 200k | Custom terms; for-profit organizations with more than three staff need a corporate license. Redistribution restrictions apply. |
| Ai4Privacy health-PHI 400k diagnostic sample | Gated access under custom commercial terms. |
| TAB , idner-news-2k and Wojood public sample | MIT; public repositories. Full Wojood is separate and was not used. |
| MEDDOCAN , SPY , MultiGraSCCo and MAPA EUR-Lex | CC BY 4.0; public distributions. Other MAPA packages may differ. |
| KLUE NER and HiNER | CC BY-SA 4.0; public distributions. |
Table 4: Data sources and reuse conditions for the acquired versions.
| Training scope | Updates | GPU / AWS instance | Hours | Estimated $ |
|---|---|---|---|---|
| Ont2-equivalent exposure | 33,300 | L40S / g6e.4xlarge | 6.0* | 8.5–18.0* |
| Ont3 fit from pretrained XLM - R | 16,000 | RTX PRO 6000 Blackwell Server / g7e.2xlarge | 2.3 | 4.3–5.4 |
Table 5: Training-only estimates at AWS Linux Spot prices observed on 22 September 2026: 1.87–2.37/hour in ap-south-1 for Ont3.
| Model | Failed format (%) | Character F1 | With XLM - R fallback* | Rescued subset F1* |
|---|---|---|---|---|
| Gemma - 4-12B | 5.01 | 83.84 | 86.38* | 84.51* |
| Gemma - 4-31B-FP8 | 6.22 | 83.98 | 90.11* | 85.13* |
| Qwen3.8-Flash - Next-NVFP4 | 8.19 | 76.03 | 87.02* | 90.77* |
| O4, sentence only | — | 91.53 | — | — |
Table 6: Annotation quality on the 659-segment Ont3 subset. Asterisks mark XLM - R substitution for invalid annotations; without fallback, invalid outputs count as empty predictions. Rescued subsets contain 33, 41 and 54 segments, respectively. O4 uses zero O bias, isolated input and boundary refinement.
| System | Params (B) | Active (B) | Labeled input tokens/s |
|---|---|---|---|
| Gemma - 4-12B | 12 | — | 2 |
| Gemma - 4-31B-IT, bfloat16 | 30.7 | — | 2.5 |
| Qwen3.8-Flash - Next-NVFP4 | 176 | 6 | 15 |
| XLM - R encoder | 0.55 | — | 2,744 |
Table 7: Gemma - 4-31B-IT serial generation measured on a 96-GB RTX PRO 6000: 35 inputs in 35 languages, repeated twice, 591 labeled input tokens per pass. Other prompted rates divide labeled input tokens by summed request durations, including retries. The encoder rate is measured throughput (Table 2 ). Parameter counts describe nominal backbones; the active count is shown for Flash-Next’s sparse main model, per token (Table 8 ).
| (Prompted) LLM / encoder | Params (B) | Head | Input state offsets | Typed span F1 (%) | Redaction-character F1 (%) |
|---|---|---|---|---|---|
| Flash-Next unprompted | 176 | Affine | 0 | 39.14 | 78.59 |
| Qwen3.8-Flash-Next | 176 | Affine | 0 | 44.56 | 80.64 |
| Qwen3.8-Flash-Next | 176 | Affine | −1, 0, +1 | 53.00 | 82.96 |
| Qwen3.8-Flash-Next | 176 | 512-unit GELU | 0 | 53.80 | 85.25 |
| Qwen3.8-Flash-Next | 176 | 512-unit GELU | −1, 0, +1 | 59.54 | 86.61 |
| Gemma - 4-31B base | 30.7 | 512-unit GELU | −1, 0, +1 | 62.41 | 87.07 |
Table 8: Flash - Next, Gemma and XLM - R heads on 659 Ont3 development segments from 468 source documents. Scores pool counts with optional references and zero O bias. Offsets are relative to the target token; 0 uses only its own state. LLM rows use the annotation prompt unless marked unprompted; XLM - R uses no prompt. Parameter counts are nominal backbone sizes in billions, excluding the BIOES head. Flash - Next has 125B main-model parameters (6B active per token) plus 51B n-gram embeddings; its unused 4B multi-token prediction module is excluded. Flash and Gemma heads use the same 32,821 training targets, 279-target selector and annotation prompt; the fit ceiling is ten epochs of 250 batches of 4,096 tokens. LLM heads train in about 15 minutes or less on frozen, already-computed encodings. The first three XLM - R heads receive 1,000 steps on a 40,303-window pool with the encoder frozen and the same selector. O4 is the fully fine-tuned serving model.
Figure 7: Precision–recall comparison on the same 659 Ont3 development inputs across 35 languages; axes start at 20% recall and 40% precision. Curves vary the O bias before constrained BIOES decoding; thin plus signs mark zero bias. Standalone points mark prompted annotation, whose format failures count as empty predictions without encoder fallback. Current corrected references and optional-reference scoring apply throughout; counts are pooled across inputs. Gemma - 4-12B uses layer-32 representations conditioned by a short task prompt. Flash - Next and both Gemma - 4-31B variants use matched 512-unit GELU heads over final-layer states at token offsets [−1, 0, +1], with the same training targets, annotation prompt and fitting schedule. O4 uses isolated input, human-gold replay and character-boundary refinement. Name-kind subspans are not scored.
Figure 8: Ont1’s union inventory. Branches show the recorded parent relations, not mutually exclusive subclasses. Under person_name, teal name components and orange person roles mix different annotation axes. The government_id branch likewise retains the finer distinctions contributed by source corpora.
Figure 9: Refinement heads read the same encoder states as the BIOES head. Token categories identify name components; independent binary predicates allow overlapping attributes. A whole-span category, such as identifier kind, is predicted from the mean token logits. Refinements do not affect the reported model comparisons.
Figure 10: The human-gold and Ont3 panels of Figure 1 , requiring identical redaction-region boundaries. GL has an unfair advantage: categories its labels cannot express are excluded from its scoring obligations, and it may omit job titles. Honorifics are optional; titles may be inside name spans or separate. All models receive corpus-coverage masks (Section A.9.1 ); Figure 13 compares O4 and GL on identical supported categories.
Figure 11: Ont3 comparison on 1,201 segments, including the O1 predecessor omitted from the main figure. Regions match at 80% symmetric overlap. GL has an unfair advantage: categories its labels cannot express are excluded from its scoring obligations, and it may omit job titles. Honorifics are optional; titles may be inside name spans or separate. All models receive corpus-coverage masks (Section A.9.1 ); Figure 13 compares O4 and GL on identical supported categories.
| Coverage | Importance | Language codes |
|---|---|---|
| Original 20 | ×4 | en |
| ×2 | ar, de, es, fr, ko, pt, vi, zh | |
| ×1 | cs, hi, id, it, ja, nl, pl, ru, sv, tr, uk | |
| +15 → 35 | ×1 | bn, da, el, fa, fi, fil, he, hr, ms, no, ro, ta, te, th, ur |
Table 9: Language weights in historical evaluation sets. The nine priority languages have weight ×2 or ×4. Main Ont3 and human-gold comparisons use pooled counts without language weights.
Figure 12: O4 and the original OpenAI Privacy Filter (PF) on a reduced task that scores only the entity categories covered by PF’s eight labels. Organizations, generic locations and every other category outside PF’s labels are removed from the task, gold and predictions alike, preserving many-to-many alternatives admitted by coarse human-gold labels, without retraining either model; characters inside removed gold regions are not scored. Curves sweep each model’s decision threshold; labels give each curve’s best character F1. Columns are human gold and Ont3; the top row uses all languages (1,283 and 1,201 segments), the bottom English only (176 and 56), the primary language named by PF’s model card. Both models use the same subset in each panel.
Figure 13: O4 and published GLiNER2 (GL) with identical category obligations. Both models are scored only on categories expressible by GL’s 42 requested labels plus organization, with the same title allowances and corpus-coverage masks. O4 predictions outside that inventory are removed as well, preserving many-to-many alternatives admitted by coarse human-gold labels. Unlike the main comparison, GL receives no exclusive category exemption. Untyped regions match at 80% symmetric overlap; labels give best measured region F1. Columns are human gold and Ont3. The Ont3 panels use only the 542 segments not used to select O4 or GL4. The top row uses all languages (1,283 and 542 segments); the bottom uses GL’s advertised English, French, Spanish, German, Italian, Portuguese and Dutch (898 and 171). Both models use the same subset in each panel.
| Updates | Human gold exact regions | Ont3 exact regions |
|---|---|---|
| 0 (transferred initialization) | 57.44 | 50.11 |
| 100 | 63.73 | 54.84 |
| 200 | 64.72 | 56.96 |
| 300 | 64.52 | 56.24 |
| 400 | 62.98 | 57.10 |
| 600 (selected) | 67.24 | 54.79 |
Table 10: Fixed-confidence F1 (%) on the paper’s 1,283 human-gold segments and the 659-segment Ont3 selection subset. With the same full-inventory scoring obligations, selected GL4 improves over the transferred initialization by 9.8 human-gold and 4.7 Ont3 F1 points. Stock GL’s category exemptions in the main graph obscure this adaptation gain. These are descriptive trajectory values; the smaller human-gold view does not choose the checkpoint.
| Baseline configuration | Languages | Documents | Our model’s F1 gain over baseline (points) | 95% interval (points) |
|---|---|---|---|---|
| GLiNER2 | 7 | 421 | +21.78 | [+19.14, +24.42] |
| OpenAI Privacy Filter | 1 | 61 | +47.31 | [+42.37, +52.25] |
| Multilingual OpenMed | 14 | 907 | +40.55 | [+37.63, +43.46] |
| OpenMed with additional task fine-tuning | 20 | 1,267 | +48.70 | [+46.08, +51.31] |
| OpenMed Nemotron | 9 | 541 | +43.76 | [+40.18, +47.35] |
Table 11: Ont2 encoder gains in untyped span-overlap F1 at fixed operating points on Fresh20. Each row uses only the baseline’s supported languages; differing subsets prevent ranking the baselines by these gains. Intervals use paired document resampling.
Figure 14: English. GLiNER2’s financial span extends from the name through the insurance number, including intervening prose. The encoder matches the gold boundaries; its finer identifier type differs.
Figure 15: Spanish. In this short clinical contact block, GLiNER2 labels only the surname; the encoder includes the given name as well. Both recover the address fields and email.
| Method | O2 | O3 | O4 |
|---|---|---|---|
| Starting encoder | Previously task-trained encoder, transferred to Ont2 | Upstream pretrained XLM - R-large | Ont3 encoder’s isolated-input branch |
| Language coverage | 20 languages | 35 languages | 35 languages |
| Supervision | Mapped annotations, translated data and Ont2 complete-label calibration mixture | Ont3 mixture plus 40% draws from four mapped human-gold corpora | 49,812 additional frontier-annotated texts; earlier labels cleaned; prompts clarify occupation/job titles as demographic_attribute , outside names. 50% mapped human-gold draws. |
| Primary label inventory | 29 types | Same 29 types plus two optional reference types | Same as O3 |
| Additional training objectives | Binary entity-presence loss, weight 2 | Subclass and type-conditioned predicate losses, each weight 1 | Same as O3 |
| O loss weight | 1 on complete annotations; 0 on partial annotations | 0.75 on supervised O tokens; corpus-unannotated types masked | Same as O3 |
Table 12: Active methods in O2, O3 and O4. O2 uses Ont2; O3 and O4 use Ont3. O4’s additional annotations, cleanup and sampling are detailed in Section A.16 . Boundary refinement is described in Section A.14 ; the main curves score primary redaction spans and ignore name-kind subspans.
Figure 16: BIOES labels mark a span’s beginning, interior, end or single token; O marks text outside spans. In this constructed example, independent winners switch fine type inside “St Mary Hospital.” Requiring one fine type loses the name; class-aware decoding recovers one organization span. Bold numbers are the independent winners. Scores are invented logits, not probabilities or measured model outputs; words stand in for tokenizer tokens.
| Decoder | Typed exact F1 (%) | Typed overlap F1 (%) | CPU decode time (s) |
|---|---|---|---|
| Matching-type Viterbi | 73.77 | 78.65 | 2.41 |
| Coarse-compatible greedy; first label | 70.51 | 75.61 | 0.32 |
| Coarse-class Viterbi; max scores | 74.00 | 78.91 | 1.54 |
| Coarse-class Viterbi; soft scores, T = 0.5 | 74.19 | 79.18 | 2.33 |
| Score-adjusted Viterbi; α = 0.95, T = 0.5 | 74.19 | 79.17 | 2.53 |
Table 13: Decoder contrast on the 487 Fresh7 development documents in seven languages, with the same XLM - R checkpoint logits and zero O bias. Decode times exclude encoder inference.
Figure 17: Exact fine-type Viterbi approaches the soft coarse-class decoder as its scores back off toward the normalized sibling aggregate. Points are the nine tested interpolation weights; dashed lines show the coarse-class decoder. F1 uses 20 shared reporting classes on the same 487 reused development documents. Fine-label quality is not measured by these curves.
| Token features and head | Head params | Initial screen F1 (%) | Repeat screen F1 (%) | Interpretation |
|---|---|---|---|---|
| final layer [24], affine | 357,725 | 82.20 | 82.14 | stable reference |
| middle + final [12,24], affine | 715,101 | 83.82 | 81.86 | initial gain does not repeat |
| layers [23,24], affine | 715,101 | — | 73.73 step 1,250 | early-screen loss |
| layers [8,24], affine | 715,101 | — | 71.67 step 1,250 | early-screen loss |
| embedding + final [0,24], affine | 715,101 | — | 69.16 step 1,250 | early-screen loss |
| final layer, 1,024-unit GELU | 1,409,373 | 69.21 step 1,250 | — | early-screen loss |
Table 14: XLM - R head and layer-feature screens on a fixed 192,000-window training subset. Entries report peak validation F1, except where an early stopping step is shown. The initial middle-layer gain did not repeat.
| Encoder and task head | Fresh6 F1 (%) [95% interval] | MultiGraSCCo F1 (%) [95% interval] | Disposition |
|---|---|---|---|
| XLM - R-large, final affine | 92.23 [91.41, 92.98] | 75.07 [74.27, 75.90] | quality base selected |
| LaBSE, final affine | 87.88 [86.68, 88.94] | 72.53 [71.69, 73.38] | common-head comparison |
| mmBERT-base, stock head | 89.25 [88.27, 90.17] | 70.61 [69.77, 71.49] | faster challenger; different head |
Table 15: Encoder systems trained on the same 1,854 MEDDOCAN/TAB documents and evaluated on Fresh6 and MultiGraSCCo. XLM - R and LaBSE each use an affine head; mmBERT uses its stock nonlinear head. Brackets give 95% intervals.
| Model and head | Human gold regions | Ont3 regions | Ont3 typed |
|---|---|---|---|
| mmBERT-base, stock GELU | 84.89 | 76.86 | 72.32 |
| mmBERT-base, affine | 87.03 | 77.02 | 73.90 |
| XLM - R-large O4, affine | 87.80 | 80.20 | 76.30 |
Table 16: Exact-span F1 (%) on 1,283 human-gold examples and 1,201 Ont3 development and held-out examples. The mmBERT head comparison uses O4’s full training pool and serving policy; O4 retains its own training lineage.
| Seed | Name repair: macro span Δ (points) [95% interval] | Name repair: macro fine Δ (points) [95% interval] |
|---|---|---|
| 154 | +1.57 [+0.79, +2.38] | +2.47 [+1.78, +3.20] |
| 155 | +1.44 [+0.64, +2.23] | +1.96 [+1.30, +2.62] |
| 156 | +1.88 [+0.92, +2.84] | +2.64 [+1.86, +3.42] |
Table 17: Name-order repair versus its matched control on 487 Fresh7 documents. Deltas are language-macro overlap-F1 points, with paired 95% intervals.
| Seed | Date repair: Korean span Δ (points) [95% interval] | Date repair: macro span Δ (points) [95% interval] |
|---|---|---|
| 154 | +5.08 [+2.64, +7.63] | +1.62 [+0.92, +2.33] |
| 155 | +7.53 [+4.61, +10.63] | +2.12 [+1.29, +2.96] |
| 156 | +4.38 [+1.06, +7.58] | +1.16 [+0.27, +2.04] |
Table 18: Korean date-format repair versus its matched control on Fresh7. Deltas are span-overlap-F1 points for Korean and the seven-language macro average, with paired 95% intervals.
| Step | Text |
|---|---|
| Source field | First Name: Kevin |
| Replace the name with a placeholder | First Name: [GIVEN_NAME_1] |
| TranslateGemma output | Nombre: [GIVEN_NAME_1] |
Table 19: Placeholder translation preserves the given-name annotation through its stable slot identifier. The translated marker is subsequently filled with an entity value.
| Surface | View | XLM - R Δ (F1 points) [95% interval] | mmBERT Δ (F1 points) [95% interval] |
|---|---|---|---|
| Fresh20 | span overlap | +0.59 [+0.13, +1.06] | +0.44 [+0.18, +0.71] |
| Fresh20 | fine overlap | +0.20 [-0.08, +0.49] | +0.30 [+0.08, +0.53] |
| MultiGraSCCo | span overlap | +0.15 [+0.04, +0.26] | +0.38 [+0.22, +0.53] |
| MultiGraSCCo | fine overlap | +0.05 [-0.04, +0.13] | +0.44 [+0.28, +0.61] |
Table 20: Checked translations versus unchanged training data, at matched exposure within each encoder. Fresh20 has 1,267 documents; MultiGraSCCo has 630 document-language instances. Deltas are F1 points with paired 95% intervals.
| Evaluation | Untyped spans | Fine labels |
|---|---|---|
| Fresh20 (twenty-language development set) | +1.36 [0.90, 1.82] | −0.45 [−1.02, 0.10] |
| MultiGraSCCo | +0.46 [0.24, 0.68] | −0.90 [−1.19, −0.61] |
Table 21: Routed translations, identifier-label repairs and locale changes evaluated with the same deployment decoder. Entries are language-macro overlap-F1 changes in points, with paired 95% intervals; untyped gains accompany fine-label losses.
| Stage | Accepted at this stage | Cumulative acceptance |
|---|---|---|
| TranslateGemma-27B, after marker repair | 68,309 / 69,480 (98.31%) | 98.31% |
| Gemma - 4-31B retry of the 1,171 failures | 1,007 / 1,171 (86.00%) | 99.76% |
Table 22: Translation-route acceptance after marker restoration and automated checks across eight target languages. Acceptance measures structural and quality-filter passes, not human-reviewed annotation accuracy.
Figure 19: Literal paired Spanish outputs from a nine-sentence, three-document pilot covering Spanish, Indonesian and Japanese. TranslateGemma-12B loses the surface delimiters; prompted Gemma - 4-12B preserves them and supplies replacement names.
| Model | Format passed | Format and replacement type passed |
|---|---|---|
| TranslateGemma-12B | 0% (0/5) | 0% (0/5) |
| Prompted Gemma - 4-12B | 100% (5/5) | 60% (3/5) |
Table 23: Joint translation and entity replacement on five entity-bearing sentences. Format requires intact markers and value delimiters; replacement type also requires a value of the requested entity type.
Figure 20: Recovery boundaries in saved transport output. The Turkish clinical example means “normal blood count”: its translated label name retains numeric ID 43 and an adjacent value. A Unicode-aware validator retrospectively recovers 19 Turkish rows in a 124-output, four-language TranslateGemma frame control. Detached values remain rejected. The county/city example summarizes the separate Gemma - 4 pilot’s semantic failure.
| Model | Before repair | After numeric-ID recovery |
|---|---|---|
| TranslateGemma-12B | 25.0% (31/124) | 40.3% (50/124) |
Table 24: Structural acceptance in a 124-output, four-language translation test. Recovering translated marker names by numeric ID rescues 19 Turkish outputs; semantic correctness was not measured.
| Weighted view | Mean delta (points) | Paired 95% interval (points) | Adjusted status |
|---|---|---|---|
| span overlap | +1.00 | [+0.73, +1.40] | win |
| character precision | -0.26 | [-0.38, -0.17] | loss |
| character recall | +2.80 | [+2.40, +3.20] | win |
| character F1 | +1.50 | [+1.20, +1.70] | win |
| fine overlap | +0.11 | [-0.20, +0.42] | unresolved |
Table 25: Ignoring unmarked Gemma teacher tokens versus treating them as background. The contrast averages the two positive-only seeds 155 and 156 and subtracts the existing complete-supervision seed-155 baseline. Results are on Fresh20, using 10,000 paired document resamples. Intervals are per metric; win/loss means the interval excludes zero after simultaneous adjustment across 276 comparisons, and unresolved means it includes zero.
| Annotator | Span density relative to Luna | Without _reference tags |
|---|---|---|
| Gemma - 4-31B | 82% | 90% |
| Qwen3.8-27B | 61% | 75% |
Table 26: First-pass span counts relative to Luna on the same 1,113 Ont3 development segments in 35 languages. Ratios measure annotation density, not recall; span splitting and false positives also affect them.
Figure 21: Ont2 development accuracy and fresh legal evaluation. Exact typed span F1 is measured separately on each population. The clinical-heavy MEDDOCAN, MultiGraSCCo and TAB development set covers 16 types; the older Final20 retention set covers 29. Fresh English HUDOC legal text reveals a large gap despite high development scores.
| Model | Languages | Sentences | Exact typed span F1 (%) | Character-redaction F1 (%) |
|---|---|---|---|---|
| Ont2 | Shared 20 | 374 | 52.52 | 76.63 |
| Ont3 | Shared 20 | 374 | 73.21 | 91.58 |
| Ont2 | 35 | 659 | 51.33 | 75.09 |
| Ont3 | 35 | 659 | 72.89 | 91.17 |
Table 27: Ont2 and the pre-replay, sentence-only Ont3 model on identical development text, with optional references. The shared-language difference is +20.68 span-F1 points (95% paired interval [+16.24, +25.20]).
| Population | Exact-match task | Without model | With model | ΔF1 (95% interval) |
|---|---|---|---|---|
| Human gold, 1,283 segments | Redaction regions | 87.87 | 87.80 | −0.07 [−0.23, 0.00] |
| Ont3, 1,201 segments | Redaction regions | 77.71 | 80.20 | +2.49 [+1.58, +3.50] |
| Ont3, 1,201 segments | Fine typed spans | 74.62 | 76.30 | +1.68 [+1.01, +2.41] |
Table 28: O4 without and with the ±1-character boundary-refinement model. F1 and differences are percentage points; intervals use 10,000 paired document resamples on reused development populations. The CPU replay and scoring took 18 seconds.
Figure 22: Effect of annotated training-pool size on 659 development rows from 468 source documents, with non-bare references retained in the full view. Quarter and half support are nested document subsets; language weights are adjusted to meet the same coverage floors. Every arm uses the same optimization ceiling and selector rule. Lines connect the three observed pool sizes. The intervals concern paired + _reference differences; they measure resampling uncertainty on reused development data.
Figure 23: Gold sampling share at 4,000 updates on the expanded annotation pool. Dots are individual seeds and lines connect their means; no fitted trend or uncertainty band is implied. The selection criterion changes little above 40%, while higher shares favor human gold over fine Ont3. The 60% setting has only one seed.
| Prompt | F1 (%) | ΔF1 (points) | Paired 95% CI | Time (s) |
|---|---|---|---|---|
| No prompt | 72.53 | 0 | — | 2.64 |
| 10 shared tokens | 71.65 | −0.880 | [−2.133, +0.394] | 2.59 |
| 8 shared + 2 language tokens | 71.94 | −0.587 | [−2.012, +0.848] | 2.59 |
| 58 learned tokens, 29 at inference | 73.16 | +0.63 | — | 2.91 |
| Gold status: 58 active in train, 29 at inference | 73.34 | +0.807 | [−0.514, +2.210] | 2.95 |
| O4: no prompt | 75.74 | 0 | — | — |
Table 29: Exact typed-span soft-prompt controls. The first five rows use 545 development sentences, with _reference supervision and scoring disabled. The O4 rows average four paired training seeds on all 1,201 Ont3 inputs, with optional references and O4’s serving chain. Compare ΔF1 within each block; absolute scores have different populations and training conditions. ΔF1 and paired 95% confidence intervals are relative to no prompt, in F1 points. During training, each entity type receives a ? token and a gold-derived + or − token; at inference, only the ? tokens are visible to the encoder. The 58-token control uses learned tokens independent of gold status, with only the first 29 visible at inference. For the 545-input control, its interval versus no prompt is not reported here. Times cover forward computation and decoding of all 545 sentences; O4 timing was not measured under those conditions. O4 intervals resample both documents and the four paired seeds, 10,000 times.