LLM assistants and agentic systems log long multi-turn conversations. AI providers often scan these conversations for Personally Identifiable Information (PII) and mask the PII before storing or processing conversation data. Yet most PII detectors and benchmarks target self-contained records rather than cross-turn evaluation. To evaluate PII detection across turns in multi-turn conversations, we introduce PII-TRACE (Tracing Recurring PII Across Conversational Exchanges), to our knowledge the first PII benchmark to assess whether detectors identify PII in conversational contexts and cover every mention of a recurring identifier across turns. PII-TRACE contains 13,148 synthetic multi-turn dialogues in 13 languages with character-level spans and identifier clusters. Across eleven baselines, including frontier LLMs, no detector achieves full entity-level coverage without substantial false positives on PII-free conversations, and single-pass reading loses a third of the gold characters on long dialogues. To close this gap, we introduce PII-Tracer, a compact 0.6B-parameter detector trained with conversation-level supervision. PII-Tracer attains the highest entity-level coverage of any system we evaluate and also performs strongly on standard single-record benchmarks.
Figures & tables
Figure 1: One identifier (highlighted) recurs across a conversation. PII-TRACE ’s consistent-detection metric counts it as covered only if every mention is found.
Benchmark
#Ex.
Langs
Setting
Entity-cluster
Cross-turn
ai4privacy ( ai4Privacy, 2024 )
225k
6
short document
✗
✗
Nemotron-PII ( NVIDIA, 2025b )
100k
1
document
✗
✗
SPY ( Savkin et al., 2025 )
8.7k
1
QA (author)
✗
✗
TAB ( Pilán et al., 2022 )
1.3k
1
legal doc
✓
✗
REDACT ( Vats et al., 2026 )
13.4k
25
mixed incl. chat
✗
✗
PII-Bench ( Shen et al., 2026 )
2.8k
1
query + description
✗
✗
Table 1: PII-TRACE vs. representative PII benchmarks: only PII-TRACE pairs multi-turn conversational text with cross-turn identifier clusters (TAB annotates coreference within single documents; REDACT includes flat chats among mixed formats but defines no cross-turn task). Sizes approximate.
Type
Covers
Mentions
private_person
name of a private person: full name, name part, handle
22,973
private_date
date identifying a person
3,381
private_url
personal URL or IP address
2,657
private_address
postal address, precise location
2,521
account_number
national ID, SSN, IBAN, card, account ids
2,013
private_email
personal email address
1,626
Table 2: The nine identifier types, with gold mention counts over all three splits (37,431 total; every occurrence counts separately).
Figure 2: Constructing PII-TRACE , on one example (source values withheld): a labeled conversation becomes a typed, cluster-preserving template; each turn is paraphrased with placeholders intact, and each placeholder is replaced by its entity’s surrogate with exact offsets. Failed paraphrases are retried, and gate-flagged documents can re-enter synthesis with fresh surrogates before a final drop (§ 3.5 ). Steps 1 and 2 are the anonymization of § 3.3 , steps 3 and 4 the synthesis of § 3.4 , and the gates, guarding the released record (step 5), the verification of § 3.5 .
Character
Span-Overlap
Span-Containment
Detector
P
R
F1
P
R
F1
P
R
F1
Presidio (rule/NER) ( Microsoft, 2024 )
0.045
0.762
0.086
0.042
0.844
0.079
0.036
0.741
0.069
OpenAI Privacy Filter (1.5B) ( OpenAI, 2026c )
0.358
0.785
0.492
0.266
0.817
0.401
0.380
0.363
0.371
GLiNER2-PII ( Zaratiana et al., 2026 )
0.211
0.475
0.292
0.181
0.525
0.269
0.173
0.514
0.259
GLiNER-PII (zero-shot) ( NVIDIA, 2025a )
0.261
0.785
0.392
0.209
0.890
0.338
0.272
0.483
0.348
Piiranha v1 ( iiiorg, 2024 )
0.161
0.299
0.209
0.117
0.339
0.174
0.078
0.157
0.104
Table 3: PII detection on the PII-TRACE test set (1,922 documents); all metrics label-agnostic, higher is better. The four frontier LLMs are prompted zero-shot; span scores match gold spans by overlap and by containment.
Presidio
OpenAI PF
GLiNER2-PII
GPT
Claude
PII-Tracer
5.4
5.6-sol
Opus 4.8
Sonnet 5
count
1
0.655
0.474
0.641
0.829
0.788
0.840
0.829
0.917
2
0.671
0.397
0.356
0.413
0.689
0.333
0.449
0.873
3–5
0.630
0.263
0.258
0.216
0.504
0.216
0.252
0.796
6–10
0.627
0.182
0.073
0.082
0.464
0.045
0.118
0.691
turn
cross-turn
0.648
0.296
0.292
0.275
0.551
0.222
0.305
0.776
Table 4: Consistent detection by an entity’s mention count and, for multi-mention entities, by turn span: each cell is the fraction of that group’s entities with every mention covered. Best per row in bold .
Length
Docs
PII docs
P
R
F1
< 1k
167
49
0.392
0.975
0.559
1k–10k
1,292
505
0.516
0.955
0.670
≥ 10k
463
243
0.507
0.687
0.583
Total
1,922
797
0.507
0.830
0.629
Table 5: PII-Tracer character P/R/F1 by conversation length, read in a single 4,096-token window. Docs: test documents per bucket; PII docs: those with gold spans; characters pooled per bucket.
Figure 3: Character F1 by language for six test languages covering the Latin, Cyrillic, and Hangul scripts.
Figure 4: PII-Tracer on five external PII benchmarks (label-agnostic character P/R/F1); gray: the OpenAI Privacy Filter under identical evaluation.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Sub-types
Definition / crosswalk
private_person
full name, name part, social handle/username
name of a private person; OMB/NIST direct, HIPAA name
private_email
email address
personal email; NIST, HIPAA electronic mail
private_phone
phone/fax number
phone of a private person; NIST, HIPAA
private_address
postal address, precise geo
location of a private person; HIPAA geographic
private_url
profile URL, IP address
URL/IP for a private audience; NIST online identifier
private_date
date of birth, identifying date
datetime identifying a person; HIPAA dates
Appendix
Table 6: The nine PII-TRACE identifier types: sub-types, one-line definition (a span is labeled only when it identifies a private person; placeholders and public entities are excluded), and regulatory crosswalk.
Detector
CD
CD-m
xCD
FP 0
Presidio
0.649
0.642
0.648
0.972
OpenAI PF
0.386
0.304
0.296
0.557
GLiNER2-PII
0.448
0.268
0.292
0.862
GLiNER-PII
0.472
0.349
0.334
0.901
Piiranha
0.128
0.090
0.097
0.685
OpenMed 44M
0.724
0.745
0.761
0.936
Appendix
Table 7: Entity-level results on the test split: consistent detection (CD), multi-mention CD (CD-m), cross-turn CD (xCD), and PII-free false-positive rate FP 0 (lower is better). Best per column in bold .
Policy
P
R
F1
CD
CD-m
R <10k
R ≥10k
trunc
0.507
0.830
0.629
0.853
0.794
0.956
0.687
chunk
0.502
0.944
0.656
0.916
0.894
0.956
0.931
slide
0.493
0.965
0.653
0.954
0.954
0.957
0.975
Appendix
Table 8: PII-Tracer under three long-input decoding policies: a single window (trunc), non-overlapping windows (chunk), and 50%-overlap sliding windows (slide). R <10k and R ≥10k split character recall at 10k characters.