WebPII: Benchmarking Visual PII Detection for Computer-Use Agents

Nathan Zhao

cs.CRMar 18, 2026

WebPII: Benchmarking Visual PII Detection for Computer-Use Agents

Authors: Nathan Zhao

Organizations: Stanford University

Abstract

Computer use agents create new privacy risks: training data collected from real websites inevitably contains sensitive information, and cloud-hosted inference exposes user screenshots. Detecting personally identifiable information in web screenshots is critical for privacy-preserving deployment, but no public benchmark exists for this task. We introduce WebPII, a fine-grained synthetic benchmark of 44,865 annotated e-commerce UI images designed with three key properties: extended PII taxonomy including transaction-level identifiers that enable reidentification, anticipatory detection for partially-filled forms where users are actively entering data, and scalable generation through VLM-based UI reproduction. Experiments validate that these design choices improve layout-invariant detection across diverse interfaces and generalization to held-out page types. We train WebRedact to demonstrate practical utility, more than doubling text-extraction baseline accuracy (0.753 vs 0.357 mAP@50) at real-time CPU latency (20ms). We release the dataset and model to support privacy-preserving computer use research.

Figures & tables

Panel of Figure 1 — Figure 1: Sample images from WebPII , rendered with different injected data. The dataset captures the visual complexity of e-commerce interfaces: variable page heights reflecting diverse checkout flows and product displays (compare compact cart in (a) with extended layout in (d)), input fields and dropdown selectors, modal overlays with backdrops that occlude underlying content, ad hoc identifying information such as gift messages (c) and proposed pickup locations (e), and derived values requiring computation of taxes and totals. Bounding boxes respect occlusion boundaries. Pink indicates product annotations, purple denotes empty input fields, and red identifies PII.

Figure 2: Data injection maps configuration values to rendered UI elements. Left: annotated screenshot with bounding boxes. Right: selected subset of data injected for this page—Faker-generated PII, ABO product data, LLM-extracted metadata, and values derived at render time. The same layout rendered with different configurations produces diverse training examples with automatic annotations.

Panel of Figure 3 — Figure 3: Form fill states for anticipatory detection. (a) Partial: mid-entry state with later fields incomplete (city field shows “New M” mid-typing). (b) Empty: pristine form with placeholder text and input field annotations. Yellow indicates partially filled fields; grey denotes empty fields.

Method	mAP@50	Latency
OCR + Presidio	0.183	1.3s
LayoutLMv3 + GPT-4o-mini	0.357	2.9s
WebRedact (ours)	0.753	20ms
WebRedact-large (ours)	0.842	312ms

Table 1: Baseline results on Test

{}_{\text{Cross-Company}}

(Amazon, full-fill images only). OCR+Presidio uses Tesseract; LayoutLMv3 uses GPT-4o-mini for classification.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Size	Visual	Semantic	Web	Ext.	Antic.
AI4Privacy ai4Privacy (2023)	Text	300K	✗	✓	✗	✗	✗
PANORAMA Selvam and Ghosh (2025)	Text	385K	✗	✓	✗	✗	✗
Nemotron-PII Steier et al. (2025)	Text	100K	✗	✓	✗	✗	✗
Gretel Finance Gretel AI (2024)	Text	56K	✗	✓	✗	✗	✗
BigCode PII Li et al. (2023)	Code	12K	✗	✓	✗	✗	✗
PIILO Holmes et al. (2023)	Essays	22K	✗	✓	✗	✗	✗

Appendix

Table 2: Comparison of PII detection benchmarks. WebPII is the first to combine visual localization, semantic categories, web interface targeting, extended identifiers, and anticipatory form state support.

Figure 4: Dataset composition and statistics. (a) Distribution across form-fill variants—empty forms (13.4%), fully-filled forms (22.7%), and partial-fill states (63.9%)—enabling anticipatory detection training. (b) Annotation density distribution with median of 19 boxes per image (mean 22.1), ranging from 0 to 145 annotations per image. (c) Breakdown of all 9 annotation classes, with address (25.7%), order info (23.0%), and product text (18.8%) dominating. PII classes (red) comprise 52.4% of annotations, while non-PII classes (blue) comprise 47.6%. (d) HTML element type distribution, showing most annotations target rendered text (78.1%) versus input fields (13.6%) and images (8.3%).

Figure 5: Distribution of base images across 10 e-commerce companies. Apple and Amazon have the most coverage (1,400 images each), while Slack and Ulta Beauty represent smaller verticals (300 images each). Total base images: 10,200 (multiplied across variants to produce 44,865 total dataset images).

Figure 6: Distribution of base images across 19 page types spanning checkout flows, account management, and product pages. Account selection (900) and delivery/shipping (875) pages dominate, representing critical moments where PII exposure is highest.

By Company
Category	Samples Refined	Total Fixes
Home Depot	22	56
Lowes	23	44
Amazon	19	43
Apple	25	39
Macy’s	13	39

Appendix

Table 3: Distribution of human-initiated refinements across companies and page types, ordered by total fixes.

Figure 7: Distribution of refinement iterations required per layout across 160 layouts requiring refinement. Over half (55.6%) converged after a single correction iteration, while 21.9% required two iterations. Four layouts required more than 5 iterations.

Figure 8: Breakdown of 312 refinement iterations by error type across 160 layouts. Layout issues (53.8%, primarily viewport overflow) dominated refinements. Data attribute errors (24.4%) involved spurious attributes on decorative components (particularly in Apple’s minimalist interfaces) and missing markup on product metadata fields. Incomplete fields (14.7%) included dropdown initialization issues. Hardcoded store locations (7.1%) required parametrization.

Model	Resolution	mAP@50	Precision	Recall
WebRedact	640 $\times$ 640	0.807	0.848	0.762
WebRedact-large	1280 $\times$ 1280	0.909	0.868	0.834

Appendix

Table 4: Architecture comparison (2-class, cross-page, 10 text variants, trained on full+partial).

Test Variant	mAP@50	Precision	Recall
full	0.886	0.896	0.815
partial	0.943	0.897	0.878
empty	0.898	0.809	0.809
Average	0.909	0.868	0.834

Appendix

Table 5: WebRedact-large per-variant performance (2-class, cross-page, 10 text variants).

Model	CPU (i5/Ryzen 5)	GPU (RTX 30xx)
WebRedact	$\sim$ 20ms	$<$ 5ms
WebRedact-large	$\sim$ 312ms	$\sim$ 13ms

Appendix

Table 6: Inference latency comparison (milliseconds per image).

Method	mAP@50	Prec.	Rec.	F1	Latency
LayoutLMv3 Huang et al. (2022)	0.357	58.5%	45.1%	50.9%	2.9s
Donut Kim et al. (2021)	0.350	55.8%	43.6%	48.9%	3.5s
Tesseract Smith (2007) + LLM	0.302	49.8%	49.6%	49.7%	2.8s

Appendix

Table 7: Document understanding models vs. OCR+LLM baselines on Test

{}_{\text{Cross-Company}}

(Amazon, full-fill images). All methods use GPT-4o-mini for classification.

OCR Engine	mAP@50	Prec.	Rec.	F1	Latency
PaddleOCR Cui et al. (2025)	0.329	78.7%	31.0%	44.5%	6.6s
Tesseract Smith (2007)	0.302	49.8%	48.6%	49.2%	2.8s
EasyOCR JaidedAI (2024)	0.286	67.6%	30.7%	42.2%	3.8s

Appendix

Table 8: OCR engine ablation with GPT-4o-mini classification.

OCR Engine	Latency
Tesseract Smith (2007)	453ms
EasyOCR JaidedAI (2024)	715ms
PaddleOCR Cui et al. (2025)	2,143ms

Appendix

Table 9: OCR engine latency (extraction only, no classification).

OCR Engine	mAP@50	Prec.	Rec.	F1	Latency
Tesseract Smith (2007)	0.183	41.7%	26.3%	32.2%	1.3s
EasyOCR JaidedAI (2024)	0.178	42.1%	28.3%	33.9%	2.3s
PaddleOCR Cui et al. (2025)	0.176	44.5%	25.1%	32.1%	4.8s

Appendix

Table 10: OCR + Presidio NER (no LLM).

Method	mAP@50	Prec.	Rec.	F1	Latency
LayoutLMv3 Huang et al. (2022) + LLM	0.357	58.5%	45.1%	50.9%	2.9s
Donut Kim et al. (2021) + LLM	0.350	55.8%	43.6%	48.9%	3.5s
LayoutLM-QA	0.139	24.1%	47.8%	32.0%	1.5s
Donut-DocVQA	0.117	30.8%	37.7%	33.9%	11.5s

Appendix

Table 11: Document QA vs. LLM classification on Amazon test images.

Split	Text	Image
Cross-Page	0.623	0.870
Cross-Company	0.471	0.855
Cross-Type	0.552	0.875

Appendix

Table 12: Per-class AP@50 on fully-filled test images (test_full variant).

Split Strategy	full	partial	empty	Avg
Test ${}_{\text{Cross-Page}}$	0.747	0.842	0.803	0.797
Test ${}_{\text{Cross-Company}}$	0.663	0.839	0.756	0.753
Test ${}_{\text{Cross-Type}}$	0.713	0.727	0.744	0.728

Appendix

Table 13: Split strategy comparison ( WebRedact , 2-class, trained on full+partial). Test

{}_{\text{Cross-Page}}

performs best, while Test

{}_{\text{Cross-Type}}

presents the strongest generalization challenge.

Train	full	part.	empty	Avg
empty	0.672	0.818	0.777	0.756
full	0.736	0.810	0.767	0.771
full+empty	0.734	0.774	0.768	0.758
full+part.	0.747	0.842	0.803	0.797
full+part.+empty	0.762	0.881	0.831	0.825

Appendix

Table 14: Fill state ablation on Test

{}_{\text{Cross-Page}}

( WebRedact , 2-class). Partial-fill data is essential; empty screenshots help only when combined with partials.

Partials	Train	test_full	test_partial	test_empty	Avg mAP	Prec.	Rec.
1	521	0.727	0.774	0.774	0.758	0.793	0.688
3	763	0.738	0.794	0.779	0.770	0.811	0.709
5	938	0.762	0.835	0.810	0.802	0.822	0.727

Appendix

Table 15: Progressive fill density ablation ( WebRedact , 2-class, cross-page, trained on full+partial). Shows per-variant breakdown and overall metrics.

Variants	test_full	test_partial	test_empty	Avg mAP	Prec.	Rec.
1	0.762	0.831	0.791	0.795	0.805	0.713
10	0.774	0.856	0.804	0.811	0.829	0.722
25	0.783	0.866	0.812	0.820	0.842	0.731

Appendix

Table 16: Text variant ablation on Test

{}_{\text{Cross-Page}}

( WebRedact , 2-class, trained on full+partial). Shows per-variant breakdown and overall metrics.

Panel of Figure 9 — Figure 9: Qualitative comparison of WebRedact (left) and WebRedact-large (right) predictions on cross-page test images. Green boxes indicate correct alignment with ground truth, red boxes indicate false positives, and blue boxes indicate false negatives.

PII Detection

Personally identifiable information (PII) detection finds sensitive personal details in text, conversations, images, and other data.

See all 39 papers

REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection
Jun 18, 2026Guneesh Vats, Anubha Agrawal, Shikha Singhal +5Privacy Auditing PII Detection
ServiceNow
"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents
May 30, 2026Soham Roy, Sarthakbrata Halder, Arya Bharaty +5Data Leakage LLM Agent Security
KIIT Bhubaneshwar · BITS Pilani · Lam Research
Broad and Efficient PII Detection with Unified Labels
Oct 15, 2026Andy Hsu, Konstantin BerlinPII Detection Responsible AI
Cisco AI Defense

Jun 18, 2026J/K move · Enter open · S save

REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection

Guneesh Vats, Anubha Agrawal, Shikha Singhal, Ajita Dash+4

ServiceNow

Privacy Auditing PII Detection

Open paper AskPDF arXiv

Benchmark infrastructure for personally identifiable information (PII) detection remains limited: existing corpora cover few entity types, use ad hoc generation conditions, and do not show which surface conditions cause detector failures. We present REDACT, a systematically controlled multilingual PII benchmark with 13,427 records, 324,078 entity annotations, 51 entity types, 4,127 surface-form patterns, and 25 languages across 9 scripts. A strength-2 covering-array sampler controls nine generation axes: domain, format, difficulty, length, density, code-switching, language, adjacency, and co-occurrence. Three entity-level metadata fields (disclosure status, disclosure form, and a GDPR-aligned sensitivity tier) enable stratified evaluation beyond aggregate or per-type F1. From the full benchmark, we evaluate five detectors (Presidio, GLiNER, the OpenAI Privacy Filter, GPT-4.1, and Claude Sonnet 4.6) on a locked, language-stratified sample of 1,000 records. Aggregate F1 masks an architecture-dependent failure structure: the rule-based detector performs poorly on the highest-stakes data, including HIGH-sensitivity categories (recall 0.07) and non-verbatim disclosure forms, while the LLM detectors remain more robust, with the HIGH tier as their strongest sensitivity slice. A three-model reference-free LLM-as-judge assessment corroborates that sensitivity-tier assignment is the task's hardest axis. We release the benchmark, schema, prompts, and stratified evaluation harness.

Abstract

PII Detection

Explore similar work

REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection

"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents

Broad and Efficient PII Detection with Unified Labels