PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI Agents

Zian Yuan; Wei Li

cs.CROct 12, 2026

PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI Agents

Authors: Zian Yuan, Wei Li

Abstract

Cloud graphical user interface (GUI) agents send instructions and screenshots to remote planners, exposing personally identifiable information (PII). Existing placeholder defenses mask detected spans and reuse exact strings. Fragmented or changed forms can expose PII or break entity continuity across turns. PrivacyGuard addresses this private reference problem with source-linked sanitization (SAN) and a structured reference lifecycle (REF). SAN reconstructs fragmented or overlapping evidence and rewrites the complete source extent. REF links changed appearances to session entities and supports locally resolvable references without returning raw values to the upload path. In five-turn sessions scored against common gold, PrivacyGuard reduces audited source-value leakage by 33.55%--82.60% relative to the closest prior full-pipeline system in all four reader--language combinations. REF raises strict cross-form continuity from 0.04 under exact-value reuse to 0.81. In reference-required Android tasks, verified completion rises from 8/30 and 7/30 to 17/30 and 20/30. On AndroidLab, PrivacyGuard achieves the highest or tied-highest task-success point estimate among protection pipelines for all four planners. On PrivScreen, it reduces fixed-reader field exposure by 62.72%--62.84% relative to the same baseline. PrivacyGuard lowers upload exposure while preserving a locally resolvable task path.

Figures & tables

Figure 1: PrivacyGuard’s device-resident architecture. OCR and processing feed a shared detection and interpretation path. SAN rewrites accepted source-linked occurrences, while REF binds the resulting typed entities to session references and renders them in the released instruction or screenshot. The local privacy repository supplies detection evidence and the session mapping store retains roots, projections, and raw values on the trusted client. A protocol-conforming remote planner receives the released view and may return reference-bearing actions, which are validated, resolved, and restored locally before GUI-agent execution.

		English		Chinese
Input	Method	GenR @1.0 $\uparrow$	$\mathrm{LR}_{\mathrm{char}}$ $\downarrow$	GenR @1.0 $\uparrow$	$\mathrm{LR}_{\mathrm{char}}$ $\downarrow$
Text	Best baseline	0.7912	0.1283	0.5048	0.3998
	Ours	0.9740	0.0125	0.9971	0.0023
Address stress	AbI	0.3448	0.5094	0.0986	0.8921
	PrA	0.1160	0.3968	0.1944	0.6628
	PrG	0.6247	0.0537	0.6567	0.0114

Table 1: SAN source removal on controlled text and structured-address inputs.

Lang.	Method	Native $\mathrm{LR}_{\mathrm{detect}}\downarrow$	Shared Det-P $\uparrow$	Shared [email protected] $\uparrow$	Shared $\mathrm{LR}_{\mathrm{detect}}\downarrow$
Dataset: CrossTurnPII
EN	AbI	0.1850	0.3249	0.5521	0.3397
EN	PrA	0.1733	0.4313	0.5629	0.0482
EN	PrG	0.4454	0.4542	0.5486	0.3071
EN	Ours	0.0316	0.8125	0.9113	0.0316
ZH	AbI	0.5673	0.9802	0.3720	0.5657

Table 2: SAN source removal on controlled CrossTurnPII and external PrivScreen screenshots.

Mechanism	Endpoint	Without	With
Adjudication	EN address [email protected] $\uparrow$	0.3315	0.8981
OCR recon.	PrivScreen [email protected] $\uparrow$	0.7219	0.8600
	PrivScreen Det-P $\uparrow$	0.6760	0.9122
Local repo.	EN prompt [email protected] $\uparrow$	0.9740	0.9867
	ZH prompt [email protected] $\uparrow$	0.9971	1.0000

Table 3: Matched SAN mechanism tests with same inputs.

Planner	SAN	Local	Exact	Full
Qwen	0	0/30	8/30	17/30
Gemini	0	1/30	7/30	20/30

Table 4: REF capability and continuity on separate workloads. Panels use different denominators and are not pooled.

	Single view		Five turns
Reader	AbI	Ours	AbI	Ours
Qwen	0.0602	0.0130	0.0622	0.0299
Gemini	0.0716	0.0159	0.0644	0.0273

Table 5: REF reconstruction cost on CrossTurnPII.

Lang.	Reader	AbI	Ours	$\Delta$ [95% CI]
EN	Qwen	0.3486	0.2080	$-0.1406$ [ $-0.1698,-0.1080$ ]
EN	Gemini	0.3271	0.2174	$-0.1098$ [ $-0.1429,-0.0750$ ]
ZH	Qwen	0.7022	0.1222	$-0.5801$ [ $-0.6126,-0.5458$ ]
ZH	Gemini	0.7163	0.1426	$-0.5737$ [ $-0.6049,-0.5431$ ]

Table 6: CrossTurnPII audited source-value leakage on common-gold five-turn Full uploads.

\Delta

is Ours minus AbI.

	Qwen		Gemini
Method	Exposure $\downarrow$	Utility $\uparrow$	Exposure $\downarrow$	Utility $\uparrow$
AbI Full	0.2134	0.914	0.2229	0.950
Ours Full	0.0793	0.910	0.0831	0.966

Table 7: PrivScreen privacy exposure and visual-answer utility for the two Full pipelines.

Planner	Original	AbI Full	Ours Full
Gemini-3-Flash-Preview	60.87	57.25	63.04
Qwen3.7-Plus	55.80	56.52	56.52
Qwen3.7-Max	58.70	55.07	55.80
GPT-4o	45.65	38.41	42.03

Table 8: AndroidLab task success on the same 138 tasks.

Metric	Value
OCR latency, mean	1142.6 ms
Preprocessing latency, mean	60.5 ms
Detection latency, mean	35.7 ms
Entity-binding latency, mean	18.6 ms
Reference-rendering latency, mean	120.9 ms
Return resolution and restore, mean	15.5 ms

Table 9: Packaged Android-prototype latency and memory.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Lang	[email protected] $\uparrow$	[email protected] $\uparrow$	$\mathrm{LR}_{\mathrm{char}}\downarrow$	EPred_gt $\downarrow$	ECov_char $\downarrow$
Presidio + rules	EN	0.4985	0.4985	0.5940	0.1250	0.5356
PrA	EN	0.6749	0.6749	0.2627	0.4782	0.0070
PrG	EN	0.7912	0.8593	0.1283	0.3089	0.0829
AbI	EN	0.5456	0.8212	0.2979	0.7252	0.0000
Ours	EN	0.9740	0.9794	0.0125	0.0016	0.0000
Presidio + rules	ZH	0.4725	0.4725	0.4767	0.0000	0.0000

Appendix

Table 10: Full structured-prompt results.

Lang	PII type	$N$	AbI	PrG	Presidio + rules	PrA	Ours
EN	Account ID	126	0.9841/0.0202	0.6111/0.1759	0.9762/0.0151	0.0000/0.8652	0.9365/0.0469
EN	Address	625	0.0000/0.5208	0.0000/0.2245	0.0000/0.8237	0.0000/0.4218	0.9200/0.0136
EN	Bank card	150	1.0000/0.0000	1.0000/0.0000	1.0000/0.0000	1.0000/0.0000	1.0000/0.0000
EN	Driver license	328	1.0000/0.0000	1.0000/0.0000	1.0000/0.0000	0.9848/0.0000	1.0000/0.0000
EN	EMAIL	262	1.0000/0.0000	0.9924/0.0004	1.0000/0.0000	1.0000/0.0000	1.0000/0.0000
EN	License plate	135	0.9926/0.0000	1.0000/0.0000	0.3704/0.5825	0.0000/0.8653	1.0000/0.0000

Appendix

Table 11: Per-type structured-prompt source removal. Cells are [email protected] /

\mathrm{LR}_{\mathrm{char}}

Method	Surfaces	Protected	Neutral	Explicit neg.
Ours	319	269 (84.33%)	26 (8.15%)	24 (7.52%)
AbI	238	182 (76.47%)	32 (13.45%)	24 (10.08%)

Appendix

Table 12: Author-reviewed AndroidLab protection-output composition.

Planner	Baseline	$\Delta$ SR pp [95% CI]	Ours win/loss/tie
Qwen3.7-Plus	SAN-only	$6.52$ [ $-1.45$ , $14.49$ ]	20/11/107
Qwen3.7-Plus	Original	$0.72$ [ $-5.80$ , $7.25$ ]	12/11/115
Qwen3.7-Plus	AbI	$0.00$ [ $-7.25$ , $7.25$ ]	12/12/114
GPT-4o	Original	$-3.62$ [ $-10.14$ , $2.90$ ]	8/13/117
GPT-4o	AbI	$3.62$ [ $-4.35$ , $11.59$ ]	17/12/109
Gemini-3-Flash-Preview	SAN-only	$2.90$ [ $-2.90$ , $8.70$ ]	11/7/120

Appendix

Table 13: Task-paired uncertainty for AndroidLab utility.

Planner	SAN-only	Full	$\Delta$ pp
Gemini-3-Flash-Preview	60.14	63.04	$+2.90$
Qwen3.7-Plus	50.00	56.52	$+6.52$

Appendix

Table 14: AndroidLab SAN-only-to-Full task-success contrast.

Variant	EN $N$	EN (Exact / Under / Empty)	ZH $N$	ZH (Exact / Under / Empty)
to_city	360/360	1.000 / 0.000 / 0.000	360/360	1.000 / 0.000 / 0.000
to_district	360/360	1.000 / 0.000 / 0.000	360/360	0.936 / 0.064 / 0.000
to_road	339/360	1.000 / 0.000 / 0.000	360/360	0.758 / 0.242 / 0.000
full	360/360	0.000 / 1.000 / 0.000	360/360	0.000 / 1.000 / 0.000
ocr_noise	360/360	0.000 / 1.000 / 0.000	360/360	0.000 / 1.000 / 0.000

Appendix

Table 15: Representative SPEC-aware address projections.

Method	2-block		3-block
	EN	ZH	EN	ZH
Presidio + rules + Exact	0.442 / 0.000	0.000 / 0.000	0.383 / 0.000	0.000 / 0.000
PrA+Exact	0.979 / 0.000	0.681 / 0.000	0.881 / 0.000	0.582 / 0.000
PrG+Exact	1.000 / 0.000	0.475 / 0.000	1.000 / 0.000	0.482 / 0.000
AbI	0.971 / 0.000	0.367 / 0.000	0.708 / 0.000	0.250 / 0.000
Ours	0.992 / 0.925	0.952 / 0.667	0.944 / 0.817	0.915 / 0.538

Appendix

Table 16: Detection and reassembly of split-block addresses.

Lang	Method	$C_{\mathrm{pair}}\uparrow$	$\mathrm{PCR}_{\mathrm{cond}}\uparrow$	$\mathrm{PCR}_{\mathrm{strict}}\uparrow$	Chain $\uparrow$
EN	GLiNER + AbI	0.2538	0.5773	0.1465	0.0000
EN	PrA+Exact	0.0882	0.3921	0.0346	0.0000
EN	PrG+Exact	0.9150	0.0546	0.0499	0.0000
EN	Ours	0.8886	0.9098	0.8084	0.8708
ZH	GLiNER + AbI	0.0047	0.0000	0.0000	0.0000
ZH	PrA+Exact	0.0313	0.2351	0.0074	0.0000

Appendix

Table 17: Full cross-type continuity matrix at

\tau=0.80

Planner	Reference case	Family	Turn-local	Exact reuse	Full lifecycle
Qwen	PHONE	Root-like	0/6	5/6	6/6
	FULL_ADDRESS	Root-like	0/6	3/6	5/6
	CITY	Component-like	0/6	0/6	3/6
	DISTRICT	Component-like	0/6	0/6	1/6
	DTL	Component-like	0/6	0/6	2/6
Gemini	PHONE	Root-like	0/6	5/6	4/6

Appendix

Table 18: Automatically verified reference paths by planner and case.

Outcome	Qwen	Gemini	Automatic definition
Successful reference path	17	20	One issued target reference has the required projection and restores to gold.
Reference not submitted	7	5	The final answer does not submit a target-class reference.
Projection or granularity mismatch	5	0	The submitted address projection does not match the requested component or requested full-address granularity.
Wrong issued-root selection	0	3	The planner submits another valid root issued earlier in the trajectory.
Spurious parsed root	1	1	The submitted root was issued from a detector/parser candidate that does not denote the target entity.
Unresolved or malformed restore	0	1	Restoration fails or leaves a reference token unresolved.

Appendix

Table 19: Planner-separated automatic outcomes for Full structured mediation.

Policy	Unique reuse	Multi-root selection	Order-dependent groups	Policy violations	Path agreement
Component-projection recent-first	119/119	290/290	16/16	0/443	443/443

Appendix

Table 20: Exhaustive recent-first conformance over observable binder states.

Policy	Unique success	Ambiguous success	Overall exact success	Exact failures
Recent-first	4,236/4,236	1,050/1,764	5,286/6,000 (0.881)	714
Fresh-on-ambiguity	4,236/4,236	0/1,764	4,236/6,000 (0.706)	1,764

Appendix

Table 21: Generator-conditioned exact-continuation success under two binding policies on the same 6,000 gold continuations.

Lang	Method	View	Qwen $\downarrow$	Gemini $\downarrow$
EN	AbI	SAN	0.2775	0.2800
EN	PrA	SAN	0.4758	0.4826
EN	PrG	SAN	0.5754	0.5815
EN	Ours	SAN	0.1289	0.1332
EN	AbI	Full	0.3559	0.3444
EN	Ours	Full	0.1310	0.1353

Appendix

Table 22: Complete CrossTurnPII single-view reader exposure on common gold. SAN rows are placeholder-free erase-only views. Full rows are the two reference-releasing pipelines.

		Qwen		Gemini
Method	View	Exposure $\downarrow$	Utility $\uparrow$	Exposure $\downarrow$	Utility $\uparrow$
Original	Original	0.9802	0.912	0.9811	0.968
AbI	SAN	0.1992	0.914	0.2096	0.990
PrA	SAN	0.3683	0.896	0.3692	0.876
PrG	SAN	0.3579	0.844	0.3560	0.934
Ours	SAN	0.0585	0.916	0.0576	0.994

Appendix

Table 23: Complete PrivScreen privacy exposure and visual-answer utility under the Original, SAN, and Full configurations.

			AbI Full			Ours Full
Lang	Reader	Gold	Per-view	Five-turn	Increment	Per-view	Five-turn	Increment
EN	Qwen	1,394	0.3486	0.3486	0.0000	0.2066	0.2080	0.0014
EN	Gemini	1,394	0.3271	0.3271	0.0000	0.2145	0.2174	0.0029
ZH	Qwen	1,424	0.7022	0.7022	0.0000	0.1208	0.1222	0.0014
ZH	Gemini	1,424	0.7163	0.7163	0.0000	0.1405	0.1426	0.0021

Appendix

Table 24: Per-view and five-turn decomposition of CrossTurnPII audited source-value leakage on common-gold sessions.

Method	Overall	Person	Email	Phone	Location	Account	IP address
Original	.9802/.9811	.9752/.9752	.9867/.9867	.9900/.9900	.9700/.9800	.9742/.9742	1/1
AbI SAN	.1992/.2096	.4134/.4257	.0533/.0533	0/0	.3400/.4000	.0129/.0129	0/0
PrA SAN	.3683/.3692	.7574/.7500	0/0	.1250/.1250	.2300/.2600	.2194/.2194	.0400/.0600
PrG SAN	.3579/.3560	.1955/.1931	.9200/.9133	.0200/.0200	.4600/.4600	.7226/.7226	0/0
Ours SAN	.0585/.0576	.1139/.1114	0/0	0/0	.1100/.1100	0/0	.1000/.1000
AbI Full	.2134/.2229	.4406/.4505	.0533/.0800	0/0	.3800/.4000	.0129/.0129	0/0

Appendix

Table 25: PrivScreen field exposure by PII type. Each cell reports Qwen / Gemini; lower is better. SAN rows are placeholder-free erase-only views; Full rows are the two complete reference-releasing configurations.

Reader	Method	Lang	New/protected	RER [95% CI]	Net $\Delta$ [95% CI]
Qwen	AbI	EN	240/2,018	0.1189 [0.0966,0.1442]	$+0.0784$ [ $+0.0621,+0.0978$ ]
		ZH	20/706	0.0283 [0.0161,0.0411]	$+0.0046$ [ $+0.0014,+0.0078$ ]
	Ours	EN	26/2,433	0.0107 [0.0050,0.0175]	$+0.0022$ [ $-0.0053,+0.0104$ ]
		ZH	107/2,715	0.0394 [0.0274,0.0527]	$+0.0359$ [ $+0.0240,+0.0490$ ]
Gemini	AbI	EN	199/2,011	0.0990 [0.0807,0.1177]	$+0.0645$ [ $+0.0522,+0.0772$ ]
		ZH	16/682	0.0235 [0.0130,0.0334]	$+0.0046$ [ $+0.0014,+0.0075$ ]

Appendix

Table 26: Method-specific CrossTurnPII conditional re-exposure and signed net contrasts.

Reader	Single EN	Single ZH	Five-turn EN	Five-turn ZH
Qwen	23/2,510	43/2,550	42/1,454	43/1,393
Gemini	25/2,510	55/2,521	39/1,454	38/1,365

Appendix

Table 27: Exact PrivacyGuard reference reconstruction by reader and language.

Method	OCR	Det-P (core) $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	$\mathrm{LR}_{\mathrm{detect}}\downarrow$	Mod $\downarrow$
AbI	EasyOCR	0.7165	0.6410	0.6476	0.2018	2.1025
	PP-v5-Mobile	0.6611	0.6743	0.6762	0.1582	2.5224
	PP-v5-Server	0.7083	0.7029	0.7076	0.1636	2.5433
	Tesseract	0.4141	0.3286	0.3581	0.3669	2.8747
PrA	EasyOCR	0.3769	0.3848	0.3876	0.1882	6.3520
	PP-v5-Mobile	0.4820	0.4200	0.4200	0.1779	4.6360

Appendix

Table 28: Controlled OCR-backend results on PrivScreen.

Coverage family	Cases	Representative forms
Canonical roots, projections, and length metadata	17	ADDR#1 , ADDR#1.DIST , NAME#4 , ALNUM#8.LEN=8
Session-history reuse	9	Turn-2 address, phone, number, and name roots/projections
Allowlisted normalized variants	14	Whitespace, full-width, and separator-confusable forms

Appendix

Table 29: Valid reference-resolution cases by coverage family.

Invalid family	Cases	Representative references
Unknown entity id	3	[[ADDR#999.CITY]] , [[PHONE#999]] , [[NAME#777]] .
Foreign-session root absent locally	3	Address, email, and license-plate roots recorded only in another session and absent from the active-session mapping.
Future-turn unavailable	3	Address, ID-number, and alphanumeric roots that are not yet present in the current session state.
Type mismatch	4	[[PHONE#1]] , [[EMAIL#4]] , [[NAME#5]] .
Unsupported scalar projection	7	[[NAME#4.CITY]] , [[ORG#10.CITY]] , [[TIME#11.ROAD]] .
Unsupported address projection	3	[[ADDR#1.DOMAIN]] , [[ADDR#1.LEN=4]] , [[ADDR#1.TIME]] .

Appendix

Table 30: Unsupported, malformed, and confusable reference cases by family.

Evidence family	PrivacyGuard profile	Evidence supplied
Text, address, screenshot metrics	Common RTX 4090 server	Algorithm and end-to-end output under shared inputs.
Controlled REF tests	Common RTX 4090 server	Binding and local-restoration mechanism behavior.
Audited planner-use paths	Server-hosted service + Android UI	Cloud sees mediated views; trusted base includes interface, runner, and server-hosted service.
Android runtime profile	Native OCR + embedded Python mediation in packaged Android host	Handset endpoint feasibility and component cost.
Live Android utility	Server-hosted protection + Android UI/harness	End-to-end planner utility under common protection-pipeline placement.

Appendix

Table 31: Execution profile associated with each evidence family.

State source	Lifetime	Permitted role
Static detection resources	Packaged, read-only	Public priors, formats, contexts, and negative boundaries used only as candidate evidence.
Local privacy repository	Persistent, opt-in	User-linked typed values, aliases, and address/name components used only as detection evidence.
Session mapping store	Session/goal scoped; in-memory by default	Issued roots, typed records, projections, and raw values; the sole authority for local reference resolution and restoration.

Appendix

Table 32: Roles and authority of device-local state.

Evidence source	Available evidence	Role and scope
CrossTurnPII controlled corpus	Exact values and spans, rendered regions, text–image links, five-turn entity identity, and component projections	Supports RQ1 SAN source removal and reader exposure, RQ2 attachment and reconstruction, and RQ3 controlled Full upload-view leakage. It does not provide external screenshot or broad task generality.
Structured-address stress	7,800 forms, component gold, 11 English and 12 Chinese perturbation families, and two-/three-block forms	Supports RQ1 complete-source coverage and RQ2 cross-form continuity and PCR. It is a controlled form-distribution stress test rather than external workload evidence.
PrivScreen [ 33 ]	500 captured screens with visible-PII and geometry labels, benchmark privacy questions, and visual questions	Supports external RQ1 SAN evaluation and RQ3 Full final-view privacy and visual utility. It does not provide cross-turn identity or local-resolution gold.
Fixed live-interface suite	Fixed Android tasks, trajectory-issued references, resolved actions, and automatically scored answers	Supports RQ2 verification of reference-required completion and planner-use paths. It is a targeted capability suite rather than a broad task benchmark.
AndroidLab [ 27 ]	138 public tasks over nine apps, fixed resets and judgments, task and action metrics, and retained trajectories	Its SAN-only-to-Full contrast supports RQ2 mechanism attribution, while complete Full-system comparisons support RQ3 task utility. It does not provide exact source-extent, entity-identity, or projection gold.
Randomized multi-root stress	Seeded compatible roots, generator-provided intended roots, ambiguity strata, and exact continuation outcomes	Supports RQ2 binding-policy conformance and policy–intent agreement. Its outcomes are conditioned on the declared generator.

Appendix

Table 33: Evidence sources, available annotations, and evaluation scope.

Corpus / type	Unique gold	Whole-value exact	Identifier exact	Structural atom median / p95	Public-prior atom median / p95
Prompt EN / all	2,313	0 (0.00%)	0	0.0000 / 0.5000	0.0000 / 0.6000
NAME	321	0 (0.00%)	0	0.0000 / 0.0000	0.5556 / 0.6667
ADDRESS	598	0 (0.00%)	0	0.2222 / 0.6000	0.0000 / 0.5833
Prompt ZH / all	2,186	15 (0.69%)	0	0.0000 / 0.2500	0.0000 / 0.3333
NAME	328	0 (0.00%)	0	0.0000 / 0.0000	0.3333 / 0.3333
ADDRESS	474	15 (3.16%)	0	0.1429 / 0.4286	0.0769 / 0.5000

Appendix

Table 34: Type-aware lexicon–gold content audit.

Reader	Lang	AbI Full ALR	Ours SAN-only-or-Full ALR	$\Delta$ (95% CI)
Qwen	EN	0.3486	0.2202	$-0.1284$ [ $-0.1607,-0.0937$ ]
	ZH	0.7022	0.1264	$-0.5758$ [ $-0.6094,-0.5408$ ]
Gemini	EN	0.3271	0.2260	$-0.1012$ [ $-0.1344,-0.0670$ ]
	ZH	0.7163	0.1503	$-0.5660$ [ $-0.5970,-0.5353$ ]

Appendix

Table 35: Conservative PrivacyGuard-only monotone sensitivity on common-gold sessions.

Setting	Lang	Method	[email protected] [95% CI]	$\mathrm{LR}_{\mathrm{char}}$ [95% CI]
Text	EN	PrG	0.7912 [0.7808, 0.8015]	0.1283 [0.1202, 0.1359]
Text	EN	Ours	0.9740 [0.9619, 0.9846]	0.0125 [0.0075, 0.0186]
Text	ZH	PrA	0.5048 [0.4710, 0.5389]	0.3998 [0.3808, 0.4193]
Text	ZH	Ours	0.9971 [0.9931, 0.9997]	0.0023 [0.0002, 0.0054]
Address	EN	PrG	0.6247 [0.6127, 0.6352]	0.0537 [0.0465, 0.0612]
Address	EN	Ours	0.8981 [0.8691, 0.9238]	0.0430 [0.0318, 0.0559]

Appendix

Table 36: Bootstrap intervals for the main text/address SAN comparison.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

PIIGuard: Mitigating PII Harvesting under Adversarial Sanitization
May 4, 2026Mingshuo Liu, Yiwei Zha, Min ChenData Leakage Cybersecurity
Vrije Universiteit Amsterdam
MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution
May 27, 2026Yanqiu Zhao, Dongying Zheng, Kaibo Huang +3GUI Agents
Beijing University of Posts and Telecommunications
SlotGuard: Stop Oversharing Private Local Context in LLM Agent Transcri
Jul 19, 2026Haocheng Xia, Yongjoo ParkLLM Agent Security Privacy Leakage in Language Models
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign.

May 4, 2026J/K move · Enter open · S save

PIIGuard: Mitigating PII Harvesting under Adversarial Sanitization

Mingshuo Liu, Yiwei Zha, Min Chen

Vrije Universiteit Amsterdam

Data Leakage Cybersecurity Privacy Leakage in Language Models Prompt Injection Defense

Open paper AskPDF arXiv

Browsing-enabled LLM assistants can fetch webpages and answer contact-seeking queries, creating a practical channel for scraping contact-style personally identifiable information (PII) from public pages. Many prior defenses are deployed at the model, service, or agent layer rather than at the webpage itself, leaving ordinary page owners with limited deployable options. We present PIIGuard, a webpage-level defense that repurposes indirect prompt injection as a protective mechanism: the page owner embeds optimized hidden HTML fragments that steer the model away from verbatim or reconstructible disclosure of contact PII. PIIGuard searches over fragment text and insertion position using rule-based leakage scoring, evolutionary mutation, and final judge-based recoverability assessment. In direct-HTML evaluation on three target models (GPT-5.4-nano, Claude-haiku-4.5, and DeepSeek-chat(latest v3.2)), PIIGuard achieves at least 97.0% defense success rate under both rule-based and judge-based leakage evaluation, often reaching 100.0%, while preserving benign same-page QA utility. We further evaluate two harder settings: public-URL browsing and attacker-side LLM sanitization of fetched webpage. These results show that page-side defensive fragments can remain effective in deployment for some model-position pairs, but robustness varies substantially across browsing interfaces and sanitizer prompts. Overall, PIIGuard demonstrates that page owners can use page-side fragments as a practical mitigation for web-grounded PII leakage.

Abstract

GUI Agents

Explore similar work

PIIGuard: Mitigating PII Harvesting under Adversarial Sanitization

MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution

SlotGuard: Stop Oversharing Private Local Context in LLM Agent Transcri