cs.CVAug 26, 2026

Defending Wearable VLMs Against Private Attribute Inference

Authors: Zhimin Li, Pan Wang, Jingxian Chen, Yuantao Tang, Anthony Chen, Qian Lou, Jingtong Hu

Organizations: Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · College of Engineering, Michigan State University · Department of Computer Science, University of Central Florida

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Wearable VLM pipelines promise continuous multimodal assistance from egocentric visual capture: a user asks a task-driven question about the surrounding scene, and the system uses compact visual tokens to support language reasoning. The challenge motivating this work is that the same egocentric evidence needed for useful assistance can also reveal private attributes about the wearer or nearby bystanders. We investigate this as a joint privacy-utility problem for split VLM inference, where visual encoding occurs within a trusted device boundary but intermediate visual tokens may be transmitted to downstream reasoning components. This exposes an understudied leakage surface: even when final textual responses are benign, external attackers or untrusted downstream components can recover private attributes from transmitted visual tokens. To evaluate this tension, we construct a paired privacy-utility benchmark with 3,221 image-question records, each paired with a utility question and privacy labels covering location, income, sex, and interests. We further propose Token-Guided Attribute Privacy (TGAP), a pre-LLM token disentangler that learns a residual transformation of visual tokens before they leave the trusted boundary. TGAP combines utility preservation, identity regularization, semantic privacy suppression, and image-driven representation suppression, avoiding the utility loss caused by coarse hard or attention masking. On the benchmark used for source-model evaluation, TGAP reduces privacy accuracy from 56.7% to 7.4%, a 49.3% absolute drop, while maintaining relaxed utility at 74.4%. These results suggest that securing the compact token interface is a practical path toward privacy-preserving wearable multimodal AI.

Figures & tables

Figure 1: Motivation. In a split VLM, the tokens used to answer a utility question (the traffic light is red) can also encode a location cue (Stonebrook Pkwy, Frisco). TGAP applies a residual transformation before these tokens leave the trusted boundary. The example illustrates the intended behavior: retain the utility answer while reducing evidence for private-attribute recovery.
Work lineEgo. VLMPaired P–UToken leak.VLM atk.Token mit.Low-cost mit.
General VQA benchmarks [ 12 , 13 ]×\times×\times×\times×\times×\times×\times
Private-attribute inference [ 14 ]∘\circ×\times×\times✓\checkmark×\times×\times
Model/feature inversion [ 9 , 15 ]×\times×\times∘\circ×\times×\times×\times
Split inference systems [ 16 , 17 ]∘\circ×\times∘\circ×\times×\times∘\circ
Backbone fine-tuning/unlearning [ 4 , 5 , 18 ]∘\circ×\times×\times×\times×\times∘\circ
Image-space privacy filters [ 6 , 19 , 20 ]∘\circ✓\checkmark×\times∘\circ×\times✓\checkmark
Table 1: Coverage map of related work. ✓\checkmark denotes direct coverage, ∘\circ partial or indirect coverage, and ×\times no primary coverage. Column abbreviations: “Ego. VLM” is egocentric/wearable VLM inference; “Paired P–U” is paired privacy-utility evaluation; “Token leak.” is leakage from intermediate visual tokens; “VLM atk.” is a VLM/LLM-based attacker; “Token mit.” is mitigation at the token interface; “Low-cost mit.” indicates whether the mitigation avoids repeated backbone-level retraining.
Figure 2: Overview. (1) A paired single-image benchmark supplies a utility question and evidence-gated private-attribute labels. (2) Inside the trusted boundary, the encoder and resampler produce visual tokens and TGAP transforms them before transmission. (3) Source-model prompting measures semantic leakage and utility. (4) Fresh attacker bridges and heads, trained directly on the transmitted representation, measure attribute recoverability with frozen surrogate LLMs.
Figure 3: TGAP disentangler architecture. Post-resampler tokens z=R(E(x))z=R(E(x)) pass through layer normalization, a bottleneck MLP Δθ\Delta_{\theta} , token-wise gate mθ(z)m_{\theta}(z) , scalar strength gg , and residual addition to produce filtered tokens z~=Tθ(z)\tilde{z}=T_{\theta}(z) before offloading to the frozen LLM backbone.
 Util.Loc.Inc.SexInt.Priv. Σ\SigmaAll Σ\Sigma
Entries3,2211,8483034022,6705,2238,444
Conf. 1–20002–
Conf. 2–16318213966550–
Conf. 3–13680101351668–
Conf. 4–654271081,4902,279–
Conf. 5–89314547631,724–
Table 2: Utility-task and privacy-annotation composition of the full benchmark. Confidence is the 1–5 annotation support score; utility tasks do not have privacy confidence.
1:procedure TGAP-Optimize (D,E,R,M,Cω)(\mathcal{D},E,R,M,C_{\omega})
2:Freeze E,R,ME,R,M , pre-train/freeze CωC_{\omega} , init TθT_{\theta}
3:for e=(x,qu,y,{(qjp,aj,cj)}j=1m)∼De=(x,q^{u},y,\{(q^{p}_{j},a_{j},c_{j})\}_{j=1}^{m})\sim\mathcal{D} do
4:z←R(E(x))z\leftarrow R(E(x))
5:z~←z+g mθ(z)⊙Δθ(LN(z))\tilde{z}\leftarrow z+g\,m_{\theta}(z)\odot\Delta_{\theta}(\mathrm{LN}(z))
6:ϕ←m−1∑jϕsem(M(z~,qjp),aj)\phi\leftarrow m^{-1}\sum_{j}\phi_{\mathrm{sem}}(M(\tilde{z},q^{p}_{j}),a_{j})
7:if CωC_{\omega} is active then
Table 6
TypeModelP.Acc ↓\downarrowU.Rel ↑\uparrow
Closed-sourceGPT-4o58.2282.99
GPT-4o-mini49.8174.20
GPT-4.1-mini57.7884.23
Claude Haiku 4.545.8077.24
Gemini 2.5 Flash70.4284.68
Gemini 2.5 Flash-Lite53.0585.00
Table 3: Representative closed- and open-source VLMs on the paired benchmark. Values are percentages.
ModelMethodP.Acc ↓\downarrowP.Cont ↓\downarrowP.Rel ↓\downarrowP.WR ↓\downarrowU.EM ↑\uparrowU.Rel ↑\uparrowTradeoff ↑\uparrow
MiniCPM-VBaseline0.567 ±\pm 0.0030.199 ±\pm 0.0200.206 ±\pm 0.0190.275 ±\pm 0.0110.652 ±\pm 0.0130.766 ±\pm 0.0110.613 ±\pm 0.009
Hard mask0.498 ±\pm 0.0120.174 ±\pm 0.0170.180 ±\pm 0.0170.229 ±\pm 0.0130.568 ±\pm 0.0160.692 ±\pm 0.0120.572 ±\pm 0.009
Attn. mask0.563 ±\pm 0.0070.198 ±\pm 0.0200.204 ±\pm 0.0190.271 ±\pm 0.0120.634 ±\pm 0.0040.749 ±\pm 0.0060.600 ±\pm 0.011
TGAP (ours)0.074 ±\pm 0.0640.037 ±\pm 0.0290.037 ±\pm 0.0290.039 ±\pm 0.0450.633 ±\pm 0.0210.744 ±\pm 0.0210.716 ±\pm 0.009
Qwen2.5-VLBaseline0.476 ±\pm 0.0100.365 ±\pm 0.0040.366 ±\pm 0.0040.177 ±\pm 0.0040.496 ±\pm 0.0100.594 ±\pm 0.0110.377 ±\pm 0.010
Hard mask0.409 ±\pm 0.0150.287 ±\pm 0.0090.288 ±\pm 0.0090.156 ±\pm 0.0030.415 ±\pm 0.0020.515 ±\pm 0.0130.367 ±\pm 0.014
Table 4: Source-model privacy and utility on the 20% validation split. Values are mean ±\pm standard deviation over three seeds. The Qwen/GLM baseline and attention-mask rows use the corrected instrumented backend described in Appendix D.5 . MiniCPM-V uses the complete multi-constraint objective; the other source models use fixed architecture-specific configurations across seeds. Lower privacy is better; higher utility is better.
VariantP.Acc ↓\downarrowP.Cont ↓\downarrowU.Rel ↑\uparrowTradeoff ↑\uparrow
Full objective0.074 ±\pm 0.0640.037 ±\pm 0.0290.744 ±\pm 0.0210.716 ±\pm 0.009
w/o image0.124 ±\pm 0.0340.047 ±\pm 0.0130.741 ±\pm 0.0220.706 ±\pm 0.025
w/o semantic0.553 ±\pm 0.0100.179 ±\pm 0.0060.767 ±\pm 0.0120.630 ±\pm 0.014
w/o identity0.093 ±\pm 0.0250.044 ±\pm 0.0090.724 ±\pm 0.0280.692 ±\pm 0.020
Table 5: Objective ablation under the same MiniCPM-V setting as Table 4 . Entries are mean ±\pm standard deviation over three seeds; Tradeoff follows (1−P.Cont)×U.Rel(1-\mathrm{P.Cont})\times\mathrm{U.Rel} .
Figure 4: External LLM attacker benchmark on validation leaked tokens. Each panel compares raw post-resampler tokens with TGAP-filtered tokens for the same attacker. Lower balanced accuracy, macro-F1, and Cohen’s κ\kappa indicate weaker private-attribute recovery.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

SourceRecordsUtility tasksPrivacy annotationsLocationIncomeSex / GenderInterests
Reddit2,2842,2844,2721,3592863802,247
ST-VQA36436436415211210
InfographicVQA363363377239720111
TextVQA2072072079791100
DocVQA3331002
Total3,2213,2215,2231,8483034022,670
Appendix
Table 6: Dataset sources and annotation coverage. Reddit records are user-posted images from public subreddits; the other rows are open VQA-style datasets converted into the same schema.
Figure 5: Representative paired image-question examples in our benchmark. Each example contains an image, an intended utility question-answer pair, and a privacy question-answer pair for one private attribute. These examples are shown only to illustrate the benchmark schema; the raw Reddit-derived dataset is not planned for unrestricted public release.
AttributeAnnotationsRe-auditedErrorsError rateUnanimous agreement
Location1,84870030.43%97%
Income30330310.33%98%
Sex/gender40220000.00%99%
Interests2,6701,000101.00%88%
Total5,2232,203140.64%96% macro avg.
Appendix
Table 7: Human review and annotation audit. Error rates use the re-audited count as the denominator.
Evidence typeAnnotated / re-auditedErrorsError rate
Explicit numeric self-report286 / 28600.00%
Explicit categorical self-report17 / 1715.88%
Total303 / 30310.33%
Appendix
Table 8: Income-annotation audit by admissible evidence type.
SplitRecordsRatioUse
Full benchmark3,221–Dataset statistics and closed-source benchmarking
TGAP-eligible subset3,084100%Utility task and at least one non-uncertain privacy target
TGAP train2,46780%Adapter optimization and image-probe fitting
TGAP validation61720%Model selection and reported source-model metrics
Appendix
Table 9: Split protocol used by TGAP source-model and token-bank experiments.
Task / attributeParsed fieldNormalization rule
Utilityassistant answerstrip prefixes, punctuation, and compare with references
Locationfirst Guess entrycity/country containment and semantic location match
Incomefirst Guess entrymap to canonical income class
Sexfirst Guess entrymap to canonical sex class
Interestsfirst Guess entrykeyword match, then semantic equivalence if needed
Appendix
Table 10: Output parsing and normalization rules used by the benchmark evaluator.
Attacker familyToken inputTrainable modulesEpochsBatchLR
Qwen2.5-VL-3Braw or TGAP tokensbridge + head445×10−55{\times}10^{-5}
Phi-3.5-Visionraw or TGAP tokensbridge + head445×10−55{\times}10^{-5}
InternVL2.5-2Braw or TGAP tokensbridge + head445×10−55{\times}10^{-5}
GLM-4.1V-9Braw or TGAP tokensbridge + head445×10−55{\times}10^{-5}
Appendix
Table 11: External-attacker setup. The LLM backbone is frozen in every row; the bridge and attribute head are trainable.
AttackerAttributeRaw BAFiltered BADrop (95% CI)
QwenSex0.602±0.0550.602\pm 0.0550.546±0.0800.546\pm 0.0800.056  [−0.029,0.146]0.056\;[-0.029,0.146]
QwenIncome0.518±0.0450.518\pm 0.0450.442±0.0950.442\pm 0.0950.076  [−0.072,0.189]0.076\;[-0.072,0.189]
QwenLocation0.326±0.1790.326\pm 0.1790.170±0.0030.170\pm 0.0030.156  [0.013,0.346]0.156\;[0.013,0.346]
PhiSex0.672±0.0490.672\pm 0.0490.713±0.0340.713\pm 0.034−0.042  [−0.101,0.015]-0.042\;[-0.101,0.015]
PhiIncome0.574±0.1460.574\pm 0.1460.417±0.0870.417\pm 0.0870.157  [−0.051,0.361]0.157\;[-0.051,0.361]
PhiLocation0.507±0.2000.507\pm 0.2000.186±0.0340.186\pm 0.0340.320  [0.120,0.551]0.320\;[0.120,0.551]
Appendix
Table 12: Adaptive-readout balanced accuracy over three attacker-training seeds.
1:procedure Assemble-TGAP-Step (z,Tθ,Cω,A)(z,T_{\theta},C_{\omega},\mathcal{A})▹\triangleright Called inside Algorithm 3.3
2:z~←Tθ(z)\tilde{z}\leftarrow T_{\theta}(z)▹\triangleright Residual post-resampler map
3:Lid←∥z~−z∥22\mathcal{L}_{\mathrm{id}}\leftarrow\|\tilde{z}-z\|_{2}^{2}▹\triangleright Identity regularizer
4:if CωC_{\omega} is active and labels A\mathcal{A} are available then▹\triangleright Image-driven branch
5:p^det←Cω(sg(z~))\hat{p}_{\mathrm{det}}\leftarrow C_{\omega}(\mathrm{sg}(\tilde{z}))▹\triangleright Stop gradient to TθT_{\theta}
6:Lprobe←CE(p^det,A)\mathcal{L}_{\mathrm{probe}}\leftarrow\mathrm{CE}(\hat{p}_{\mathrm{det}},\mathcal{A})▹\triangleright Train private-attribute decoder
7:ω←AdamW(ω,∇ωLprobe)\omega\leftarrow\mathrm{AdamW}(\omega,\nabla_{\omega}\mathcal{L}_{\mathrm{probe}})▹\triangleright Probe-only update
Appendix
Figure 10: Implementation subroutine for the detached private-attribute decoder update and signed TGAP loss construction. The routine complements Algorithm 3.3 ; sg(⋅)\mathrm{sg}(\cdot) denotes stop-gradient.
ModelSettingP.Acc ↓\downarrowP.Cont ↓\downarrowU.EM ↑\uparrowU.Rel ↑\uparrow
Qwen2.5-VL-3BBaseline0.476±0.0100.476\pm 0.0100.365±0.0040.365\pm 0.0040.496±0.0100.496\pm 0.0100.594±0.0110.594\pm 0.011
Qwen2.5-VL-3BAttention mask0.411±0.0310.411\pm 0.0310.299±0.0180.299\pm 0.0180.426±0.0100.426\pm 0.0100.532±0.0160.532\pm 0.016
GLM-4.1V-9B-BaseBaseline0.434±0.0220.434\pm 0.0220.087±0.0020.087\pm 0.0020.000±0.0000.000\pm 0.0000.473±0.0120.473\pm 0.012
GLM-4.1V-9B-BaseAttention mask0.324±0.0130.324\pm 0.0130.075±0.0030.075\pm 0.0030.000±0.0000.000\pm 0.0000.371±0.0160.371\pm 0.016
Appendix
Table 13: Corrected attention-mask audit on the complete held-out split. Values are mean ±\pm standard deviation over three seeds.
SettingValue
Source VLMMiniCPM-V open-source VLM
Train/validation split2,467 / 617 per seed (3,084 eligible records)
Visual token shapeMiniCPM-V: 64×409664\times 4096 post-resampler tokens
Adapter bottleneck dimension64
Adapter optimizerAdamW, learning rate 2×10−42{\times}10^{-4} , weight decay 10−410^{-4}
Training epochs2 adapter epochs + 1 image-probe pretraining epoch
Appendix
Table 14: TGAP training hyperparameters for the MiniCPM-V results in Tables 4 and 5 .
Variantλutil\lambda_{\mathrm{util}}λpriv\lambda_{\mathrm{priv}}λimg,ce\lambda_{\mathrm{img,ce}}λimg,ent\lambda_{\mathrm{img,ent}}λimg,conf\lambda_{\mathrm{img,conf}}λid\lambda_{\mathrm{id}}gmax⁡g_{\max}
Full objective1.00.060.0150.010.010.120.16
w/o image-driven objective1.00.060.0000.000.000.120.16
w/o source-model privacy objective1.00.000.0150.010.010.120.16
w/o identity regularization1.00.060.0150.010.010.000.16
Appendix
Table 15: Objective-ablation grid used in Table 5 . The full row is the MiniCPM-V TGAP configuration in Table 4 ; a zero weight removes the named term. λimg,unif=0\lambda_{\mathrm{img,unif}}=0 throughout.
ComponentResource / sizeNotes
TGAP adapter536,641 parametersMiniCPM-V LayerNorm + 4096–64–4096 bottleneck + scalar gate
Image-driven probe2,374,155 parametersUsed for training constraint; not required at deployment
Leaked-token shapeMiniCPM-V: 64×409664\times 4096Same interface for raw and TGAP-filtered tokens
Full TGAP runtime213.9–301.5 minutesThree runs, including evaluation
Attacker training4 epochs, batch size 4Frozen LLM + trainable bridge/head
Closed-sourceAPI inferenceNo local gradient training
Appendix
Table 16: Compute and footprint summary for the reproducibility-critical runs.
EvidenceSource protocolRepeatsRole in the paper
Table 4Native source-VLM configurationsThree seedsSource-model privacy–utility comparison
Table 5Matched MiniCPM-V full objectiveThree seedsIsolates the contribution of each objective term
External attackers and frozen probesSeparate MiniCPM-V complete-objective checkpoint; matched raw/filtered splits within studyThree seedsTests empirical recoverability from transmitted tokens; no checkpoint-level comparison with Table 4
Table 21Fixed MiniCPM-V checkpoint without per-dataset retuningFixed checkpointDelimits transfer to image/OCR and video tasks; not a repeated-run efficacy estimate
Table 13Native Qwen/GLM source settingsThree seedsVerifies that the model-specific attention intervention is active
Tables 22 – 24Different model, token, or pixel interfacesControl-specificProvides qualitative interface comparisons only; no direct TGAP ranking
Appendix
Table 17: Claim–evidence map. “Matched” means that the experiment reuses the corresponding Table 4 checkpoint and seed-specific split without retuning the residual gate.
ModelP.Acc ↓\downarrowLoc. ↓\downarrowInc. ↓\downarrowSex ↓\downarrowInt. ↓\downarrowU.Rel ↑\uparrowU.LLM ↑\uparrow
GPT-4o58.2264.6653.1478.5451.1082.9989.38
GPT-4o-mini49.8143.7942.3876.1350.8874.2080.78
GPT-4.1-mini57.7863.1031.0277.5654.1684.2390.69
Claude Haiku 4.545.8043.8834.4469.1144.9577.2483.11
Gemini 2.5 Flash70.4264.8414.2983.6558.5484.6890.31
Gemini 2.5 Flash-Lite53.0563.0218.5166.4149.0985.0088.48
Appendix
Table 18: Full closed-source VLM benchmark on the paired dataset. Values are percentages. Lower privacy metrics are better ( ↓\downarrow ); higher utility metrics are better ( ↑\uparrow ).
AttributeBaselineHard token maskAttention maskTGAP Disentangler
location0.196 ±\pm 0.0400.139 ±\pm 0.0340.196 ±\pm 0.0400.009 ±\pm 0.010
income0.765 ±\pm 0.1360.713 ±\pm 0.0750.760 ±\pm 0.1320.067 ±\pm 0.061
sex0.965 ±\pm 0.0100.961 ±\pm 0.0160.965 ±\pm 0.0100.339 ±\pm 0.313
interests0.021 ±\pm 0.0080.016 ±\pm 0.0070.020 ±\pm 0.0070.001 ±\pm 0.001
Appendix
Table 19: Repeated-run attribute-level source-model privacy containment. Values are mean ±\pm standard deviation over repeated runs. Lower is better.
DecoderRaw BA ↑\uparrowFilt. BA ↓\downarrowBA Drop ↑\uparrowRaw F1 ↑\uparrowFilt. F1 ↓\downarrowF1 Drop ↑\uparrowRaw κ\kappa ↑\uparrowFilt. κ\kappa ↓\downarrowκ\kappa Drop ↑\uparrow
Linear probe0.6780.4260.2520.6830.4090.2740.5840.1910.393
MLP probe0.6500.4080.2420.6340.3950.2390.4980.2120.286
Appendix
Table 20: Frozen private-attribute decoder transfer evaluation. A decoder is trained on raw visual tokens and evaluated on raw versus TGAP-filtered validation tokens. Lower filtered balanced accuracy, macro-F1, and Cohen’s κ\kappa are better; larger drops indicate weaker direct transfer from raw tokens to filtered tokens.
Figure 12: Attribute-level frozen private-attribute decoder transfer. Arrows move from raw-token balanced accuracy to TGAP-filtered balanced accuracy. Lower filtered balanced accuracy indicates weaker direct transfer of image-driven private-attribute decoders.
DatasetU.EMU.RelOfficial metric
TextVQA0.772 / 0.7590.808 / 0.810Acc.: 0.724 / 0.714
DocVQA0.392 / 0.3910.454 / 0.450ANLS: 0.553 / 0.549
MSVD-QA0.441 / 0.3730.566 / 0.521–
ActivityNet-QA0.641 / 0.5720.654 / 0.587–
Appendix
Table 21: External-task utility, reported as baseline / TGAP. Higher is better.
Model / settingP.Acc ↓\downarrowP.Rel ↓\downarrowU.EM ↑\uparrowU.Rel ↑\uparrow
CLEAR base0.51450.26490.50730.6288
CLEAR-LLMU0.11750.12430.44730.5673
CLEAR-DPO0.36420.18020.47650.6078
Appendix
Table 22: Released CLEAR checkpoints on our held-out questions.
KKP.Cont ↓\downarrowP.Rel ↓\downarrowP.WR ↓\downarrowU.EM ↑\uparrowU.Rel ↑\uparrow
00.20810.21480.30110.67100.7828
40.19270.19850.27490.64340.7601
80.18020.18590.25510.59640.7196
160.18210.18790.24350.53810.6677
Appendix
Table 23: Gaussian replacement on the complete held-out split. We report only metrics stored by this control’s evaluator.
VLMPrivacy leak ↓\downarrowUtility ↑\uparrow
InternVL3.5-2B (training surrogate)0.1184 / 0.10610.7349 / 0.7189
Qwen2.5-VL-3B0.1170 / 0.11560.7269 / 0.7289
Phi-3.5-Vision0.0327 / 0.02990.5562 / 0.5562
InternVL2.5-2B0.0367 / 0.02990.6888 / 0.6627
GLM-4.1V-9B0.0680 / 0.05710.7028 / 0.7108
Appendix
Table 24: Learned image-space perturbation. Each cell is raw / perturbed under the image-space evaluator.
More in this topic

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

Explore similar work

CardsList

  1. Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models

    Sep 14, 2026Md Khalid Syfullah, Alvi Ataur KhalilVisual Question AnsweringPrivacy Protection in VLMs

    Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Computer Science, Southern Illinois University Carbondale, USA

  2. SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

    Oct 8, 2026Zhimin Li, Pan Wang, Jingxian Chen +3Efficient VLM InferenceVLM Inference

    Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida

  3. How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study

    May 6, 2026Junran Wang, Xinjie Shen, Zehao Jin +1VLM EvaluationVision-Language Models

    Georgia Institute of Technology