cs.CRNov 19, 2026

Erasing the Wrong Thing: Conditional Attribute Leakage and Concept Erasure in Shared Foundation-Model Embeddings

Authors: Haijun He, Fei Li, Donghong Ji, Junjun He

Organizations: School of Cyber Science and Engineering, Wuhan University · Shanghai Innovation Institute

Abstract

Medical-image embeddings are increasingly shared in place of raw images for authorized downstream tasks, yet they may reveal sensitive attributes. Standard concept-erasure methods target population-level guardedness. However, we prove that, when binary task and sensitive labels are associated, global and task-conditional linear guardedness cannot both hold while a nonzero linear task signal is preserved, so global erasure must either eliminate that signal or retain task-conditional linear leakage. We further show how global erasure affects linear readers' discrimination between positive and negative samples with different sensitive attributes, deriving an approximate closed-form prediction of its utility cost from pre-erasure statistics. Motivated by this conflict, we introduce Task-Conditional LEACE (TC-LEACE), a closed-form variant of LEACE that enforces linear guardedness within each task stratum of its fitting data. We evaluate four vision foundation models on 85 CelebA attribute pairs spanning a broad range of associations. After LEACE, conditional leakage persists across all four encoders, the discrepancy between marginal and conditional audits increases with association strength, and predicted utility costs track observed losses. We then examine hospital chest radiographs using three foundation models, including two biomedical models, in which chest-tube status is strongly associated with diagnosis. Although a marginal attack on LEACE-erased embeddings underperforms a diagnosis-only reference, chest-tube status remains detectable within diagnosis strata. TC-LEACE recovers most of LEACE's linear task-utility loss while keeping task-conditional leakage within a prespecified margin of LEACE's.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Principled Nonlinear Concept Erasure and Steering

    Oct 23, 2026Idan Simai, Shauli Ravfogel, Uri ShahamConcept ErasurePrivacy-Preserving Representation Learning

  2. Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure

    Mar 8, 2026Ramin Akbari, Milad Afshari, Vishnu Naresh BoddetiConcept ErasureAdversarial Robustness

  3. Privacy-Preserving Deep Joint Source-Channel Coding with In-Loop Concept Erasure

    Sep 11, 2026Rami Eid, Maria Slim, Mariette Awad +1Deep Joint Source-Channel CodingPrivacy-Preserving ML