SoK: Privacy Attacks on Machine Learning via Explainable AI

Abdullah Caglar Oksuz; Anisa Halimi; Erman Ayday

cs.CRSep 9, 2026

SoK: Privacy Attacks on Machine Learning via Explainable AI

Authors: Abdullah Caglar Oksuz, Anisa Halimi, Erman Ayday

Organizations: Case Western Reserve University · IBM Research

Abstract

Machine learning explanations reveal model behavior beyond predictions, creating attack surfaces for model confidentiality and data privacy. We systematize 25 studies that exploit explanations for model extraction, membership inference, and model inversion, treating attribute inference as partial inversion. Existing work is often labeled only black- or white-box, obscuring substantial differences in what explanation signal reaches an adversary. We therefore separate model knowledge from explanation acquisition and identify five paths: target-released, attacker-derived, secondary disclosure, privileged access, and released global artifacts. Across these paths, explanations reduce extraction cost, expose membership signals through explanation statistics, recourse distance, and explanation-guided robustness, and support spatial or algebraic reconstruction of private inputs. We compare system and threat models, explanation signals, auxiliary knowledge, target models, modalities, query budgets, evaluation metrics, reported performance, and defenses. Our analysis shows that no explanation family is uniformly unsafe and no defense is uniformly effective. Risk depends on which signal is exposed, how it is acquired, which asset is targeted, and what the attacker already knows. We argue that explanation privacy should therefore be evaluated as an end-to-end disclosure problem, with defenses matched to the acquisition path and protected asset.

Figures & tables

Signal	Typical scope	Model requirement	Security-relevant content	Acquisition path	Representative studies
Input gradients / saliency	Local	Differentiable model for direct computation	Local derivative, feature sensitivity, decision geometry	P1 , P4	Milli et al. [ 23 ] ; Miura et al. [ 24 ] ; Shokri et al. [ 39 ]
IG / Grad-CAM / LRP	Local	Usually a differentiable neural network; method dependent	Spatial or feature relevance; class-specific structure	P1 , P2 , P3	Zhao et al. [ 50 ] ; Liu et al. [ 18 ] ; Yan et al. [ 46 , 48 ]
LIME	Local	Model-agnostic query access	Local feature weights and a local surrogate	P1 , P2	Oksuz et al. [ 31 ] ; Yan et al. [ 47 ]
SHAP values	Local ; Global aggregation	Model-specific or model-agnostic variants	Per-feature marginal contributions; correlations with private inputs	P1 , P3 , P2	Oksuz et al. [ 31 ] ; Luo et al. [ 20 ] ; Toma and Kikuchi [ 42 ] ; Duddu and Boutet [ 7 ]
CF / recourse	Local	Prediction plus counterfactual-generation service	Boundary-adjacent instance, direction and magnitude of change	P1	Aïvodji et al. [ 2 ] ; Wang et al. [ 45 ] ; Pawelczyk et al. [ 34 ] ; Khouna et al. [ 15 ]
Influence	Local ; training record	Usually model and training-process access	Contribution of training records to loss or prediction	P4	Cohen et al. [ 6 ]

Table 1: Explanation signals used in the core corpus. P1 – P5 denote the acquisition paths defined in Section 4 .

Study	Explanation (path)	Mechanism	Modality	Target models	Evaluation	Representative finding
Milli et al. [ 23 ]	Gradients or saliency; P1	Parameter recovery; gradient matching	Image	Linear; two-layer ReLU; convolutional network; VGG11; ResNet-18	Accuracy; queries	MNIST convolutional network: 95% accuracy with 10 gradient queries versus 1,000 label queries; theory gives $O(h\log h)$ gradient queries.
Aïvodji et al. [ 2 ]	Prediction + CF; P1	Surrogate training on factual and counterfactual pairs	Tabular	Logistic regression; multilayer perceptron	Fidelity; accuracy; queries	Adult: about 93% fidelity with 1,000 queries under marginal-distribution knowledge; CFs remain useful with unknown distribution.
Kuppa and Le-Khac [ 17 ]	Prediction + CF; P1	CF-assisted coverage and distillation	Tabular (security)	Gradient-boosted trees; neural networks	Surrogate accuracy; queries	CF-assisted extraction outperforms prediction-only and distillation baselines in the evaluated security setting.
Wang et al. [ 45 ]	Prediction + CF + second CF; P1	Two-sided boundary sampling	Tabular	Hidden-parameter classifiers	Fidelity; queries	Two-sided CF information improves extraction efficiency over one-sided CF extraction.
Yan et al. [ 46 ]	Prediction + four feature-importance methods; P1	Joint prediction and explanation training	Image	DNNs	Accuracy; explanation MSE	Prediction+explanation gains reach 1.25, 1.53, 9.25, and 7.45 points on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.
Yan et al. [ 47 ]	Prediction + Grad-CAM or LIME; P1	Data-free generation with explanation loss	Image	ResNet-34; ResNet-18	Surrogate accuracy	Outperforms DFME and DaST on CIFAR-10 and SVHN; simple explanation perturbation changes attack accuracy by at most 0.65 percentage points.

Table 2: Model extraction attacks using explanations. Results follow each study’s threat model and are not a single leaderboard.

Study	Explanation (path)	Membership signal	Modality	Access	Evaluation	Representative finding
Shokri et al. [ 39 ]	Gradients; Integrated Gradients; LRP; SmoothGrad; P1	Explanation variance or full vector	Tabular ; Image	Prediction + local explanation	Accuracy	Gradient explanations leak membership; full-vector attacks are slightly stronger than variance-only attacks, with leakage shaped by overfitting and dimensionality.
Kuppa and Le-Khac [ 17 ]	CF; P1	Proximity to factual and CF neighborhoods	Tabular (security)	Prediction + CF API	Accuracy; queries	Demonstrates CF-assisted membership inference; its evaluation predates current low-FPR MIA practice.
Naretto et al. [ 27 ]	Global surrogate; P5	Surrogate behavior	Tabular	Released global explainer	Accuracy; AUC	Global surrogate explanations can leak more membership information than the original classifier; leakage tracks fidelity and overfitting.
Pawelczyk et al. [ 34 ]	CF or recourse; P1	CF distance; likelihood ratio	Tabular	One recourse query per record	AUC; balanced accuracy; low-FPR TPR	HELOC+CCHVAE: AUC 0.679 and TPR 5.13% at 1% FPR; several Adult and Diabetes settings remain near random.
Cohen et al. [ 6 ]	Self-influence; P4	Self-influence score	Image	Model and training access	Accuracy; AUC	Demonstrates membership leakage from self-influence under a substantially stronger, non-public access model.
Liu et al. [ 18 ]	Seven feature-importance methods; P1 / P2	Explanation-guided robustness trajectory	Image	Prediction + explanation; shadow variants	AUC; balanced accuracy; TPR at 0.1% FPR	Representative AUC 0.915–0.931 and TPR 2.2–3.7% at 0.1% FPR; ablations show both XAI and the multi-query procedure contribute.

Table 3: Membership inference attacks using explanations. Low-FPR results are highlighted when available.

Study	Explanation (path)	Mechanism	Private target	Target models	Evaluation	Representative finding
Zhao et al. [ 50 ]	Gradient, CAM, and LRP; P1 / P2 / P3	Spatial explanation as inversion input	Image records	DNN image classifiers	MSE; SSIM; PSNR; attack accuracy	Spatial explanations improve over prediction-only inversion; richer class-specific CAM sets increase reconstruction risk.
Duddu and Boutet [ 7 ]	Feature-importance explanations; P1	Classifier maps explanation to hidden attribute	Tabular records	Four classifier families	Attribute-inference accuracy	Explanations outperform prediction-only inference; leakage remains when the sensitive feature is censored via proxy correlations.
Luo et al. [ 20 ]	Shapley report; P1 (calibration), P3 (target)	Learned inverse from local linearity	Tabular query records	Neural network; support vector machine; random forest; gradient-boosted tree	MAE; success rate	Demonstrated on Google, Microsoft, and IBM services; important features and neural networks are generally more vulnerable.
Toma and Kikuchi [ 42 ]	Shapley values; P3	Inverse optimization; exact linear case	Tabular query records	Linear and nonlinear models	Error; exact recovery	Exact reconstruction for linear regression with exact Shapley values; empirical risk varies by model and optimizer.

Table 4: Comparison of explanation-assisted model inversion studies, including attribute inference as partial inversion. We distinguish reconstruction of training records from reconstruction of query records.

Defense	Layer	Primary asset	Evidence	Not automatically covered	Main observation
DP-SGD [ 1 ]	Training	Training membership and records	Formal and empirical [ 33 , 18 , 34 ]	Query-record privacy; model confidentiality	When explanations are post-processing of a DP model, the training-data guarantee is preserved; practical utility and explanation-quality loss can be substantial.
Regularization and dropout [ 40 ]	Training	Empirical membership leakage	Indirect or general evidence	Formal privacy; query inversion; model extraction	Can reduce overfitting-driven leakage but provides no formal privacy guarantee.
DP local explanations [ 33 ]	Explainer	Reference or training data	Formal and empirical analysis	Model confidentiality; unprotected query records	Separates DP of model training from privacy of data used to construct local explanations.
DP counterfactuals [ 8 ]	Explainer	Model confidentiality	Model extraction: evaluated	Attacker-derived explanations; other XAI families	Reduces CF-assisted extraction toward no-CF baselines while retaining some recourse utility.
Noise on feature explanations [ 3 ]	Explanation	Query-record attributes	Inversion; attribute inference: evaluated	Raw gradients; attacker-derived channels	Evaluates attack success, explanation faithfulness, and model utility jointly.
Clipping, masking, and noise [ 12 ]	Explanation	Membership leakage	Membership inference: evaluated	Extraction; inversion unless evaluated	Reduces leakage by 46.5–95% with at most 3.3% average utility loss in DeepLeak.

Table 5: Defenses relevant to explanation-assisted attacks. Evidence is classified as direct empirical evaluation, a formal privacy argument, or an unvalidated proposal for the corresponding attack family.

Model Extraction Attacks

Model extraction attacks infer a machine-learning model’s behavior, architecture, parameters, or hidden assets from queries and system leakage.

See all 47 papers

Jul 3, 2026

TIER: Trajectory-Invariant Explanation Regularization for Membership Privacy

Explainability is central to building trustworthy AI, yet explanation interfaces can inadvertently provide adversaries with an expanded privacy-related attack surfaces. Recent studies show that advanced membership-inference attacks succeed by exploiting confidence-drop trajectories, induced through attribution-guided perturbations, as discriminative features, rather than directly using confidence scores or explanation vectors. Existing defenses against membership inference fail to directly mitigate such explanation-driven attacks. In this work, we investigate whether, during training, a model's own gradients can be leveraged as defense signals against such attacks, thereby aligning explanation profiles between members and non-members. To this end, we propose a Trajectory-Invariant Explanation Regularization (TIER) defense that penalizes erratic fluctuations in confidence drops simulated through gradient-guided perturbations and simultaneously minimizes the distributional shifts via KL-divergence. Unlike conventional adversarial training, which emphasizes label robustness, our approach targets explanation robustness by enforcing self-consistency through KL-divergence and reducing the variance of confidence drops between members and non-members. Extensive experiments confirm that our method effectively mitigates these attacks, delivering privacy protection while maintaining model utility and explanation fidelity.

Varun Sharma, Kar Wai Fok, Vrizlynn L. L. Thing

ST Engineering, Singapore

Membership Inference Attacks Explanation Stability Privacy-Preserving ML

Ask PDF

May 28, 2026

Can Subgraph Explanations Be Weaponized to Steal Graph Neural Networks?

Graph Machine Learning as a Service (GMLaaS) platforms increasingly implement explainability interfaces to meet regulatory transparency requirements. However, this transparency creates exploitable vulnerabilities for model extraction attacks. We present the first model extraction attack specifically designed for graph classification under strict black-box constraints where the attacker observes only discrete class labels and binary explanation masks (no probability scores, gradients, or confidence values). Our method (1) uses model explanation outputs to guide Monte Carlo edge sensitivity estimation toward decision boundaries, with Hoeffding concentration guarantees on estimation accuracy and (2) exploits explanation subgraphs to efficiently narrow the boundary search space. Extensive experiments on benchmark graph datasets across multiple domains demonstrate our method's superiority over comparable baselines. These findings demonstrate that such explainability interfaces create exploitable attack surfaces, informing both defensive mechanisms and policy frameworks for explainable AI mandates. The implementation code is provided in https://github.com/LabRAI/XSTEAL/.

Ojas Nimase, Jiate Li, Yue Zhao +1

University of Southern California Los Angeles, USA · Florida State University Tallahassee, USA

Graph Neural Networks Model Extraction Attacks Graph Classification GNN Explainability

Ask PDF

Mar 24, 2026

A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks

Membership inference attacks (MIAs) aim to determine whether a data sample was included in a machine learning (ML) model's training set and have become the de facto standard for measuring privacy leakages in ML. We propose an evaluation framework that defines the conditions under which MIAs constitute a genuine privacy threat, and review representative MIAs against it. We find that, under the realistic conditions defined in our framework, MIAs represent weak privacy threats. Thus, relying on them as a privacy metric in ML can lead to an overestimation of risk and to unnecessary sacrifices in model utility as a consequence of employing too strong defenses.

Najeeb Jebreel, David Sánchez, Josep Domingo-Ferrer

Universitat Rovira i Virgili, Department of Computer Engineering and Mathematics, CYBERCAT-Center for Cybersecurity Research of Catalonia, ComSCIAM-Center for Computational Science and Applied Mathematics Av. Països Catalans 26, 43007 Tarragona, Catalonia

Membership Inference Attacks Privacy-Preserving ML

Ask PDF

Abstract

Model Extraction Attacks

Explore similar work

TIER: Trajectory-Invariant Explanation Regularization for Membership Privacy

Can Subgraph Explanations Be Weaponized to Steal Graph Neural Networks?

A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks