Privacy-Preserving ML

ML: Machine Learning

Ask about this topic

Overview

The field studies how to train, evaluate, and deploy machine-learning systems while reducing disclosure of personal data, model information, or sensitive patterns. Main approaches include differential privacy during optimization, federated learning with secure aggregation, homomorphic encryption and secure multiparty computation, as well as synthetic data, representation obfuscation, and machine unlearning. Methods are tested across clinical, language, vision, and graph tasks, using measures such as prediction quality, reconstruction and membership-inference attacks, communication, and runtime. A central tension is that federated learning or transformed data do not guarantee privacy on their own, while stronger formal protections can reduce utility or add substantial computational and communication costs.

Latest papers

CardsList

433 papers37 papers in the last four weeks, down 3% on the four weeks before. 0.2% of all new papers.

Feb 23, 2026

Safe Multi-Agent Deep Reinforcement Learning for Privacy-Aware Edge-Device Collaborative DNN Inference

As Deep Neural Network (DNN) inference becomes increasingly prevalent on edge and mobile platforms, critical challenges emerge in privacy protection, resource constraints, and dynamic model deployment. This paper proposes a privacy-aware collaborative inference framework, in which adaptive model partitioning is performed across edge devices and servers. To jointly optimize inference delay, energy consumption, and privacy cost under dynamic service demands and resource constraints, we formulate the joint problem as a Constrained Markov Decision Process (CMDP) that integrates model deployment, user-server association, model partitioning, and resource allocation. We propose a Hierarchical Constrained Multi-Agent Proximal Policy Optimization with Lagrangian relaxation (HC-MAPPO-L) algorithm, a safe reinforcement learning-based framework that enhances Multi-Agent Proximal Policy Optimization (MAPPO) with adaptive Lagrangian dual updates to enforce long-term delay constraints. To ensure tractability while maintaining coordination, we decompose the CMDP into three hierarchically structured policy layers: an auto-regressive based model deployment policy, a Lagrangian-enhanced user association and model partitioning policy, and an attention-based resource allocation policy. Extensive experimental results demonstrate that HC-MAPPO-L consistently satisfies stringent delay constraints while achieving a superior balance among energy consumption and privacy cost, outperforming representative baseline algorithms across varying problem scales and resource configurations.

Hong Wang, Xuwei Fan, Zhipeng Cheng +4

School of Future Science and Engineering, Soochow University, Suzhou 215006, China · College of Computer and Information Sciences, Fujian Agriculture and Forestry University, Fuzhou 350002, China · School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China +2

Constrained MARLEfficient Neural Network InferenceMulti-Agent Reinforcement LearningPrivacy-Preserving ML
AskPDF
Feb 23, 2026

Evaluating the Impact of Data Anonymization on Image Retrieval

With the growing importance of privacy regulations such as the General Data Protection Regulation, anonymizing visual data is becoming increasingly relevant across institutions. However, anonymization can negatively affect the performance of Computer Vision systems that rely on visual features, such as Content-Based Image Retrieval (CBIR). Despite this, the impact of anonymization on CBIR has not been systematically studied. This work addresses this gap, motivated by the DOKIQ project, an artificial intelligence-based system for document verification actively used by the State Criminal Police Office Baden-Württemberg. We propose a simple evaluation framework: retrieval results after anonymization should match those obtained before anonymization as closely as possible. To this end, we systematically assess the impact of anonymization using two public datasets and the internal DOKIQ dataset. Our experiments span three anonymization methods, four anonymization degrees, and four training strategies, all based on the state of the art backbone Self-Distillation with No Labels (DINO)v2. Our results reveal a pronounced retrieval bias in favor of models trained on original data, which produce the most similar retrievals after anonymization. The findings of this paper offer practical insights for developing privacy-compliant CBIR systems while preserving performance.

Marvin Chen, Manuel Eberhardinger, Johannes Maucher

Institute of Applied AI, Stuttgart Media University, Nobelstr. 10, 70569 Stuttgart, Germany

Image RetrievalPrivacy-Preserving MLRetrieval RobustnessIR Evaluation
AskPDF
Feb 22, 2026

HybridFL: A Federated Learning Approach for Financial Crime Detection

Federated learning (FL) is a privacy-preserving machine learning paradigm that enables multiple parties to collaboratively train models on privately owned data without sharing raw information. While standard FL typically addresses either horizontal or vertical data partitions, many real-world scenarios exhibit a complex hybrid distribution. This paper proposes Hybrid Federated Learning (HybridFL) to address data split both horizontally across disjoint users and vertically across complementary feature sets. We evaluate HybridFL in a financial crime detection context, where a transaction party holds transaction-level attributes and multiple banks maintain private account-level features. By integrating horizontal aggregation and vertical feature fusion, the proposed architecture enables joint learning while strictly preserving data locality. Experiments on AMLSim and SWIFT datasets demonstrate that HybridFL significantly outperforms the transaction-only local model and achieves performance comparable to a centralized benchmark.

Afsana Khan, Marijn ten Thij, Guangzhi Tang +1

Department of Advanced Computing Sciences, Maastricht University, Netherlands · Department of Cognitive Science and Artificial Intelligence, Tilburg University, Netherlands

Anti-Money LaunderingFederated LearningPrivacy-Preserving MLFinancial Fraud Detection
AskPDF
Feb 21, 2026

SLDP: Semi-Local Differential Privacy for Density-Adaptive Analytics

Density-adaptive domain discretization is essential for high-utility privacy-preserving analytics but remains challenging under Local Differential Privacy (LDP) due to the privacy-budget costs associated with iterative refinement. We propose a novel framework, Semi-Local Differential Privacy (SLDP), that assigns a privacy region to each user based on local density and defines adjacency by the potential movement of a point within its privacy region. We present an interactive (ε,δ)(\varepsilon, δ)-SLDP protocol, orchestrated by an honest-but-curious server over a public channel, to estimate these regions privately. Crucially, our framework decouples the privacy cost from the number of refinement iterations, allowing for high-resolution grids without additional privacy budget cost. We experimentally demonstrate the framework's effectiveness on estimation tasks across synthetic and real-world datasets.

Alexey Kroshnin, Alexandra Suvorikova

Weierstrass Institute for Applied Analysis and Stochastic

Differential PrivacyPrivacy-Preserving ML
AskPDF
Feb 21, 2026

PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems

Privacy preserving machine learning deployments in sensitive deep learning applications; from medical imaging to autonomous systems; increasingly require combining multiple techniques. Yet, practitioners lack systematic guidance to assess the synergistic and non-additive interactions of these hybrid configurations, relying instead on isolated technique analysis that misses critical system level interactions. We introduce PrivacyBench, a benchmarking framework that reveals striking failures in privacy technique combinations with severe deployment implications. Through systematic evaluation across ResNet18 and ViT models on medical datasets, we uncover that FL + DP combinations exhibit severe convergence failure, with accuracy dropping from 98% to 13% while compute costs and energy consumption substantially increase. In contrast, FL + SMPC maintains near-baseline performance with modest overhead. Our framework provides the first systematic platform for evaluating privacy-utility-cost trade-offs through automated YAML configuration, resource monitoring, and reproducible experimental protocols. PrivacyBench enables practitioners to identify problematic technique interactions before deployment, moving privacy-preserving computer vision from ad-hoc evaluation toward principled systems design. These findings demonstrate that privacy techniques cannot be composed arbitrarily and provide critical guidance for robust deployment in resource-constrained environments.

Nnaemeka Obiefuna, Samuel Oyeneye, Similoluwa Odunaiya +2

1ML Collective · 2Friedrich-Alexander-Universität Erlangen-Nürnberg · 3Carnegie Mellon University

Privacy-Preserving MLDifferential PrivacyFederated LearningPrivacy-Utility Tradeoff
AskPDF
Feb 20, 2026

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.

Fangyuan Xu, Sihao Chen, Zinan Lin +13

New York University · Microsoft · University of Southern California

Differentially Private Synthetic Data GenerationSynthetic Data Generation for Language ModelsPrivacy-Preserving Language ModelsPrivacy-Preserving MLDifferential PrivacyRL for Generative Models
AskPDF
Feb 19, 2026

JAX-Privacy: A library for differentially private machine learning

JAX-Privacy is a library designed to simplify the deployment of robust and performant mechanisms for differentially private machine learning. Guided by design principles of usability, flexibility, and efficiency, JAX-Privacy serves both researchers requiring deep customization and practitioners who want a more out-of-the-box experience. The library provides verified, modular primitives for critical components for all aspects of the mechanism design including batch selection, gradient clipping, noise addition, accounting, and auditing, and brings together a large body of recent research on differentially private ML.

Ryan McKenna, Galen Andrew, Borja Balle +6

Privacy-Preserving MLDifferential PrivacyDifferentially Private Stochastic Gradient Descent
AskPDF
Feb 19, 2026

Guarding the Middle: Protecting Intermediate Representations in Federated Split Learning

Big data scenarios, where massive, heterogeneous datasets are distributed across clients, demand scalable, privacy-preserving learning methods. Federated learning (FL) enables decentralized training of machine learning (ML) models across clients without data centralization. Decentralized training, however, introduces a computational burden on client devices. U-shaped federated split learning (UFSL) offloads a fraction of the client computation to the server while keeping both data and labels on the clients' side. However, the intermediate representations (i.e., smashed data) shared by clients with the server are prone to exposing clients' private data. To reduce exposure of client data through intermediate data representations, this work proposes k-anonymous differentially private UFSL (KD-UFSL), which leverages privacy-enhancing techniques such as microaggregation and differential privacy to minimize data leakage from the smashed data transferred to the server. We first demonstrate that an adversary can access private client data from intermediate representations via a data-reconstruction attack, and then present a privacy-enhancing solution, KD-UFSL, to mitigate this risk. Our experiments indicate that, alongside increasing the mean squared error between the actual and reconstructed images by up to 50% in some cases, KD-UFSL also decreases the structural similarity between them by up to 40% on four benchmarking datasets. More importantly, KD-UFSL improves privacy while preserving the utility of the global model. This highlights its suitability for large-scale big data applications where privacy and utility must be balanced.

Obaidullah Zaland, Sajib Mistry, Monowar Bhuyan

Department of Computing Science, Umeå Unviersity, Umeå, SE-90187, Sweden · Curtin University, Bentley WA 6102, Australia

Training Data Reconstruction AttacksPrivacy-Preserving MLSplit Federated LearningDifferentially Private Federated Learning
AskPDF
Feb 19, 2026

Jolt Atlas: Verifiable Inference via Lookup Arguments in Zero Knowledge

We present Jolt Atlas, a zero-knowledge machine learning (zkML) framework that extends the Jolt proving system to model inference. Unlike zkVMs (zero-knowledge virtual machines), which emulate CPU instruction execution, Jolt Atlas adapts Jolt's lookup-centric approach and applies it directly to ONNX tensor operations. The ONNX computational model eliminates the need for CPU registers and simplifies memory consistency verification. In addition, ONNX is an open-source, portable format, which makes it easy to share and deploy models across different frameworks, hardware platforms, and runtime environments without requiring framework-specific conversions. Our lookup arguments, which use sumcheck protocol, are well-suited for non-linear functions -- key building blocks in modern ML. We apply optimisations such as neural teleportation to reduce the size of lookup tables while preserving model accuracy, as well as several tensor-level verification optimisations detailed in this paper. We demonstrate that Jolt Atlas can prove model inference in memory-constrained environments -- a prover property commonly referred to as streaming. Furthermore, we discuss how Jolt Atlas achieves zero-knowledge through the BlindFold technique, as introduced in Vega. In contrast to existing zkML frameworks, we show practical proving times for classification, embedding, automated reasoning, and small language models. Jolt Atlas enables cryptographic verification that can be run on-device, without specialised hardware. The resulting proofs are succinctly verifiable. This makes Jolt Atlas well-suited for privacy-centric and adversarial environments. In a companion work, we outline various use cases of Jolt Atlas, including how it serves as guardrails in agentic commerce and for trustless AI context (often referred to as AI memory).

Wyatt Benno, Alberto Centelles, Antoine Douchet +1

ICME Labs

Zero-Knowledge MLPrivacy-Preserving MLMemory-Efficient Inference
AskPDF
Feb 19, 2026

MeGU: Machine-Guided Unlearning with Target Feature Disentanglement

The growing concern over training data privacy has elevated the "Right to be Forgotten" into a critical requirement, thereby raising the demand for effective Machine Unlearning. However, existing unlearning approaches commonly suffer from a fundamental trade-off: aggressively erasing the influence of target data often degrades model utility on retained data, while conservative strategies leave residual target information intact. In this work, the intrinsic representation properties learned during model pretraining are analyzed. It is demonstrated that semantic class concepts are entangled at the feature-pattern level, sharing associated features while preserving concept-specific discriminative components. This entanglement fundamentally limits the effectiveness of existing unlearning paradigms. Motivated by this insight, we propose Machine-Guided Unlearning (MeGU), a novel framework that guides unlearning through concept-aware re-alignment. Specifically, Multi-modal Large Language Models (MLLMs) are leveraged to explicitly determine re-alignment directions for target samples by assigning semantically meaningful perturbing labels. To improve efficiency, inter-class conceptual similarities estimated by the MLLM are encoded into a lightweight transition matrix. Furthermore, MeGU introduces a positive-negative feature noise pair to explicitly disentangle target concept influence. During finetuning, the negative noise suppresses target-specific feature patterns, while the positive noise reinforces remaining associated features and aligns them with perturbing concepts. This coordinated design enables selective disruption of target-specific representations while preserving shared semantic structures. As a result, MeGU enables controlled and selective forgetting, effectively mitigating both under-unlearning and over-unlearning.

Haoyu Wang, Zhuo Huang, Xiaolong Wang +3

School of Automation, Beijing Institute of Technology · Sydney AI Centre, The University of Sydney · Department of Computer Science, Hong Kong Baptist University

Machine UnlearningPrivacy-Preserving ML
AskPDF
Feb 18, 2026

Protecting the Undeleted in Machine Unlearning

Machine unlearning aims to remove specific data points from a trained model, often striving to emulate "perfect retraining", i.e., producing the model that would have been obtained had the deleted data never been included. We demonstrate that this approach, and security definitions that enable it, carry significant privacy risks for the remaining (undeleted) data points. We present a reconstruction attack showing that for certain tasks, which can be computed securely without deletions, a mechanism adhering to perfect retraining allows an adversary controlling merely ω(1)ω(1) data points to reconstruct almost the entire dataset merely by issuing deletion requests. We survey existing definitions for machine unlearning, showing they are either susceptible to such attacks or too restrictive to support basic functionalities like exact summation. To address this problem, we propose a new security definition that specifically safeguards undeleted data against leakage caused by the deletion of other points. We show that our definition permits several essential functionalities, such as bulletin boards, summations, and statistical learning.

Aloni Cohen, Refael Kohen, Kobbi Nissim +1

University of Chicago · Tel Aviv University · Dept. of Computer Science, Georgetown University

Training Data Reconstruction AttacksMachine UnlearningPrivacy-Preserving MLData Leakage
AskPDF
Feb 18, 2026

Utility-Preserving De-Identification for Math Tutoring: Investigating Numeric Ambiguity in the MathEd-PII Benchmark Dataset

Large-scale sharing of dialogue data is key to advancing the science of teaching and learning, yet rigorous de-identification remains a major barrier. In mathematics tutoring transcripts, numeric expressions frequently resemble structured identifiers (e.g., dates or IDs), leading generic Personally Identifiable Information (PII) detection systems to over-redact core instructional content and reduce data utility. This work asks how to detect PII while preserving educational utility, focusing on this "numeric ambiguity" problem. We introduce MathEd-PII, the first benchmark dataset for PII detection in math tutoring dialogues, built with human-in-the-loop LLM annotation. Using density-based segmentation, we show that false PII redactions cluster in math-dense regions, confirming numeric ambiguity as a key failure mode. We then compare four detection strategies: a Presidio baseline and three LLM-based approaches with basic, math-aware, and segment-aware prompting. Domain-aware prompting, including both math-aware (F1: 0.802) and segment-aware versions (F1: 0.821), substantially outperforms the baseline (F1: 0.379) while reducing numeric false positives, demonstrating that de-identification must incorporate domain context to preserve analytic utility. This work provides a new benchmark and evidence that utility-preserving de-identification for tutoring data requires domain-aware modeling.

Zhuqian Zhou, Kirk Vanacore, Bakhtawar Ahtisham +7

Cornell University · Fresh Cognate · UPchieve +1

PII DetectionPrivacy-Preserving ML
AskPDF
Feb 17, 2026

Hybrid Federated and Split Learning for Privacy Preserving Clinical Prediction and Treatment Optimization

Collaborative clinical decision support is often constrained by governance and privacy rules that prevent pooling patient-level records across institutions. We present a hybrid privacy-preserving framework that combines Federated Learning (FL) and Split Learning (SL) to support decision-oriented healthcare modeling without raw-data sharing. The approach keeps feature-extraction trunks on clients while hosting prediction heads on a coordinating server, enabling shared representation learning and exposing an explicit collaboration boundary where privacy controls can be applied. Rather than assuming distributed training is inherently private, we audit leakage empirically using membership inference on cut-layer representations and study lightweight defenses based on activation clipping and additive Gaussian noise. We evaluate across three public clinical datasets under non-IID client partitions using a unified pipeline and assess performance jointly along four deployment-relevant axes: factual predictive utility, uplift-based ranking under capacity constraints, audited privacy leakage, and communication overhead. Results show that hybrid FL-SL variants achieve competitive predictive performance and decision-facing prioritization behavior relative to standalone FL or SL, while providing a tunable privacy-utility trade-off that can reduce audited leakage without requiring raw-data sharing. Overall, the work positions hybrid FL-SL as a practical design space for privacy-preserving healthcare decision support where utility, leakage risk, and deployment cost must be balanced explicitly.

Farzana Akter, Rakib Hossain, Deb Kanna Roy Toushi +3

Dept. of Information Technology, Washington University of Science and Technology, Alexandria, VA, USA · Cognitive Links LLC, Maryland, USA

Uplift ModelingPrivacy-Preserving MLSplit Federated LearningClinical Outcome Prediction
AskPDF
Feb 16, 2026

LRD-MPC: Efficient MPC Inference through Low-rank Decomposition

Secure Multi-party Computation (MPC) enables untrusted parties to jointly compute a function without revealing their inputs. Its application to machine learning (ML) has gained significant attention, particularly for secure inference services deployed across multiple cloud virtual machines (VMs), where each VM acts as an MPC party. Model providers secret-share model weights, and users secret-share inputs, ensuring that each server operates only on random shares. While MPC provides strong cryptographic guarantees, it incurs substantial computational and communication overhead. Deep neural networks rely heavily on convolutional and fully connected layers, which require costly matrix multiplications in MPC. To reduce this cost, we propose leveraging low-rank decomposition (LRD) for linear layers, replacing one large matrix multiplication with two smaller ones. Each matrix multiplication in MPC incurs a round of communication, meaning decomposing one matrix multiplication into two leads to an additional communication round. Second, the added matrix multiplication requires an additional truncation step to maintain numerical precision. Since truncation itself requires communication and computation, these overheads can offset the gains from decomposition. To address this, we introduce two complementary optimizations: truncation skipping and efficient linear layer concatenation. Truncation skipping removes the extra truncation induced by LRD, while linear layer concatenation pipelines operations to hide the additional communication round. Together, these techniques mitigate the main overheads of LRD in MPC and improve overall efficiency. Our approach is broadly applicable across MPC protocols. Experiments show up to 25% speedup in n-PC and 33% in 3-PC protocols over full-rank baselines, along with up to 52% GPU energy savings and 88% reduction in offline-phase latency.

Tingting Tang, Yongqin Wang, Murali Annavaram

University of Southern California, Los Angeles, California, USA

Privacy-Preserving MLEfficient Neural Network InferenceSecure Multi-Party ComputationLow-Rank Matrix Decomposition
AskPDF
Feb 9, 2026

ERIS: Enhancing the Privacy-Utility-Scalability Trade-Off in Federated Learning via Federated Shard Aggregation

Scaling Federated Learning (FL) to billion-parameter models forces a challenging trade-off between privacy, scalability, and model utility. Existing solutions often tackle these challenges in isolation, sacrificing accuracy, relying on costly cryptographic tools, or introducing communication and optimization inefficiencies that affect convergence. We introduce ERIS, an FL framework centered on Federated Shard Aggregation (FSA), a novel mechanism that partitions each client update into non-overlapping shards whose aggregation is distributed across multiple client-side aggregators. FSA removes the central aggregation bottleneck, limits the information visible to any single observer, and preserves the centralized FL update after reassembly. ERIS can further readily integrate Distributed Shifted Compression (DSC) to reduce transmitted payloads and exposed coordinates. We prove that ERIS preserves convergence under standard assumptions and bounds mutual information leakage by the observable fraction of each update, decreasing with the number of client-side aggregators, and with the compression level when DSC is enabled. Experiments across image and text tasks, including large language models, show that ERIS achieves FedAvg-level utility while substantially reducing communication bottlenecks and improving robustness to membership inference and reconstruction attacks, without relying on heavy cryptography or utility-degrading perturbations.

Dario Fenoglio, Pasquale Polverino, Akash Dhasade +3

Universit`a della Svizzera italiana Lugano, Switzerland · Carnegie Mellon University Pittsburgh, PA, United States

Federated LearningFederated Learning AggregationPrivacy-Preserving Federated LearningPrivacy-Preserving MLCommunication-Efficient Distributed Training
AskPDF
Feb 6, 2026

Taipan: A Query-free Transfer-based Multiple Sensitive Attribute Inference Attack Solely from Auxiliary Graphs

Graph-structured data underpin a wide spectrum of modern applications, yet their multiple sensitive attributes are not isolated but deeply coupled with graph topology. This coupling facilitates intersectional privacy leakage through attribute inference attacks (AIAs). Existing AIAs predominantly assume adversaries with two capabilities: (I) access to either auxiliary data sampled from the same distribution as the target or partial training records, and (II) the ability to repeatedly query the victim model. In reality, however, these assumptions are often impractical due to stringent data protection regulations, prohibitive query budgets, and heightened detection risks. Moreover, as prior attacks are designed to infer a single sensitive attribute at a time, they fail to simultaneously recover multiple attributes and thus underestimate overall privacy risks arising from cross-attribute dependencies. More critically, they obscure a systemic blind spot: auxiliary graphs alone are sufficient to expose multiple sensitive attributes in the target graph. This threat persists even under distribution shifts, enabling stealthy offline attacks without any model interactions. To investigate this unexplored vulnerability, we introduce Taipan, the first query-free framework for multiple sensitive attribute inference attacks on graphs (G-MSAIAs). Taipan incorporates Hierarchical Attack Knowledge Routing to capture intricate inter-attribute correlations, and Prompt-guided Attack Prototype Refinement to mitigate distribution shifts between auxiliary and target graphs. Extensive experiments on diverse real-world graph datasets demonstrate that Taipan consistently achieves strong attack performance across same-, similar-, and out-of-distribution settings, and remains effective under partial label coverage or even rigorous privacy guarantees.

Ying Song, Balaji Palanisamy

University of Pittsburgh Pittsburgh, PA, USA

Privacy-Preserving ML
AskPDF
Feb 2, 2026

FaceLinkGen: A Re-evaluation of Identity Leakage in Privacy-Preserving Face Recognition and Face Anonymization Systems Using Simple Distillation

Privacy-preserving face recognition (PPFR) and face anonymization have different goals, but both must retain some identity-related information for their intended use. We show that an adaptive attacker can learn this information. We propose FaceLinkGen, a simple distillation-based attack that trains a face recognition model to map protected inputs back to standard face embeddings. FaceLinkGen applies to keyless PPFR systems and perception-preserving face de-identification (De-ID) systems. For PPFR, the recovered embeddings can be used to regenerate faces that match the original person. Across MinusFace, PartialFace, and DecoyFace, the regenerated faces achieve acceptance rates of 81.0--99.4% on Face++ and 74.9--99.6% on Amazon. For De-ID, FaceLinkGen links protected faces to unprotected images of the same person, reaching Recall@1 values of 48.4--89.6% in the one-side-protected setting across the evaluated methods. FaceLinkGen exposes identity leakage across all evaluated methods, including DecoyFace and WDP, whose protection resists the tested U-Net attacks on face recovery and protected-to-unprotected linkage, respectively. The attack also remains effective when trained with limited paired data. Code is available at https://github.com/weathon/FaceLinkGenRelease.

Wenqi Guo, Qingyun Qian, Mohamed Shehata +1

Department of Computer Science, Mathematics, Physics and Statistics University of British Columbia, Kelowna, BC, Canada

Model Inversion AttacksPrivacy-Preserving MLFace Recognition
AskPDF
Feb 2, 2026

On the Inherent Privacy Amplification of Missing Data

Privacy preservation is critical in many high-stakes domains such as medicine and finance, where sensitive data must be analyzed without compromising individual confidentiality. At the same time, these applications often involve datasets with inherent missing values due to non-response or data corruption for example. Missing data is traditionally analyzed through its impact on statistical efficiency and model performance. In fact, it reduces the information available to analysts and can degrade the final utility of the model. In this work, we take an alternative approach and study missing data through the lens of privacy preservation. Intuitively, when features are missing, less information is revealed about individuals, suggesting that data missingness could inherently enhance privacy. We formalize this intuition within a novel framework that integrates missing data into differential privacy. In essence, our approach accounts for the de facto missingness in the data to refine existing privacy guarantees without modifying the underlying mechanism. Using this framework, we show for the first time that missing data can induce inherent privacy amplification for differentially private algorithms, highlighting a previously overlooked interaction between missing data and formal privacy guarantees..

Simon Roburin, Rafa{ë}l Pinot, Erwan Scornet

LPSM · Sorbonne Université, Université Paris Cité, CNRS Laboratoire de Probabilités, Statistique et Modélisation (LPSM) F-75005 Paris, France

Privacy-Preserving MLDifferential Privacy
AskPDF
Jan 30, 2026

Rethinking Anonymity Claims in Synthetic Data Generation: A Model-Centric Privacy Attack Perspective

Training generative machine learning models to produce synthetic tabular data has become a popular approach for enhancing privacy in data sharing. As this typically involves processing sensitive personal information, releasing either the trained model or generated synthetic datasets can still pose privacy risks. Yet, recent research, commercial deployments, and privacy regulations like the General Data Protection Regulation (GDPR) largely assess anonymity at the level of an individual dataset. In this paper, we rethink anonymity claims about synthetic data from a model-centric perspective, arguing that meaningful assessments must account for the underlying generative model and be grounded in state-of-the-art privacy attacks. This perspective better reflects real-world deployments, where trained models are often accessible for interaction or querying. We interpret the GDPR's definitions of personal data and anonymization under such access assumptions to identify the identifiability risks that must be mitigated and map them to privacy attacks across threat settings. We then argue that synthetic data techniques alone do not ensure sufficient anonymization. Finally, we compare the two mechanisms most commonly used with synthetic data -- Differential Privacy (DP) and Similarity-based Privacy Metrics (SBPMs) -- and argue that while DP can offer robust protections against identifiability risks, SBPMs lack adequate safeguards. Overall, our work connects regulatory notions of identifiability with model-centric privacy attacks, enabling more responsible and trustworthy assessment of synthetic data systems by researchers, practitioners, and policymakers.

Georgi Ganev, Emiliano De Cristofaro

UCL · SAS · UC Riverside

Privacy AuditingSynthetic Tabular Data GenerationPrivacy-Preserving ML
AskPDF
Dec 21, 2025

FedVideoMAE: Efficient Federated Video Moderation with Differential Privacy and Secure Aggregation

Short-form video moderation is increasingly pushed toward edge and privacy-sensitive settings, where users may intend videos for a limited audience, such as friends or private groups, but sending raw clips to a central server can broaden exposure, consume bandwidth, and add moderation latency. Federated learning can keep videos on device, but unprotected model updates may still leak information, and full-video backbones are expensive to communicate. We present FedVideoMAE, a privacy-preserving federated framework for violence detection that adapts a frozen VideoMAE backbone with lightweight LoRA and prompt parameters. Each training round combines self-supervised masked video reconstruction with client-side differential privacy and pairwise masked aggregation (SA) of adapter updates. Violence labels are held out from federation and used only for downstream evaluation, separating private representation learning from supervised assessment. On RWF-2000, exchanging 5,518,848 trainable parameters instead of the 156,371,328-parameter instantiated pretraining state gives a 28.3x model-state payload ratio. FedVideoMAE reaches 77.25% test accuracy without DP or SA, while accuracy under DP+SA remains in the 65.25-66.00% range. Transfer experiments on RLVS and binary UCF-Crime show similar behavior. These results characterize the privacy-utility trade-off for edge video moderation. Code is available at: https://github.com/zyt-599/FedVideoMAE

Ziyuan Tao, Chuanzhi Xu, Sandaru Jayawardana +4

School of Computing, Macquarie University, Sydney, Australia · School of Computer Science, The University of Sydney, Sydney, Australia · School of Electrical and Computer Engineering, The University of Sydney, Sydney, Australia

Federated Learning AggregationVideo Content ModerationVideo Representation LearningPrivacy-Preserving MLSecure AggregationFederated Learning
AskPDF
Oct 6, 2025

Federated Computation of ROC and PR Curves

Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves are fundamental analytical tools for evaluating classification models, providing a comprehensive view of performance across decision thresholds. In modern data management settings, such evaluation must often be performed over data that is distributed across multiple parties and subject to strict privacy constraints. This setting arises naturally in federated learning (FL) and collaborative data platforms, where raw prediction scores and labels cannot be centrally collected. We study the problem of computing ROC and PR curves over distributed data under formal privacy guarantees. The key challenge is that exact computation requires access to all prediction scores and labels, leading to linear communication costs and privacy risks. To address this, we propose a novel framework for approximating ROC and PR curves using quantiles of the score distribution, which can be computed efficiently under secure aggregation and distributed differential privacy. Our approach requires only O(Q)O(Q) communication, where QQ is the number of quantiles, and avoids sharing raw data entirely. We provide theoretical guarantees on the approximation quality by bounding the Area Error (AE) between the true and estimated curves, demonstrating principled trade-offs among approximation accuracy, privacy protection, and communication cost. Our method is robust to data heterogeneity and skew, making it suitable for real-world distributed data management scenarios. Experiments on real-world datasets show that our approach achieves high accuracy with low communication overhead under strong privacy constraints. Overall, this work provides a practical and theoretically grounded solution for privacy-preserving evaluation of classification models over distributed data.

Xuefeng Xu, Graham Cormode

University of Warwick

Privacy-Preserving MLSecure AggregationFederated Learning
AskPDF
Sep 26, 2025

Trajectory-Aware Information Matching for Multi-Step Gradient Inversion in Federated Learning

Federated learning enables distributed information sharing and collaborative model training without exposing raw client data. However, shared gradients or model updates may still contain sensitive information, making federated learning vulnerable to gradient inversion attacks. Most existing gradient inversion attacks rely on simplified update observations, such as single-step gradients or endpoint-based matching. In practical FL, however, FedAvg produces an accumulated trajectory-dependent update after multiple local steps, rather than a gradient computed at a single model state.To address this issue, we propose NL-SME, a trajectory-aware information matching method for multi-step gradient inversion. NL-SME constructs a learnable nonlinear surrogate trajectory to approximate hidden local states and integrates trajectory-level information with calibrated gradient matching. For perturbed updates, NL-SME can further use an observed-update reliability-aware strategy to reduce the influence of unreliable components. Extensive experiments under diverse multi-step FedAvg settings show that NL-SME outperforms state-of-the-art gradient inversion baselines in reconstruction quality and update-matching accuracy. Additional evaluations on natural and medical images, as well as under fused-update observations and representative defense strategies, further suggest that observable multi-step updates may still retain reconstruction signals. These results reveal potential privacy leakage risks in federated information sharing. Code is available at https://anonymous.4open.science/r/NL-SME-main/README.md.

Li Xia, Jing Yu, Zheng Liu +3

Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing, 100081, China

Gradient Inversion AttacksAdversarial AttacksPrivacy-Preserving MLFederated Learning
AskPDF
Sep 14, 2025

Membership Inference Attacks on Recommender System: A Survey

Recommender systems (RecSys) have been widely applied to various applications, including E-commerce, finance, healthcare, social media and have become increasingly influential in shaping user behavior and decision-making, highlighting their growing impact in various domains. However, recent studies have shown that RecSys are vulnerable to membership inference attacks (MIAs), which aim to infer whether user interaction record was used to train a target model or not. MIAs on RecSys models can directly lead to a privacy breach. For example, via identifying the fact that a purchase record that has been used to train a RecSys associated with a specific user, an attacker can infer that user's special quirks. In recent years, MIAs have been shown to be effective on other ML tasks, e.g., classification models and natural language processing. However, traditional MIAs are ill-suited for RecSys due to the unseen posterior probability. Although MIAs on RecSys form a newly emerging and rapidly growing research area, there has been no systematic survey on this topic yet. In this article, we conduct the first comprehensive survey on RecSys MIAs. This survey offers a comprehensive review of the latest advancements in RecSys MIAs, exploring the design principles, challenges, attack and defense associated with this emerging field. We provide a unified taxonomy that categorizes different RecSys MIAs based on their characterizations and discuss their pros and cons. Based on the limitations and gaps identified in this survey, we point out several promising future research directions to inspire the researchers who wish to follow this area. This survey not only serves as a reference for the research community but also provides a clear description for researchers outside this research domain.

Jiajie He, Xintong Chen, Xinyang Fang +4

University of Maryland, Baltimore County, USA · Mcmaster University, Canada · University of Southern California, USA +1

Membership Inference AttacksPrivacy-Preserving MLRecommender Systems
AskPDF
Sep 13, 2025

Privacy-Preserving Generation of Clinical Narratives from Medical Terminologies

In high-stakes domains such as healthcare, privacy concerns severely limit the use of real-world training data. Differentially private (DP) synthetic data offers a promising alternative with formal privacy guarantees, but achieving strong utility remains challenging for clinical note generation due to domain specificity and long-form text complexity. We present Term2Note, a method for synthesising full-length clinical notes under DP constraints. By structurally separating content and form, Term2Note generates section-wise note content conditioned on medical terms, with terms and notes privatised under separate DP constraints, and applies a DP quality maximiser to improve outputs. Experiments demonstrate that Term2Note produces synthetic notes with statistical properties closely aligned with real clinical notes, and that downstream models trained on these notes achieve performance comparable to those trained on real clinical data. Compared to existing DP text generation baselines, Term2Note substantially improves both fidelity and utility, without relying on label distribution assumptions, highlighting its effectiveness as a practical privacy-preserving alternative to real clinical notes.

Yuping Wu, Viktor Schlegel, Warren Del-Pinto +11

Univeristy of Manchester · Imperial College London, Imperial Global Singapore · Khoo Teck Puat Hospital, Singapore +2

Medical Report GenerationHealthcareDifferential PrivacyConditional Generative ModelingSynthetic Data GenerationPrivacy-Preserving ML+2
AskPDF
Sep 1, 2025

Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption

In contemporary cloud-based services, protecting users' sensitive data and ensuring the confidentiality of the server's model are critical. Fully homomorphic encryption (FHE) enables inference directly on encrypted inputs, but its practicality is hindered by expensive bootstrapping and inefficient approximations of non-linear activations. We introduce Safhire, a hybrid inference framework that executes linear layers under encryption on the server while offloading non- linearities to the client in plaintext. This design eliminates bootstrapping, supports exact activations, and significantly reduces computation cost. To safeguard model confidentiality despite client access to intermediate outputs, Safhire applies randomized shuffling, which obfuscates intermediate values and makes it practically impossible for a client to reconstruct the model. To further reduce inference latency, Safhire incorporates advanced optimizations such as fast ciphertext packing and partial extraction. Evaluations on multiple standard models and datasets show that Safhire achieves up to 7.6x and 7.4x lower inference latency than Orion and Concrete-ML, respectively, which are two state-of-the-art FHE baselines, with manageable communication overhead and comparable accuracy. These results underline the practicality of our hybrid FHE inference scheme.

Sayan Biswas, Philippe Chartier, Akash Dhasade +7

EPFL · Inria · IRMAR +3

Privacy-Preserving MLHomomorphic EncryptionEfficient Neural Network Inference
AskPDF
Aug 7, 2025

Algorithmic Shortlisting in Participatory Budgeting

Participatory budgeting is a democratic innovation that allows citizens to propose and vote on public investment projects. To help organizers manage large volumes of submissions, we design and test privacy-preserving methods for algorithmic shortlisting. These algorithms predict which projects are likely to be funded using only project features and anonymous historical voting data. We demonstrate the limitations of a naive approach that uses a large language model to rank projects based on past success and propose a vote-based pipeline that enables state-of-the-art LLMs to perform on par with classical machine learning. Our findings indicate that user preferences in participatory budgeting are stable enough to allow algorithmic shortlisting to approximate an initial selection of projects effectively.

Juan Ignacio Zambrano, Clément Contet, Jairo Gudiño-Rosero +3

IRIT · Université Toulouse Capitole · Université de Toulouse +3

Learning to RankPrivacy-Preserving MLUser Preference Modeling
AskPDF
Aug 7, 2025

High-Dimensional Differentially Private Quantile Regression: Distributed Estimation and Statistical Inference

With the development of big data and machine learning, privacy concerns have become increasingly critical, especially when handling heterogeneous datasets containing sensitive personal information. Differential privacy provides a rigorous framework for safeguarding individual privacy while enabling meaningful statistical analysis. In this paper, we propose a differentially private quantile regression method for high-dimensional data in a distributed setting. Quantile regression is a powerful and robust tool for modeling the relationships between the covariates and responses in the presence of outliers or heavy-tailed distributions. To address the computational challenges due to the non-smoothness of the quantile loss function, we introduce a Newton-type transformation that reformulates the quantile regression task into an ordinary least squares problem. Building on this, we develop a differentially private estimation algorithm with iterative updates, ensuring both near-optimal statistical accuracy and formal privacy guarantees. For inference, we further propose a differentially private debiased estimator, which enables valid confidence interval construction and hypothesis testing. Additionally, we propose a communication-efficient and differentially private bootstrap for simultaneous hypothesis testing in high-dimensional quantile regression, suitable for distributed settings with both small and abundant local data. Extensive simulations demonstrate the robustness and effectiveness of our methods in practical scenarios.

Ziliang Shen, Caixing Wang, Shaoli Wang +1

School of Statistics and Data Science, Shanghai University of Finance and Economics · School of Statistics and Data Science, Southeast University · School of Statistics and Data Science, Shanghai University of International Business and Economics

Quantile RegressionPrivacy-Preserving MLDifferential Privacy
AskPDF
Jul 7, 2025

Efficient Unlearning with Privacy Guarantees

Privacy protection laws, such as the GDPR, grant individuals the right to request the forgetting of their personal data not only from databases but also from machine learning (ML) models trained on them. Machine unlearning has emerged as a practical means to facilitate model forgetting of data instances seen during training. Although some existing machine unlearning methods guarantee exact forgetting, they are typically costly in computational terms. On the other hand, more affordable methods do not offer forgetting guarantees and are applicable only to specific ML models. In this paper, we present efficient unlearning with privacy guarantees (EUPG), a novel machine unlearning framework that offers formal privacy guarantees to individuals whose data are being unlearned. EUPG involves pre-training ML models on data protected using privacy models, and it enables {\em efficient unlearning with the privacy guarantees offered by the privacy models in use}. Through empirical evaluation on four heterogeneous data sets protected with kk-anonymity and εε-differential privacy as privacy models, our approach demonstrates utility and forgetting effectiveness comparable to those of exact unlearning methods, while significantly reducing computational and storage costs. Our code is available at https://github.com/najeebjebreel/EUPG.

Josep Domingo-Ferrer, Najeeb Jebreel, David Sánchez

1CYBERCAT (Center for Cybersecurity Research of2026 Catalonia), ComSCIAM-Center for Computational Science and Applied Mathematics, Department of Computer EngineeringJun and Mathematics, Universitat Rovira i Virgili, Av. Pa¨ısos26 Catalans 26, E-43007 Tarragona, Catalonia

Privacy-Preserving MLMachine Unlearning
AskPDF
Jun 18, 2025

Federated Learning for MRI-based BrainAGE: a multicenter study on post-stroke functional outcome prediction

Objective:\textbf{Objective:} Brain-predicted age difference (BrainAGE) is a neuroimaging biomarker reflecting brain health. However, training robust BrainAGE models requires large datasets, often restricted by privacy concerns. This study evaluates the performance of federated learning (FL) for BrainAGE estimation in ischemic stroke patients treated with mechanical thrombectomy, and investigates its association with clinical phenotypes and functional outcomes. Methods:\textbf{Methods:} We used FLAIR brain images from 1674 stroke patients across 16 hospital centers. We implemented standard machine learning and deep learning models for BrainAGE estimates under three data management strategies: centralized learning (pooled data), FL (local training at each site), and single-site learning. We reported prediction errors and examined associations between BrainAGE and vascular risk factors (e.g., diabetes mellitus, hypertension, smoking), as well as functional outcomes at three months post-stroke. Logistic regression evaluated BrainAGE's predictive value for these outcomes, adjusting for age, sex, vascular risk factors, stroke severity, time between MRI and arterial puncture, prior intravenous thrombolysis, and recanalisation outcome. Results:\textbf{Results:} While centralized learning yielded the most accurate predictions, FL consistently outperformed single-site models. BrainAGE was significantly higher in patients with diabetes mellitus across all models. Comparisons between patients with good and poor functional outcomes, and multivariate predictions of these outcomes showed the significance of the association between BrainAGE and post-stroke recovery. Conclusion:\textbf{Conclusion:} FL enables accurate age predictions without data centralization. The strong association between BrainAGE, vascular risk factors, and post-stroke recovery highlights its potential for prognostic modeling in stroke care.

Vincent Roca, Marc Tommasi, Paul Andrey +8

Univ. Lille, CNRS, Inserm, CHU Lille, Institut Pasteur de Lille, US 41 - UAR 2014 - PLBS, F-59000 Lille, France · Univ. Lille, Inria, F-59000 Lille, France · Univ. Montpellier, Inria, F-34090 Montpellier, France +3

Brain Age PredictionClinical Outcome PredictionPrivacy-Preserving MLBrain MRIFederated Learning
AskPDF
Jun 4, 2025

Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization

Gaussian Mixture Models (GMMs) are widely used statistical models for representing multi-modal data distributions, with numerous applications in data mining, pattern recognition, data simulation, and machine learning. However, recent research has shown that releasing GMM parameters poses significant privacy risks, potentially exposing sensitive information about the underlying data. In this paper, we address the challenge of releasing GMM parameters while ensuring differential privacy (DP) guarantees. Specifically, we focus on the privacy protection of mixture weights, component means, and covariance matrices. We propose to use Kullback-Leibler (KL) divergence as a utility metric to assess the accuracy of the released GMM, as it captures the joint impact of noise perturbation on all the model parameters. We develop a KL-divergence-driven optimization framework for differentially private GMM release that jointly designs privacy-budget allocation and parameter perturbations. Through theoretical analysis, we quantify the effects of privacy budget allocation and perturbation statistics on the DP guarantee, and derive a tractable expression for evaluating KL divergence. We formulate and solve an optimization problem to minimize the KL divergence between the released and fitted non-private models, subject to a given (ε,δ)(ε, δ)-DP constraint. Extensive experiments on both synthetic and real-world datasets demonstrate that our approach achieves strong privacy guarantees while maintaining high utility.

Hang Liu, Anna Scaglione, Sean Peisert

State Key Laboratory of Internet of Things for Smart City and the Department of Electrical and Computer Engineering, University of Macau, Macao S.A.R. · Department of Electrical and Computer Engineering, Cornell Tech, Cornell University, New York, NY, 10044 USA. · Computing Sciences Research, Lawrence Berkeley National Laboratory, Berkeley, CA 94720 USA.

Privacy-Preserving MLGaussian Mixture ModelsDifferential Privacy
AskPDF