Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness

Sleem Abdelghafar; Maryam Aliakbarpour; Christopher Jermaine

cs.LGOct 13, 2025

Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness

Authors: Sleem Abdelghafar, Maryam Aliakbarpour, Christopher Jermaine

Organizations: Rice University Houston, USA

Abstract

Auditing information disclosure across every datapoint during the training of LLMs is challenging. We propose a principled, attack-agnostic approach that uses mutual information to measure what the final model reveals about a datapoint's training membership. We show that this final-model disclosure is upper bounded by the sum of per-iteration gradient disclosures and that, under a reasonable set of assumptions, these gradient disclosures increase with Gradient Uniqueness (GNQ), which measures how distinguishable a datapoint's gradient is relative to other gradients in the batch. While naively computing GNQ requires forming and inverting a $P\times P$ matrix for every datapoint (for a model with $P$ parameters), we introduce Batch-Space Ghost (BS-Ghost). This efficient algorithm performs all computations in a much smaller batch space and uses ghost kernels to compute GNQ "in-run" for every datapoint in the training corpus, with minimal computational and memory overhead. Our experiments show the following: (i) GNQ predicts MIA vulnerability without the need for shadow models. (ii) Beyond membership disclosure, GNQ predicts the success of reconstruction attacks. (iii) GNQ-guided removal and retraining identify datapoints that causally contribute to disclosure. (iv) GNQ outperforms counterfactual memorization in text extraction and common-knowledge discrimination without the need for additional model training. (v) For data attribution, GNQ-guided filtering reduces emergent misalignment in Qwen2.5-7B more than baselines. Further, GNQ attributes 1000 datapoints in 17 seconds---roughly $70\times$ faster than the baselines. (vi) GNQ explains how training choices affect training-set disclosure and captures how per-datapoint disclosure emerges during training.

Figures & tables

	Desiderata
Prior work	C1	C2	C3	C4
Attack-based: ( Carlini et al., 2021 ; Nasr et al., 2025 ; Hayes et al., 2025 ; Yu et al., 2023 ; Carlini et al., 2023 ; Peng et al., 2023 ; Ippolito et al., 2023 ; Zhou et al., 2024 ; Kim et al., 2023 ; Lukas et al., 2023 ; Shokri et al., 2017 ; Mattern et al., 2023 )	✗	✗	✓	✗
Canary-based: ( Carlini et al., 2019 ; Parikh et al., 2022 ; Steinke et al., 2023 )	✗	✗	✗	✗
Counterfactual: ( Zhang et al., 2023 ; Feldman, 2020 ; Feldman and Zhang, 2020 )	✓	✗	✗	✗

Table 1: Disclosure auditing approaches versus desiderata defined in the Introduction (C1–C4).

Panel of Figure 1 — Figure 1: What does GNQ Measure?

Table 3

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parameters	Training peak	Extra GNQ peak
GPT-2 Small	124M	5641 MiB	$<0.1$ MiB
GPT-2 Medium	355M	12594 MiB	$<0.1$ MiB

Appendix

Table 4: Peak GPU memory during training. GNQ adds less than 0.1 MiB to the peak.

Panel of Figure 12 — Figure 12: High-GNQ classes are easier to reconstruct. Faces class 6 (highest GNQ) is recognizable after one epoch at 10–30% accuracy and after 30 epochs at about 95% (Figure (b)).

Selection	Inspected	Surprising/false	Common facts
GNQ	20	20	0
GNQ	40	32	3
GNQ	80	58	6
Counterfactual	20	7	0
Counterfactual	40	19	0
Counterfactual	80	39	2

Appendix

Table 5: Recovered common facts and surprising false assertions by selection method (

n=400

Panel of Figure 13 — Figure 13: GNQ and MIA performance across four training choices. Blue: AUC-ROC, left axis. Red: mean GNQ, right axis, normalized for model size. Both rise overall with iterations, model size, and batch size, and fall with learning rate.

Trajectory	Avg. GNQ	Training text
High #1	12,973	Sleek Looks and Superb Performance to Woo Mac Fans and PC Buyers Most all-in-one desktops with LCD panels (the Sony VAIO and Gateway Profile systems come to mind) are two units permanently connected together: the part of the case housing the motherboard and drives, and the monitor…
High #2	5,334	EU seeks revamp budget pact The European Commission on Friday suggested ways of revamping the EU’s Stability and Growth Pact and said the proposed changes would restore the credibility of its much-flouted budget discipline rules…
Low #1	543	UN warns of Sudan refugee exodus Some 30,000 Sudanese refugees might cross into Chad to escape persecution by Arab militia, the UN warns…
Low #2	551	Philadelphia Considers Wireless Internet for All By DAVID B. CARUSO PHILADELPHIA (AP) – For about $10 million, city officials believe they can turn all 135 square miles of Philadelphia into the world’s largest wireless Internet hot spot…

Appendix

Table 6: Training texts for the two highest- and two lowest-average GNQ trajectories in Fig. .

Figure 14: Misaligned Opinions after GNQ-guided removal and retraining of Qwen2.5-7B. Lower scores are better. Bad Medical Advice is shown in Figure .

Explore similar work

Jun 23, 2026cs.CV

Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients

Vision-Language Large Models (VLLMs) trained on massive crawled corpora raise pressing copyright and data-provenance concerns. These concerns are particularly acute in healthcare, where patient medical images paired with clinical reports demand rigorous privacy safeguards. However, existing training data detection methods either fail in cross-modal scenarios or rely on superficial output signals with insufficient discriminative power. We introduce GradAudit, a gradient-based auditing framework that examines internal optimization dynamics rather than treating VLLMs as black boxes. Our approach builds on a key observation: model parameters converge to regions where gradients on training samples become stable and well-aligned, whereas gradients on non-training samples remain noisy and inconsistent. By analyzing these gradient signatures, GradAudit achieves strong separability and detects genuine image-text associations learned during training, not merely individual modality membership. Empirically, across both medical and general-domain datasets, GradAudit substantially outperforms state-of-the-art baselines in both pretraining and fine-tuning VLLMs. In a case study employing copyrighted content, we show that existing training data detection methods not only underestimate the extent of unauthorized data usage, but that this underestimation becomes more pronounced as models become more recent and more advanced.

Zhihao Zhu, Hongyi Tang, Yi Yang +1

Department of Information Systems, Business Statistics and Operations Management (ISOM), Hong Kong University of Science and Technology, Hong Kong, China. · Department of IT, Analytics, and Operations, University of Notre Dame, Notre Dame, Indiana, USA.

Training Data Generation Provenance Gradient

Ask PDF

Jun 3, 2026cs.LG

Data Attribution in Large Language Models via Bidirectional Gradient Optimization

Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance. Understanding which training data most influenced a model's output remains a fundamental open problem. We address this challenge through training data attribution (TDA) for auto-regressive LLMs by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training? Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples. Our framework supports attribution at arbitrary data granularity, enabling both factual and stylistic attribution. We evaluate our method against baselines on pretrained models with known datasets, and show that it outperforms previous work on influence metrics, thereby enhancing model interpretability, an essential requirement for accountable AI systems.

Frédéric Berdoz, Luca A. Lanzendörfer, Kaan Bayraktar +1

ETH Zurich

Large Language Model Training Autoregressive Language Models Interpretability Gradient

Ask PDF

Jul 2, 2025cs.LG

Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training

Gradient-based optimization is the workhorse of deep learning, offering efficient and scalable training via backpropagation. However, exposing gradients during training can leak sensitive information about the underlying data, raising privacy and security concerns such as susceptibility to data poisoning attacks. In contrast, black-box optimization methods, which treat the model as an opaque function, relying solely on function evaluations to guide optimization, offer a promising alternative in scenarios where data access is restricted, adversarial risks are high, or overfitting is a concern. This paper introduces BBoxER, an evolutionary black-box method for LLM post-training that induces an information bottleneck via implicit compression of the training data. Leveraging the tractability of information flow, we provide non-vacuous generalization bounds and strong theoretical guarantees for robustness to data poisoning attacks and extraction attacks, while ensuring privacy by design. In experiments with LLMs, we demonstrate empirically that black-box optimization methods-despite the scalability and computational challenges inherent to black-box approaches-are able to learn, showing how a few iterations of BBoxER improve performance, generalize well on a benchmark of reasoning datasets, and are robust to membership inference attacks. This positions BBoxER as an attractive add-on on top of gradient-based optimization, offering suitability for deployment in restricted environments while also providing non-vacuous generalization guarantees.

Ismail Labiad, Mathurin Videau, Matthieu Kowalski +4

Meta FAIR · Université Paris-Saclay · LISN, Inria, CNRS +2

Privacy-Preserving Machine Learning Black-Box Optimization Generalization Bounds Tuning Opacity

Ask PDF