A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures
Authors: Yaiza Bermudez, Samir M. Perlaza, Iñaki Esnaola
Organizations: Centre Inria d’Université Côte d’Azur, INRIA, 06902 Sophia Antipolis, France · College of Computing, Université Mohamed VI Polytechnique (UM6P), Ben Guerir, 43150, Morocco · Department of Electrical and Computer Engineering at Princeton University, Princeton, NJ 08544 USA · GAATI Mathematics Laboratory at the University of French Polynesia, 98702 Faaa, French Polynesia · School of Electrical and Electronic Engineering, University of Sheffield, Sheffield S1 3JD, U.K.
In this paper, three operations on Gibbs probability measures are studied: renormalization, normalized log-linear combination, and nesting. The resulting measures are shown to be Gibbs probability measures with explicit expressions for their objective functions, reference measures, and regularization factors. These measures are further characterized as unique solutions to optimization problems involving linear combinations of the original objective functions, subject to either relative entropy regularization or relative entropy constraints. When the reference measures are probability measures, geometric interpretations of these three operations are established involving Pythagorean identities and exponential geodesics.A central result is the equivalence between nesting and normalized log-linear combinations of Gibbs probability measures. These results are then applied to supervised machine learning with Gibbs algorithms, which are represented by Gibbs probability measures. In this context, either through nesting or log-linear combinations, Gibbs algorithms trained on local datasets are used to form new Gibbs algorithms trained upon the aggregation of all local training datasets. Interestingly, the parameters of such operations, determine the weights of each training data point on the aggregated dataset. %Operations on local Gibbs probability measures are shown to control the contributions of training data. This observation is leveraged for achieving: (a) exact machine unlearning in centralized, decentralized, and federated learning systems; and (b) centralized-equivalent performance, in the sense of producing algorithms trained on the aggregation of all local datasets in decentralized and federated learning .
Figures & tables
Fig. 1: Geometric representation of the (α,Q,Q1) -renormalization of PY∣X=x(h,Q,λ) in ( 6 ). The original Gibbs probability measure PY∣X=x(h,Q,λ) lies on the boundary of GQ,γ , whereas the renormalized probability measure PY∣X=x(h,Q1,αλ) in ( 17 ) lies on the boundary of GQ1,η1 . The reals γ and η1 are defined in ( 63 ) and ( 64 ), respectively. The purple arrow represents the mapping from PY∣X=x(h,Q,λ) to PY∣X=x(h,Q1,αλ) , which is induced by the renormalization operation.
Fig. 2: Geometric representation of the (α,Q,Q) -renormalization of PY∣X=x(h,Q,λ) in ( 6 ). The solid purple curve represents the exponential geodesic generated by the measure PY∣X=x(h,Q,αλ) in ( 17 ) as α varies. The panels show (a) γ<η1 , with GQ,γ⊆GQ,η1 , and (b) η1<γ , with GQ,η1⊆GQ,γ . The sets GQ,γ and GQ,η1 are of the form in ( 62 ), with γ and η1 in ( 63 ) and ( 64 ), respectively.
Fig. 3: Geometric representation of the (α,Q,Q1) -renormalization of PY∣X=x(h,Q,λ) in ( 6 ), with Q1=Q . The solid purple curve represents the exponential geodesic generated by the measure PY∣X=x(h,Q1,αλ) in ( 17 ) when α varies. The panels show (a) η1<η2 , with GQ1,η1⊆GQ1,η2 , and (b) η1>η2 , with GQ1,η2⊆GQ1,η1 . The sets GQ,γ , GQ1,η1 , and GQ1,η2 are of the form in ( 62 ), with γ , η1 , and η2 defined in ( 63 ), ( 64 ), and ( 68 ), respectively.
Fig. 4: Geometric representation of the normalized log-linear combination PY∣X=x(hβ,Q,1) in ( 39 ), with Q1=Q2=Q and β=(λ1α,λ21−α) , where 0<α<1 . The solid purple curve represents the exponential segment obtained by varying α over [0,1] . Its endpoints are PY∣X=x2(h2,Q,λ2) at α=0 and PY∣X=x1(h1,Q,λ1) at α=1 , and it passes through the normalized log-linear combination at the chosen value of α . The neighborhoods GQ,γ1 , GQ,γ2 , and GQ,γ0 are defined by ( 62 ), with γ1 and γ2 given in ( 61 ) and γ0 given in ( 69 ). Under the illustrated ordering γ1<γ2 , the panels show (a) γ0<γ1<γ2 , (b) γ1<γ0<γ2 , and (c) γ1<γ2<γ0 .
Fig. 5: Geometric representation of the normalized log-linear combination PY∣X=x(hβ,Q,1) in ( 39 ), with β=(α/λ1,(1−α)/λ2) and 0<α<1 . The solid purple curve represents the exponential geodesic obtained by varying α over [0,1] . Its endpoints are PY∣X=x2(h2,Q,λ2) at α=0 and PY∣X=x1(h1,Q,λ1) at α=1 . The original Gibbs probability measures PY∣X=x1(h1,Q1,λ1) and PY∣X=x2(h2,Q2,λ2) are represented separately on the boundaries of GQ1,γ1 and GQ2,γ2 , respectively, and do not, in general, belong to the exponential geodesic. The sets GQ,η1 , GQ,η2 , and GQ,γ0 contain the two endpoints and the normalized log-linear combination, respectively, with η1 , η2 , and γ0 defined in ( 72 ), ( 73 ), and ( 69 ).
Fig. 6: Geometric representation of nested Gibbs probability measures illustrating (a) GQ1,γ1⊆GQ1,η and (b) GQ1,η⊆GQ1,γ1 . The dotted neighborhood GQ2,γ2 has reference measure Q2 in both cases.
Fig. 7: Pythagorean interpretation of Lemma 14 .
Fig. 8: Geometric representation of recursively nested Gibbs probability measures illustrating (a) GQ1,γ1⊆GQ1,η and (b) GQ1,η⊆GQ1,γ1 .
Fig. 9: Pythagorean interpretation of Lemma 15 .
Fig. 10: Geometric representation of exact unlearning under the assumption in ( 172 ), with γ1<η . The neighborhoods GQ,γ1 , GQ,η , and GPΘ∣Z=z0(Ln0,Q,λ0),γ2 are defined as in ( 62 ), with γ1 , γ2 , and η in ( 175 ), ( 176 ), and ( 177 ), respectively. The Gibbs probability measure in ( 167 ) and the nested Gibbs probability measure in ( 170 ) coincide and satisfy ( 178 ).
Fig. 11: Nested structure induced by the forward–backward peer-to-peer communication protocol: for all k∈{1,2,…,K−1} , the Gibbs probability measure produced by client k becomes the reference measure used by client k+1 .
Fig. 12: Geometric representation of recursively nested Gibbs probability measures for K=4 , under the assumption in ( 190 ). The neighborhoods GQk,γk , with k∈{1,2,…,4} , and GQ1,η are defined as in ( 62 ), with γk and η in ( 194 ) and ( 196 ), respectively. The points Qk+1 represent the Gibbs probability measures in ( 198 ). The final point Q5 in ( 199 ) coincides with the Gibbs probability measure in ( 192 ) obtained by training upon the aggregated dataset z0 .
Fig. 13: One-shot client–server communication system. For all k∈{1,2,…,K} , the Gibbs algorithm of client k , PΘ∣Z=zk(Lnk,Qk,λk) , is defined in ( 211 ), and the probability measure representing the aggregation at the server, SK , is defined in ( 212 ).
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
In this paper, a method for achieving exact unlearning for Gibbs supervised learning algorithms is proposed using a variational formulation inspired by empirical risk minimization subject to relative entropy regularization (ERM-RER). Such a method consists of maximizing the expected empirical risk over the dataset to be unlearned subject to a regularization by relative entropy with respect to the original algorithm. The optimization variable is a probability measure on the models; and the solution is another Gibbs probability measure that represents a new Gibbs supervised learning algorithm. The method guarantees exact unlearning in the sense that the new Gibbs algorithm coincides in distribution with the algorithm that would have been obtained by retraining from scratch on the dataset to be retained. As a byproduct, a framework for reweighting data points in ERM-RER by strategically choosing both the reference measure and the regularization factor is obtained. In this framework, exact unlearning is the special case in which zero-weight is assigned to the contribution of the data points to be unlearned. More generally, depending on the choice of certain parameters, data points can be up-weighted or down-weighted in ERM-RER problems for particular purposes, e.g., controlling the generalization error of Gibbs algorithms. This paves the way for new constructive or adversarial views on classical reweighting data points in ERM-RER.
Yaiza Bermudez, Samir M. Perlaza, Iñaki Esnaola
Centre Inria d’Universit´e Cˆote d’Azur, INRIA, Sophia Antipolis, France. · Laboratoire GAATI, Universit´e de la Polyn´esie franc¸aise, Fa‘a‘¯a, French Polynesia. · ECE Dept. Princeton University, Princeton, 08544 NJ, USA. +1
In this paper, it is shown, for the first time, that centralized performance is achievable in decentralized learning without sharing the local datasets. Specifically, when clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) learning framework and a forward-backward communication between clients is established, it suffices to share the locally obtained Gibbs measures to achieve the same performance as that of a centralized ERM-RER with access to all the datasets. The core idea is that the Gibbs measure produced by client~k is used, as reference measure, by client~k+1. This effectively establishes a principled way to encode prior information through a reference measure. In particular, achieving centralized performance in the decentralized setting requires a specific scaling of the regularization factors with the local sample sizes. Overall, this result opens the door to novel decentralized learning paradigms that shift the collaboration strategy from sharing data to sharing the local inductive bias via the reference measures over the set of models.
Yaiza Bermudez, Samir M. Perlaza, Iñaki Esnaola
Centre Inria d'Université Côte d'Azur, INRIA, Sophia Antipolis, France. · Laboratoire GAATI, Université de la Polynésie française, Fa'a'ā, French Polynesia. · ECE Dept. Princeton University, Princeton, 08544 NJ, USA. +1
How can we effectively remove or ``unlearn'' undesirable information, such as specific features or the influence of individual data points, from a learning outcome while minimizing utility loss and ensuring rigorous guarantees? We introduce a unified mathematical framework based on information-theoretic regularization to address both data-point unlearning and feature unlearning. For data-point unlearning, we introduce the Marginal Unlearning Principle, an auditable and provable framework. Moreover, we provide an information-theoretic unlearning definition based on the proposed principle and provable guarantees on sufficiency and necessity of marginal unlearning. We then show that the proposed framework provides a natural solution to the marginal unlearning problem and yields auditable high-probability marginal-unlearning guarantees. For feature unlearning, the framework applies to deep learning with flexible training objectives. By combining flexibility in learning objectives with simplicity in regularization design, our approach is highly adaptable and practical for a wide range of machine learning and AI applications. From a mathematical perspective, we provide a unified analytic solution to the optimal feature unlearning problem with a variety of information-theoretic training objectives. Our theoretical analysis reveals intriguing connections between machine unlearning, information theory, optimal transport, and extremal sigma algebras. Numerical simulations support our theoretical findings.
Shizhou Xu, Thomas Strohmer
Department of Mathematics University of California Davis Davis, CA 95616-5270, USA · Department of Mathematics Center of Data Science and Artificial Intelligence Research University of California Davis Davis, CA 95616-5270, USA