Nearest-Neighbor Density Estimation for Dependency Suppression

Kathleen Anderson; Thomas Martinetz

cs.LGMar 4, 2026

Nearest-Neighbor Density Estimation for Dependency Suppression

Authors: Kathleen Anderson, Thomas Martinetz

Organizations: Institute for Neuro- and Bioinformatics, University of Lübeck, Germany

Abstract

The ability to remove unwanted dependencies from data is crucial in various domains, including fairness, robust learning, and privacy protection. In this work, we propose an encoder-based approach that learns a representation independent of a sensitive variable but otherwise preserving essential data characteristics. Unlike existing methods that rely on decorrelation or adversarial learning, our approach explicitly estimates and modifies the data distribution to neutralize statistical dependencies. To achieve this, we combine a specialized variational autoencoder with a novel loss function driven by non-parametric nearest-neighbor density estimation, enabling direct optimization of independence. We evaluate our approach on multiple datasets, demonstrating that it can outperform existing unsupervised techniques and even rival supervised methods in balancing information removal and utility.

Figures & tables

Panel of Figure 1 — Figure 1: The pipeline used to translate input samples $x$ into invariant versions $x^{\prime}$ .

Panel of Figure 2 — Figure 2: Depiction for the method described in section 4.2 . To estimate the density of $z$ , one effectively counts the points that are in its circumference $\mathcal{B}$ .

	Base	Supervised			Unsupervised
		VAE	Contrastive	Adversary	VAE	Contrastive	Ours
		Lu 2023	Akash 2021	Roy 2019	Liu 2022	Akash 2021
devices	66.5	54.0	59.4	54.9	60.7	62.9	60.3
observations	70.5	66.1	63.7	66.1	69.8	67.8	72.6
\| dev. - observ. \|	4.0	12.1	4.3	11.2	9.1	4.8	12.4

Table 2: Accuracy of a trained classifier on transformed CheXpert data. Unlike table 1 , not the highest, but the last accuracy after 10 epochs is reported.

Figure 3: Validation accuracy of an MLP on MNIST with noisy labels. The blue (upper) line shows performance after background removal, while the orange line is the baseline. A noise ratio of 0.2 means 20% of training labels were randomly replaced.

Figure 4: T-SNE embedding for the MNIST dataset with backgrounds. The top row is colored by background label, bottom by digit. From left to right, the columns depict the original data, the VAE latent ( $z_{vae}$ ) and the encoded latent ( $z_{enc}$ ).

Figure 5: Images reconstructed from StyleGAN latents (see section 6.5 ). The bottom row shows images reconstructed from latents that have been translated using our nearest-neighbor divergence, to remove gender information. Note how the gender is not "removed", but rather shifted randomly for every image.

	MNIST	FFHQ	CheXpert
sensitive	55.1	80.4	67.1
target	97.0	85.4	74.0
\| sensitive - target. \|	41.9	5.0	6.9

Table 3: Highest test accuracies of networks trained to predict from images transformed by the specialized VAE, but not the nearest neighbor latent encoder.

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 433 papers

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
May 29, 2026Zhengyang Hu, Yanzhi Chen, Hanxiang Ren +5Amortized Inference Mutual Information Estimation
The University of Hong Kong · University of Cambridge · Microsoft +2
Principled Nonlinear Concept Erasure and Steering
Oct 23, 2026Idan Simai, Shauli Ravfogel, Uri ShahamConcept Erasure Privacy-Preserving Representation Learning
Department of Computer Science, Bar-Ilan University · Center for Data Science, New York University
In-Context Density Estimation for Tabular Data
Aug 10, 2026Patryk Marszałek, Jacek Tabor, Marek ŚmiejaEnergy-Based Models Unsupervised Anomaly Detection
Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland

May 29, 2026J/K move · Enter open · S save

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Zhengyang Hu, Yanzhi Chen, Hanxiang Ren, Qunsong Zeng+4

The University of Hong Kong · University of Cambridge · Microsoft+2

Amortized Inference Mutual Information Estimation

Open paper AskPDF arXiv

Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas, a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving

100\times

speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis.

Abstract

Privacy-Preserving ML

Explore similar work

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Principled Nonlinear Concept Erasure and Steering

In-Context Density Estimation for Tabular Data