Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search

Bo Ma; Wei Qi Yan; Jinsong Wu

cs.CVMar 14, 2026

Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search

Authors: Bo Ma, Wei Qi Yan, Jinsong Wu

Organizations: Auckland University of Technology, Auckland 1024 NZ · Resideo Technologies, Inc., Austin, TX 78759, USA · Guilin University of Electronic Technology, Gui Lin, China · Universidad de Chile, Santiago, Chile

Abstract

Learning systems that preserve privacy often inject noise into hierarchical visual representations; a central challenge is to model how such perturbations align with a declared privacy budget in a way that is interpretable and applicable across vision backbones and vision--language models (VLMs). We propose Bodhi VLM, a privacy-alignment modeling framework for hierarchical neural representations: it (1) links sensitive concepts to layer-wise grouping via NCP and MDAV-based clustering; (2) locates sensitive feature regions using bottom-up (BUA) and top-down (TDA) strategies over multi-scale representations (e.g., feature pyramids or vision-encoder layers); and (3) uses an Expectation-Maximization Privacy Assessment (EMPA) module to produce an interpretable budget-alignment signal by comparing the fitted sensitive-feature distribution to an evaluator-specified reference (e.g., Laplace or Gaussian with scale $c/ε$ ). The output is reference-relative and is not a formal differential-privacy estimator. We formalize BUA/TDA over hierarchical feature structures and validate the framework on object detectors (YOLO, PPDPTS, DETR) and on the visual encoders of VLMs (CLIP, LLaVA, BLIP). BUA and TDA yield comparable deviation trends; EMPA provides a stable alignment signal under the reported setups. We compare with generic discrepancy baselines (Chi-square, K-L, MMD) and with task-relevant baselines (MomentReg, NoiseMLE, Wass-1). Results are reported as mean $\pm$ std over multiple seeds with confidence intervals in the supplementary materials. This work contributes a learnable, interpretable modeling perspective for privacy-aligned hierarchical representations rather than a post hoc audit only. Source code: Bodhi-VLM GitHub repository

Figures & tables

Symbol	Meaning
$D$	Image dataset
$S$ , $S_{\epsilon}$	Sensitive label set (subscript $\epsilon$ when emphasizing budget)
$\mathcal{M}_{\mathrm{ref}}(\epsilon;\eta)$	Evaluator-defined reference mechanism (family $\eta$ )
$f$ , $\tilde{f}$	Original and perturbed layer-wise features
$X_{i}$	Set of feature vectors at layer $i$ (e.g., tokens/patches/locations)
$\mathcal{G}^{(n)}_{i}$ , $\mathcal{G}^{(s)}_{i}$	Non-sensitive and sensitive groups at layer $i$

TABLE I: Main notation.

Fig. 1: BUA in a VLM vision encoder: image input and patch embedding (or first CNN stage) at the bottom; bottom-up aggregation through Layer 1 to Layer $n$ with NCP/MDAV partitioning into sensitive $\mathcal{G}^{(s)}_{i}$ and non-sensitive $\mathcal{G}^{(n)}_{i}$ at each layer; output to EMPA for privacy budget assessment.

Fig. 2: TDA and BUA in VLM: top-down and bottom-up feature search over the vision encoder layers (e.g., ViT blocks); layer-wise groups $\mathcal{G}^{(n)}_{i}$ , $\mathcal{G}^{(s)}_{i}$ feed EMPA for feature-level budget-alignment assessment.

Item	Specification
Feature normalization	$\ell_{2}$ normalization per feature vector before MDAV/NCP (where applied)
MDAV group size	$k=8$ (minimum cluster size for microaggregation)
NCP weights $w_{a}$	Uniform: $w_{a}=1/d$ for a $d$ -dimensional feature
Range $R_{a}$	Inter-quantile range $(q_{0.95}-q_{0.05})$ computed on the training split for each dimension
Sensitivity threshold $\tau_{i}$	$90$ th percentile of $s(\cdot;S_{\epsilon})$ within layer $i$ (per Section III-C )
BUA aggregation unit	Grouping computed per image; summary statistics averaged across images

TABLE II: Reproducibility details for BUA/TDA/EMPA used in our experiments.

Fig. 3: BUA vs. TDA deviation on YOLO (MOT20).

Fig. 4: BUA vs. TDA deviation on PPDPTS (MOT20).

Fig. 5: Deviation: TDA+EMPA vs. BUA+EMPA (MOT20, PPDPTS $\epsilon=0.001$ ).

Indicator	MDCRF-0.1	DETR-0.1	PPDPTS-0.1	PPDPTS-0.01
Chi-square (rMSE)	0.40 $\pm$ 0.01	0.36 $\pm$ 0.00	0.40 $\pm$ 0.01	0.40 $\pm$ 0.01
K-L Div.	0.01 $\pm$ 0.00	0.47 $\pm$ 0.07	0.01 $\pm$ 0.00	0.01 $\pm$ 0.00
MMD	0.001 $\pm$ 0.000	0.005 $\pm$ 0.000	0.001 $\pm$ 0.000	0.001 $\pm$ 0.000
EMPA	0.89 $\pm$ 0.00	0.63 $\pm$ 0.13	0.89 $\pm$ 0.00	0.89 $\pm$ 0.00

TABLE III: rMSE (mean

\pm

std over 5 seeds) for Chi-square, K-L, MMD, EMPA on detector backbones (aggregated over multi-seed evaluation; confidence intervals available in the supplementary materials).

Fig. 6: Privacy budget vs. rMSE: TDA+EMPA and BUA+EMPA (COCO).

$\epsilon$	Chi-square	K-L	MMD	rMSE	Wass-1	EMPA (BUA)	EMPA (TDA)
0.1	$(1.69\pm 0.31)\times 10^{7}$	$2.045\pm 0.219$	$0.0135\pm 0.0020$	$5.45\pm 0.21$	$1.94\pm 0.13$	$0.894\pm 0.000$	$0.894\pm 0.000$
0.01	$(6.27\pm 0.96)\times 10^{7}$	$4.507\pm 0.516$	$0.0197\pm 0.0028$	$54.50\pm 2.15$	$23.24\pm 1.59$	$0.894\pm 0.000$	$0.894\pm 0.000$

TABLE IV: Metrics from synthetic hierarchical feature experiments (mean

\pm

std over 5 seeds; full numerical summaries provided in the supplementary materials).

Fig. 7: EMPA bias vs. privacy budget $\epsilon$ (log scale) from the synthetic experiments. BUA+EMPA and TDA+EMPA coincide on synthetic data because the grouping yields the same partition; Random partition+EMPA (baseline) uses a random sensitive/non-sensitive partition of the same size and shows higher bias, illustrating that structured grouping (BUA/TDA) yields a more stable budget-alignment signal than a blind baseline.

Fig. 8: Left: rMSE and Chi-square (scaled $\times 10^{-7}$ ) vs. $\epsilon$ (log scale) for contrast. Right: Chi-square, K-L divergence, and MMD vs. $\epsilon$ . The dual curves in the left panel allow comparison of rMSE (budget-alignment error) with the raw Chi-square statistic across budgets.

	CLIP ViT-B/32		LLaVA-1.5 7B		BLIP ResNet-50
	BUA+EMPA	TDA+EMPA	BUA+EMPA	TDA+EMPA	BUA+EMPA	TDA+EMPA
Deviation ( $\epsilon{=}0.1$ )	0.52	0.48	0.58	0.54	0.61	0.59
Deviation ( $\epsilon{=}0.01$ )	1.24	1.18	1.31	1.26	1.38	1.35
rMSE ( $\epsilon{=}0.1$ )	1.89	1.92	2.11	2.08	2.35	2.41
rMSE ( $\epsilon{=}0.01$ )	2.34	2.38	2.67	2.62	2.89	2.95

TABLE V: BUA+EMPA vs. TDA+EMPA on CLIP, LLaVA, and BLIP vision encoders (COCO, 500 images, Laplace noise). Deviation: mean

|\mathrm{noised}-\mathrm{original}|

; rMSE: budget-alignment error w.r.t. the Laplace reference target under the same declared

\epsilon

Method	Approach	Privacy Guarantee	Compatible with Bodhi VLM assessment
ViP [ 14 ]	DP training (MAE+DP-SGD)	$(\varepsilon,\delta)$ -DP	Yes (vision features)
DP-Cap [ 15 ]	DP image captioning	$(\varepsilon,\delta)$ -DP	Yes (vision encoder)
DP-MTV [ 16 ]	DP multimodal in-context	$(\varepsilon,\delta)$ -DP	Partial (task vectors)
VisShield [ 17 ]	PHI de-identification	None (masking)	No (different paradigm)
Bodhi VLM (Ours)	Privacy-alignment modeling	N/A (modeling framework)	–

TABLE VI: Positioning table: scope and capability comparison with related privacy-aware vision/VLM methods. This is not a direct quantitative performance comparison—ViP, DP-Cap, DP-MTV, and VisShield address different tasks (training-time DP or de-identification). Bodhi VLM provides a privacy-alignment modeling framework for hierarchical representations.

Model	rMSE	EMPA (BUA)
CLIP $\varepsilon=0.01$	$0.058\pm 0.004$	$0.894\pm 0.000$
CLIP $\varepsilon=0.1$	$0.059\pm 0.004$	$0.894\pm 0.000$
BLIP $\varepsilon=0.01$	$1.32\pm 0.50$	$0.894\pm 0.000$
BLIP $\varepsilon=0.1$	$1.51\pm 0.48$	$0.894\pm 0.000$

TABLE VII: VLM vision encoder metrics from multi-seed experiments (mean

\pm

std over 5 seeds; detailed summaries and confidence intervals are provided in the supplementary materials). rMSE and EMPA (BUA) for CLIP ViT-B/32 and BLIP ResNet-50.

	PPDPTS (MOT20)		COCO (Laplace)
Method	Dev. $\epsilon{=}0.1$	Dev. $\epsilon{=}0.01$	Dev. $\epsilon{=}0.1$	Dev. $\epsilon{=}0.01$	rMSE $\epsilon{=}0.1$	rMSE $\epsilon{=}0.01$
Bodhi VLM (full)	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.36 $\pm$ 0.0	0.36 $\pm$ 0.0
w/o BUA (TDA only)	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.36 $\pm$ 0.0	0.36 $\pm$ 0.0
w/o TDA (BUA only)	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.36 $\pm$ 0.0	0.36 $\pm$ 0.0
w/o EMPA	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.22 $\pm$ 0.0	0.6 $\pm$ 0.01	0.61 $\pm$ 0.01

TABLE VIII: Ablation study of Bodhi VLM on PPDPTS (MOT20) and COCO. Mean

\pm

std over 5 seeds; 95% CI in supplementary Table S3. Deviation: mean

|\mathrm{noised}-\mathrm{original}|

; rMSE: budget-alignment error w.r.t. the Laplace reference under the same declared

\epsilon

. Full pipeline uses BUA and TDA grouping plus EMPA.

Fig. 9: Sensitive vs. non-sensitive feature distribution (BUA, layer 0; noised features).

Fig. 10: t-SNE (or PCA) of layer-wise features colored by BUA grouping: sensitive ( $\mathcal{G}^{(s)}_{i}$ ) vs. non-sensitive ( $\mathcal{G}^{(n)}_{i}$ ).

Privacy-Preserving Representation Learning

See all 9 papers

Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs
Mar 23, 2026Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna +3VLM Evaluation Privacy Auditing
University of Southern California, Los Angeles CA 90089, USA · Meta Superintelligence Labs · Amazon AGI
Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings
Mar 26, 2026Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein +8VLM Interpretability Vision-Language Alignment
Ulm University, Germany · University of Lübeck, Germany · Technical University of Munich, Germany +4
Privacy-Aware Visual Language Models
May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM Evaluation VLM Adaptation
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

Mar 23, 2026J/K move · Enter open · S save

Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs

Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna, Sai Praneeth Karimireddy+2

University of Southern California, Los Angeles CA 90089, USA · Meta Superintelligence Labs · Amazon AGI

VLM Evaluation Privacy Auditing Compositional Reasoning in VLMs VLM Reasoning Privacy Protection in VLMs

Open paper AskPDF arXiv

Existing visual privacy benchmarks largely treat privacy as a binary property, labeling images as private or non-private based on visible sensitive content. We argue that privacy is fundamentally compositional. Attributes that are benign in isolation may combine to produce severe privacy violations. We introduce the Compositional Privacy Risk Taxonomy (CPRT), a regulation-aware framework that organizes visual attributes according to standalone identifiability and compositional harm potential. CPRT defines four graded severity levels and is paired with an interpretable scoring function that assigns continuous privacy severity scores. We further construct a taxonomy-aligned dataset of 6.7K images and derive compositional risk scores. By evaluating frontier and open-weight VLMs we find that frontier models align well with compositional severity when provided structured guidance, but systematically underestimate composition-driven risks. Smaller models struggle to internalize graded privacy reasoning. To bridge this gap, we introduce a deployable 8B SFT model that closely matches frontier-level performance on compositional privacy assessment

Abstract

Privacy-Preserving Representation Learning

Explore similar work

Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs

Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

Privacy-Aware Visual Language Models