Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search
Authors: Bo Ma, Wei Qi Yan, Jinsong Wu
Organizations: Auckland University of Technology, Auckland 1024 NZ · Resideo Technologies, Inc., Austin, TX 78759, USA · Guilin University of Electronic Technology, Gui Lin, China · Universidad de Chile, Santiago, Chile
Learning systems that preserve privacy often inject noise into hierarchical visual representations; a central challenge is to model how such perturbations align with a declared privacy budget in a way that is interpretable and applicable across vision backbones and vision--language models (VLMs). We propose Bodhi VLM, a privacy-alignment modeling framework for hierarchical neural representations: it (1) links sensitive concepts to layer-wise grouping via NCP and MDAV-based clustering; (2) locates sensitive feature regions using bottom-up (BUA) and top-down (TDA) strategies over multi-scale representations (e.g., feature pyramids or vision-encoder layers); and (3) uses an Expectation-Maximization Privacy Assessment (EMPA) module to produce an interpretable budget-alignment signal by comparing the fitted sensitive-feature distribution to an evaluator-specified reference (e.g., Laplace or Gaussian with scale c/ε). The output is reference-relative and is not a formal differential-privacy estimator. We formalize BUA/TDA over hierarchical feature structures and validate the framework on object detectors (YOLO, PPDPTS, DETR) and on the visual encoders of VLMs (CLIP, LLaVA, BLIP). BUA and TDA yield comparable deviation trends; EMPA provides a stable alignment signal under the reported setups. We compare with generic discrepancy baselines (Chi-square, K-L, MMD) and with task-relevant baselines (MomentReg, NoiseMLE, Wass-1). Results are reported as mean±std over multiple seeds with confidence intervals in the supplementary materials. This work contributes a learnable, interpretable modeling perspective for privacy-aligned hierarchical representations rather than a post hoc audit only. Source code: Bodhi-VLM GitHub repository
Figures & tables
Symbol
Meaning
D
Image dataset
S , Sϵ
Sensitive label set (subscript ϵ when emphasizing budget)
Mref(ϵ;η)
Evaluator-defined reference mechanism (family η )
f , f~
Original and perturbed layer-wise features
Xi
Set of feature vectors at layer i (e.g., tokens/patches/locations)
Gi(n) , Gi(s)
Non-sensitive and sensitive groups at layer i
TABLE I: Main notation.
Fig. 1: BUA in a VLM vision encoder: image input and patch embedding (or first CNN stage) at the bottom; bottom-up aggregation through Layer 1 to Layer n with NCP/MDAV partitioning into sensitive Gi(s) and non-sensitive Gi(n) at each layer; output to EMPA for privacy budget assessment.
Fig. 2: TDA and BUA in VLM: top-down and bottom-up feature search over the vision encoder layers (e.g., ViT blocks); layer-wise groups Gi(n) , Gi(s) feed EMPA for feature-level budget-alignment assessment.
Item
Specification
Feature normalization
ℓ2 normalization per feature vector before MDAV/NCP (where applied)
MDAV group size
k=8 (minimum cluster size for microaggregation)
NCP weights wa
Uniform: wa=1/d for a d -dimensional feature
Range Ra
Inter-quantile range (q0.95−q0.05) computed on the training split for each dimension
Sensitivity threshold τi
90 th percentile of s(⋅;Sϵ) within layer i (per Section III-C )
BUA aggregation unit
Grouping computed per image; summary statistics averaged across images
TABLE II: Reproducibility details for BUA/TDA/EMPA used in our experiments.
Fig. 3: BUA vs. TDA deviation on YOLO (MOT20).
Fig. 4: BUA vs. TDA deviation on PPDPTS (MOT20).
Fig. 5: Deviation: TDA+EMPA vs. BUA+EMPA (MOT20, PPDPTS ϵ=0.001 ).
Indicator
MDCRF-0.1
DETR-0.1
PPDPTS-0.1
PPDPTS-0.01
Chi-square (rMSE)
0.40 ± 0.01
0.36 ± 0.00
0.40 ± 0.01
0.40 ± 0.01
K-L Div.
0.01 ± 0.00
0.47 ± 0.07
0.01 ± 0.00
0.01 ± 0.00
MMD
0.001 ± 0.000
0.005 ± 0.000
0.001 ± 0.000
0.001 ± 0.000
EMPA
0.89 ± 0.00
0.63 ± 0.13
0.89 ± 0.00
0.89 ± 0.00
TABLE III: rMSE (mean ± std over 5 seeds) for Chi-square, K-L, MMD, EMPA on detector backbones (aggregated over multi-seed evaluation; confidence intervals available in the supplementary materials).
Fig. 6: Privacy budget vs. rMSE: TDA+EMPA and BUA+EMPA (COCO).
ϵ
Chi-square
K-L
MMD
rMSE
Wass-1
EMPA (BUA)
EMPA (TDA)
0.1
(1.69±0.31)×107
2.045±0.219
0.0135±0.0020
5.45±0.21
1.94±0.13
0.894±0.000
0.894±0.000
0.01
(6.27±0.96)×107
4.507±0.516
0.0197±0.0028
54.50±2.15
23.24±1.59
0.894±0.000
0.894±0.000
TABLE IV: Metrics from synthetic hierarchical feature experiments (mean ± std over 5 seeds; full numerical summaries provided in the supplementary materials).
Fig. 7: EMPA bias vs. privacy budget ϵ (log scale) from the synthetic experiments. BUA+EMPA and TDA+EMPA coincide on synthetic data because the grouping yields the same partition; Random partition+EMPA (baseline) uses a random sensitive/non-sensitive partition of the same size and shows higher bias, illustrating that structured grouping (BUA/TDA) yields a more stable budget-alignment signal than a blind baseline.
Fig. 8: Left: rMSE and Chi-square (scaled ×10−7 ) vs. ϵ (log scale) for contrast. Right: Chi-square, K-L divergence, and MMD vs. ϵ . The dual curves in the left panel allow comparison of rMSE (budget-alignment error) with the raw Chi-square statistic across budgets.
CLIP ViT-B/32
LLaVA-1.5 7B
BLIP ResNet-50
BUA+EMPA
TDA+EMPA
BUA+EMPA
TDA+EMPA
BUA+EMPA
TDA+EMPA
Deviation ( ϵ=0.1 )
0.52
0.48
0.58
0.54
0.61
0.59
Deviation ( ϵ=0.01 )
1.24
1.18
1.31
1.26
1.38
1.35
rMSE ( ϵ=0.1 )
1.89
1.92
2.11
2.08
2.35
2.41
rMSE ( ϵ=0.01 )
2.34
2.38
2.67
2.62
2.89
2.95
TABLE V: BUA+EMPA vs. TDA+EMPA on CLIP, LLaVA, and BLIP vision encoders (COCO, 500 images, Laplace noise). Deviation: mean ∣noised−original∣ ; rMSE: budget-alignment error w.r.t. the Laplace reference target under the same declared ϵ .
Method
Approach
Privacy Guarantee
Compatible with Bodhi VLM assessment
ViP [ 14 ]
DP training (MAE+DP-SGD)
(ε,δ) -DP
Yes (vision features)
DP-Cap [ 15 ]
DP image captioning
(ε,δ) -DP
Yes (vision encoder)
DP-MTV [ 16 ]
DP multimodal in-context
(ε,δ) -DP
Partial (task vectors)
VisShield [ 17 ]
PHI de-identification
None (masking)
No (different paradigm)
Bodhi VLM (Ours)
Privacy-alignment modeling
N/A (modeling framework)
–
TABLE VI: Positioning table: scope and capability comparison with related privacy-aware vision/VLM methods. This is not a direct quantitative performance comparison—ViP, DP-Cap, DP-MTV, and VisShield address different tasks (training-time DP or de-identification). Bodhi VLM provides a privacy-alignment modeling framework for hierarchical representations.
Model
rMSE
EMPA (BUA)
CLIP ε=0.01
0.058±0.004
0.894±0.000
CLIP ε=0.1
0.059±0.004
0.894±0.000
BLIP ε=0.01
1.32±0.50
0.894±0.000
BLIP ε=0.1
1.51±0.48
0.894±0.000
TABLE VII: VLM vision encoder metrics from multi-seed experiments (mean ± std over 5 seeds; detailed summaries and confidence intervals are provided in the supplementary materials). rMSE and EMPA (BUA) for CLIP ViT-B/32 and BLIP ResNet-50.
PPDPTS (MOT20)
COCO (Laplace)
Method
Dev. ϵ=0.1
Dev. ϵ=0.01
Dev. ϵ=0.1
Dev. ϵ=0.01
rMSE ϵ=0.1
rMSE ϵ=0.01
Bodhi VLM (full)
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.36 ± 0.0
0.36 ± 0.0
w/o BUA (TDA only)
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.36 ± 0.0
0.36 ± 0.0
w/o TDA (BUA only)
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.36 ± 0.0
0.36 ± 0.0
w/o EMPA
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.22 ± 0.0
0.6 ± 0.01
0.61 ± 0.01
TABLE VIII: Ablation study of Bodhi VLM on PPDPTS (MOT20) and COCO. Mean ± std over 5 seeds; 95% CI in supplementary Table S3. Deviation: mean ∣noised−original∣ ; rMSE: budget-alignment error w.r.t. the Laplace reference under the same declared ϵ . Full pipeline uses BUA and TDA grouping plus EMPA.
Fig. 9: Sensitive vs. non-sensitive feature distribution (BUA, layer 0; noised features).
Fig. 10: t-SNE (or PCA) of layer-wise features colored by BUA grouping: sensitive ( Gi(s) ) vs. non-sensitive ( Gi(n) ).
Existing visual privacy benchmarks largely treat privacy as a binary property, labeling images as private or non-private based on visible sensitive content. We argue that privacy is fundamentally compositional. Attributes that are benign in isolation may combine to produce severe privacy violations. We introduce the Compositional Privacy Risk Taxonomy (CPRT), a regulation-aware framework that organizes visual attributes according to standalone identifiability and compositional harm potential. CPRT defines four graded severity levels and is paired with an interpretable scoring function that assigns continuous privacy severity scores. We further construct a taxonomy-aligned dataset of 6.7K images and derive compositional risk scores. By evaluating frontier and open-weight VLMs we find that frontier models align well with compositional severity when provided structured guidance, but systematically underestimate composition-driven risks. Smaller models struggle to internalize graded privacy reasoning. To bridge this gap, we introduce a deployable 8B SFT model that closely matches frontier-level performance on compositional privacy assessment
Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.
Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein +8
Ulm University, Germany · University of Lübeck, Germany · Technical University of Munich, Germany +4
As Visual Language Models (VLMs) become increasingly embedded in everyday applications, ensuring they can recognise and appropriately handle privacy-sensitive content is thus essential to protect users. To this end, we conduct a comprehensive evaluation of twelve state-of-the-art VLMs and identify limitations in their understanding of visual privacy. However, existing privacy-related datasets often suffer from label inconsistencies, limiting their reliability. To address this, we introduce two compact, high-quality benchmarks, PrivBench and PrivBench-H, that focus on commonly recognised visual privacy categories aligned with the General Data Protection Regulation (GDPR). Additionally, we present PrivTune, an instruction-tuning dataset specifically curated to improve privacy sensitivity. We obtain multiple Privacy VLMs by fine-tuning off-the-shelf VLMs on only a few hundred samples from PrivTune, which leads to substantial gains on all benchmarks, surpassing even GPT-4, while maintaining strong performance on other tasks. Our findings show that privacy-awareness in VLMs can be substantially improved with minimal data and careful dataset design, setting the stage for safer, more privacy-aligned AI systems.
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg