Frozen visual embeddings (e.g., CLIP, DINOv2/v3, SSCD) power retrieval and integrity systems, yet their use on face-containing data is constrained by unmeasured identity leakage and a lack of deployable mitigations. We take an attacker-aware view and contribute: (i) a benchmark of visual embeddings that reports open-set verification at low false-accept rates, a calibrated diffusion-based template inversion check, and face-context attribution with equal-area perturbations; and (ii) propose a one-shot linear projector that removes an estimated identity subspace while preserving the complementary space needed for utility, which for brevity we denote as the identity sanitization projection ISP. Across CelebA-20 and VGGFace2, we show that these encoders are robust under open-set linear probes, with CLIP exhibiting relatively higher leakage than DINOv2/v3 and SSCD, robust to template inversion, and are context-dominant. In addition, we show that ISP drives linear access to near-chance while retaining high non-biometric utility, and transfers across datasets with minor degradation. Our results establish the first attacker-calibrated facial privacy audit of non-FR encoders and demonstrate that linear subspace removal achieves strong privacy guarantees while preserving utility for visual search and retrieval.
Figures & tables
Figure 1: Identity Clustering in CLIP Embeddings. t-SNE visualization (perplexity=30, 1000 iterations) of test embeddings colored by identity. Left: RAW embeddings exhibit visible identity clustering. Right: ISP-sanitized embeddings with identity structure removed via orthogonal projection onto complement of top-192 identity directions. Embeddings were calculated over a randomly sampled VGGFace2 test split of 80 identities and 1,600 images (20 per identity). t-SNE computed on L2-normalized 512-dimension embeddings.
k=1
k=4
k=16
Dataset
Model
RAW
ISP-W
ISP-X
RAW
ISP-W
ISP-X
RAW
ISP-W
ISP-X
CelebA-20
DINOv2
4.5%
3.5%
3.7%
5.5%
3.9%
3.9%
5.7%
4.4%
4.2%
DINOv3
4.5%
2.1%
1.9%
6.7%
2.4%
2.2%
6.8%
2.8%
2.5%
CLIP
16.4%
11.9%
10.9%
19.7%
15.4%
10.6%
19.8%
13.0%
10.2%
SSCD
6.6%
3.6%
3.4%
8.4%
3.9%
3.9%
9.8%
4.4%
4.5%
ArcFace
93.7%
—
—
94.0%
—
—
94.0%
—
—
Table 1: Ridge Open-Set TAR@FAR= 10−4 (%) Across k TAR@FAR= 10−4 for each model over varied k, dataset, and projection method. ISP-W refers to the ISP projection being calculated on the same dataset, and ISP-X refers to the ISP projection being calculated from the other dataset. Results are averaged over 5 different seeds. FR models are bolded ; ISP is not applied to FR encoders (— indicates not applicable).
Figure 2: Template Inversion Verification Rate (%). Heatmap of verification rates across four attack families (DiffMI, Bob, Vec2Face, ALSUV) and five encoders. The FR encoder (ArcFace, top row) achieves consistently high rates (67–100%), while non-FR encoders (CLIP, DINOv2, DINOv3, SSCD) remain near zero across all methods. All rates averaged over ArcFace ( τ=0.1051 ) and AdaFace ( τ=0.1111 ) verifiers.
Figure 3: FCR Attribution: RAW vs ISP Overlay. (A) FII comparison: ISP reduces face-driven similarity. (B) Censoring comparison: context dominance maintained. (C) CPI delta: dramatic shift from context-only (1.0) to balanced (0.35-0.41). All images were cropped to a FCR of 0.33, with ± 2% equal-area tolerance.
Model
k -NN Top-1
Linear Probe Top-1
ISP
LEACE
ISP
LEACE
DINOv2
100.1%
100.0%
99.2%
99.3%
DINOv3
97.3%
97.4%
93.5%
93.4%
CLIP
98.3%
98.5%
100.7%
105.6%
SSCD
85.4%
85.7%
83.3%
82.6%
Table 2: ImageNet utility (normalized; 100 = raw baseline). Top-1 accuracy (percentage of raw, unprojected baseline) for k -NN and linear probe after projection.
Figure 4: Template Inversion Gallery. Target vs. DiffMI reconstruction across five encoders. ArcFace reconstructions verify at high rates; non-FR encoders do not. Protocol: DiffMI white-box, 1,000-latent pool, APGD 100 iterations, N=200 CelebA-20 targets.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Test IDs
Mated
Impostor
Min. FAR
CelebA-20
80
15,200
∼6,400
1.6×10−4
VGGFace2-20
80
15,200
∼16,000
6.3×10−5
LFW-20
13
78
1,326
7.5×10−4
Appendix
Table 3: Test set pair counts. Min. FAR shows the minimum measurable FAR ( 1/nimpostor ).
RAW
ISP-W
Model
k =1
k =4
k =16
k =1
k =4
k =16
DINOv2
19.5
6.4
9.5
0.0
0.0
0.0
DINOv3
11.5
0.8
6.7
0.0
10.3
2.1
CLIP
55.6
40.8
9.0
10.8
15.4
0.0
SSCD
2.3
14.1
0.0
2.1
0.0
0.5
ArcFace
87.4
82.1
73.9
—
—
—
Appendix
Table 4: LFW-20 Ridge Open-Set: partial AUC (%) at FAR ∈[0,10−3] (13 test IDs). ISP-W is not applied to FR models.
Dataset
Model
r=0
r=64
r=96
r=128
r=192
CelebA-20
DINOv2
4.1
3.7
2.9
1.3
1.5
DINOv3
6.2
2.3
1.4
1.0
1.5
SSCD
2.9
1.4
1.5
0.6
1.0
CLIP
7.3
6.6
6.7
5.7
3.5
VGGFace2-20
DINOv2
3.1
3.5
3.1
2.7
2.2
DINOv3
2.9
2.5
1.0
2.2
1.5
Appendix
Table 5: Rank sweep. TAR@FAR =10−4 (%) under ISP-W (Ridge, m=320 , k=16 ). r=0 is the raw (unprojected) baseline.
Model
m=320
m=640
m=1280
m=2000
DINOv2
1.5
1.9
2.9
2.7
DINOv3
1.5
5.4
4.0
3.9
SSCD
1.0
1.3
1.7
1.7
CLIP
3.5
6.7
6.7
1.5
Appendix
Table 6: Identity-count sweep. TAR@FAR =10−4 (%) as the ISP fit pool m scales (ISP-Large, CelebA-20, Ridge, r=192 , k=16 ). Fit identities are disjoint from test identities.
Bob
Vec2Face
ALSUV
Encoder
RAW
+ISP
RAW
+ISP
RAW
+ISP
CLIP
6.0
6.5
5.0
5.0
7.0
2.0
DINOv2
7.0
8.5
4.0
5.0
1.0
2.0
DINOv3
8.5
7.5
5.0
5.0
1.0
0.0
SSCD
9.0
5.5
3.0
3.0
1.0
0.0
Appendix
Table 7: Template inversion verification rates (%), RAW vs. +ISP. Non-FR encoders only (ISP not applicable to FR encoders). N=200 per encoder; ALSUV N=100 . Values averaged over ArcFace ( τ=0.1051 ) and AdaFace ( τ=0.1111 ) verifiers.
Type
Model
Pre-ISP
Post-ISP
k=1
k=4
k=16
k=1
k=4
k=16
Non-FR
DINOv2
1.4
2.2
3.8
0.0
0.8
0.4
DINOv3
2.2
3.4
5.9
0.5
0.3
0.9
SSCD
2.4
3.9
5.8
0.8
1.1
0.7
CLIP
18.3
22.9
26.9
1.6
1.8
0.5
FR
ArcFace
91.9
90.9
92.8
—
Appendix
Table 8: Projection-only MLP verifier (CelebA-20). TAR@FAR =10−4 (%). Pre-ISP uses raw embeddings; Post-ISP applies within-dataset ISP-W. FR models are shown pre-ISP only (ISP not evaluated on FR encoders).
Type
Model
Pre-ISP
Post-ISP
k=1
k=4
k=16
k=1
k=4
k=16
Non-FR
DINOv2
0.2
0.4
0.9
0.0
0.0
0.0
DINOv3
0.2
0.7
0.9
0.0
0.0
0.0
SSCD
0.0
0.0
0.0
0.0
0.0
0.0
CLIP
1.3
2.4
2.3
0.0
0.0
0.0
FR
ArcFace
89.3
89.3
89.3
—
Appendix
Table 9: Projection-only MLP verifier (VGGFace2-20). TAR@FAR =10−4 (%). Pre-ISP uses raw embeddings; Post-ISP applies within-dataset ISP-W. FR models are shown pre-ISP only (ISP not evaluated on FR encoders).
Metric
ISP
LEACE
Recall@1
95.0%
95.1%
Recall@5
96.5%
96.4%
Recall@10
96.0%
96.1%
Appendix
Table 10: DISC2021 copy-detection (SSCD). Percentage of Recall@ k retained after projection, relative to unprojected embeddings.
System
Direction
Accept
Flip
Joint
DINOv2
F→M
96.8%
14.6%
14.0%
+ ISP ( r=192 )
F→M
96.2%
30.2%
28.8%
DINOv2
M→F
98.4%
14.6%
14.4%
+ ISP ( r=192 )
M→F
98.2%
26.6%
26.2%
DINOv3
F→M
99.6%
13.0%
12.8%
+ ISP ( r=192 )
F→M
100.0%
17.8%
17.8%
Appendix
Table 11: Attribute-swap identity inference at k=128 . Joint = fraction of victims where identity is preserved (sim ≥τFAR=10−3 ) and gender is flipped. FR models allow highly reliable joint manipulation; non-FR encoders do not. 500 victims per direction on CelebA-20.