Benchmarking CLIP for Zero-Shot Face and Periocular Gender Estimation

Fernando Alonso-Fernandez; Kevin Hernandez-Diaz; Jose Maria Buades; Josef Bigun

cs.CVOct 5, 2026

Benchmarking CLIP for Zero-Shot Face and Periocular Gender Estimation

Authors: Fernando Alonso-Fernandez, Kevin Hernandez-Diaz, Jose Maria Buades, Josef Bigun

Organizations: School of Information Technology, Halmstad University, Sweden · Computer Graphics and Vision and AI Group, University of Balearic Islands, Spain

Abstract

We investigate CLIP for zero-shot gender estimation from full-face and periocular images. Three CLIP backbones are evaluated on 11,299 frontal images from Adience using image-text similarity with male/female prompts, achieving 95.54% full-face accuracy without task-specific training. For periocular, zero-shot predictions are strongly biased towards males, primarily due to a misaligned decision boundary. Threshold alignment substantially reduces this bias, reaching 85.29% accuracy. Linear SVMs trained on CLIP features provide only marginal gains, with a best periocular accuracy of 86.17%, approximately 2.8% above previous Adience results in the literature. Nevertheless, the gap with full-face performance confirms the greater difficulty of periocular gender estimation

Figures & tables

Fig. 1: Top left: CLIP model architecture (from [ 1 ] ). Top-right and bottom-left: our adaptation for face/periocular gender estimation. Bottom right: images from the Adience database (from [ 2 ] ).

Image encoder						Text encoder
	Input	Input		Para-	Vector		Para-	Vector
Model	size	patches	Layers	meters	Size	Layers	meters	size
ViT-B/16	224 $\times$ 224	16 $\times$ 16	107	86.2M	512	104	63.4M	512
ViT-L/14	224 $\times$ 224	14 $\times$ 14	203	304M	768	104	123.7M	768
ResNet-50	384 $\times$ 384	-	437	167.4M	768	104	123.7M	768

TABLE I: CLIP models evaluated in this paper.

#	Face	Periocular
1	a face photograph of a man/woman	a photograph of a male/female eye
2	a face image of a man/woman	an image of a male/female eye
3	a photograph of a man/woman	an eye image of a man/woman
4	a photograph of a male/female person	an ocular image of a man/woman
5	a photograph of a male/female human face	an ocular image of a male/female eye
6	a portrait photo of a man/woman	a photograph of a male/female person’s eye

TABLE II: Prompts employed for face/periocular gender classification.

Panel of Figure 4 — Fig. 2: Zero-shot face/periocular gender estimation accuracy per prompt on the Adience database for different CLIP backbones. The dots indicate the mean accuracy over the five cross-validation test folds, while error bars represent one standard deviation.

Fig. 3: Zero-shot face/periocular gender estimation accuracy on the Adience database for different CLIP backbones and downsampling values (mean accuracy over the five cross-validation test folds). Results correspond to the average encoding of all prompts.

Fig. 4: Histograms of zero-shot face/periocular scores on the test folds of the Adience database. Results correspond to the average encoding of all prompts.

Panel of Figure 7 — Fig. 5: Face/periocular gender estimation accuracy per prompt after embeddings alignment on the Adience database for different CLIP backbones. The dots indicate the mean accuracy over the five cross-validation test folds, while error bars represent one standard deviation.

	clipViTB16
	FACE			PERIOCULAR
Features	Male	Female	Both	Male	Female	Both
sim (zero shot)	95.11 $\pm$ 1.19	93.1 $\pm$ 2.6	94.06 $\pm$ 1.32	97.7 $\pm$ 0.87	49.76 $\pm$ 1.21	72.36 $\pm$ 1.61
sim (aligned)	93.76 $\pm$ 2.53	93.99 $\pm$ 2.63	93.95 $\pm$ 1.28	81.59 $\pm$ 6.96	77.75 $\pm$ 2.02	79.66 $\pm$ 3.48
sim + SVM	94.37 $\pm$ 1.87	94 $\pm$ 2	94.21 $\pm$ 1.37	79.55 $\pm$ 5.59	80.12 $\pm$ 2.36	79.9 $\pm$ 3.35
I + SVM	94.13 $\pm$ 1.84	93.83 $\pm$ 2.17	94.01 $\pm$ 1.41	76.67 $\pm$ 4.55	82.69 $\pm$ 2.91	79.9 $\pm$ 3.21

TABLE III: Comparative summary of face/periocular gender classification on the Adience database with the different methods evaluated in this paper. Bold indicates the best accuracy per column.

FACE
Ref	Features	Male	Female	Both
this paper:	clipViTB16	95.11 $\pm$ 1.19	93.1 $\pm$ 2.6	94.06 $\pm$ 1.32
sim zero shot	clipViTL14	97.55 $\pm$ 1	93.71 $\pm$ 2.2	95.54 $\pm$ 1.52
	clipR50	94.85 $\pm$ 2.01	89.71 $\pm$ 2.6	92.17 $\pm$ 1.28
this paper:	clipViTB16	93.76 $\pm$ 2.53	93.99 $\pm$ 2.63	93.95 $\pm$ 1.28
sim aligned	clipViTL14	95.58 $\pm$ 2.05	96.08 $\pm$ 1.48	95.87 $\pm$ 1.58

TABLE IV: Summary of the best reported accuracy of the experiments of this paper. The SVM combinations shown are those with the best overall accuracy in Table III . The table also includes results of recent works using the Adience database. Different works may not employ the same number of images per fold, so results may not be completely comparable.

Explore similar work

A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP
Jun 29, 2026Hodaya Krakover, Meir Yossef Levi, Eyal Gofer +1Unsupervised Detection
Technion - Israel Institute of Technology, Haifa, Israel
When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP
Sep 1, 2026Shota Sato, Hajime Kiyama, Tosho Hirasawa +1Modality Gap Contrastive Language-Image Pre-Training Model
Hitotsubashi University
Transductive Zero-Shot Audio Classification with Audio-Language Models
Jun 15, 2026Jingwen Zhou, Mingzhe WangZero-Shot Transductive Learning
Xidian University, Xi’an, China

Jun 29, 2026, cs.CVJ/K move · Enter open · S save

A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

Hodaya Krakover, Meir Yossef Levi, Eyal Gofer, Guy Gilboa

Technion - Israel Institute of Technology, Haifa, Israel

Unsupervised Detection

Open paper AskPDF arXiv

Adversarial attacks pose a challenge to the reliability of deep learning models, motivating effective detection methods. Existing techniques often rely on attack-specific assumptions, access to adversarial samples, or knowledge of the underlying classifier (white-box). We propose

A^4D

Attack- and Architecture-Agnostic Adversarial Detector, a completely black-box, zero-shot adversarial attack detection framework that utilizes prompt-based similarity scores derived from CLIP. To the best of our knowledge this is the first attempt to utilize CLIP for such a task. The method is based on two key observations: (i) CLIP is sensitive even to small imperceptible non-semantic perturbations; (ii) The shift in CLIP embedding space is not arbitrary and can be used as a robust attack indicator. Experiments across multiple attacks, datasets and classifiers validate that

A^4D

achieves SOTA detection results in the attack-agnostic and classifier-agnostic setting.

Abstract

Explore similar work

A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP

Transductive Zero-Shot Audio Classification with Audio-Language Models