Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience
Authors: Jacob Piland, Byron Dowling, Christopher Sweet, Adam Czajka
Organizations: Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, USA · Center for Research Computing, University of Notre Dame, Notre Dame, IN 46556, USA
Iris presentation attack detection (PAD) is critical for secure biometric deployments, yet developing specialized models faces significant practical barriers: collecting data representing future unknown attacks is impossible, and collecting diverse-enough data, yet still limited in terms of its predictive power, is expensive. Additionally, sharing biometric data raises privacy concerns. Due to rapid emergence of new attack vectors demanding adaptable solutions, we thus investigate in this paper whether general-purpose multimodal large language models (MLLMs) can perform iris PAD when augmented with human expert knowledge, operating under strict privacy constraints that prohibit sending biometric data to public cloud MLLM services. Through analysis of vision encoder embeddings applied to our dataset, we demonstrate that pre-trained vision transformers in MLLMs inherently cluster many iris attack types despite never being explicitly trained for this task. However, where clustering shows overlap between attack classes, we find that structured prompts incorporating human salience (verbal descriptions from subjects identifying attack indicators) enable these models to resolve ambiguities. Testing on an IRB-restricted dataset of 224 iris images spanning seven attack types, using only university-approved services (Gemini 2.5 Pro) or locally-hosted models (e.g., Llama 3.2-Vision), we show that Gemini with expert-informed prompts outperforms both a specialized convolutional neural networks (CNN)-based baseline and human examiners, while the locally-deployable Llama achieves near-human performance. Our results establish that MLLMs deployable within institutional privacy constraints offer a viable path for iris PAD.
Figures & tables
Figure 1: Experimental pipeline and paper contributions. We start with the generation of synthetic MESH salience from human annotations and corresponding image dataset. This salience, along with the control and human salience, is then combined with our novel prompts for testing. Results are obtained for both models to which we have ethical access and compared with a human and salience-guided CNN baseline.
Figure 2: Uniform Manifold Approximation and Projection (UMAP) visualization of iris samples encoded by (left) SigLIP vision-only embeddings and (right) SigLIP + Gemma multimodal embeddings using a simple binary prompt asking whether the iris is “real and healthy” or “synthetic/unhealthy.” Despite never being trained specifically for iris PAD, SigLIP alone achieves partial separation of attack types. However, adding even minimal semantic guidance through Gemma shows much improved visual separation between live vs. spoof discrimination, with clearer cluster boundaries and reduced overlap between classes. This visual separation motivates our investigation into whether general-purpose MLLMs can address specialized biometric security tasks through appropriate prompting.
Figure 3: From left to right (with third-party dataset sources, where appropriate): live iris (with no abnormalities) [ 31 ] , StyleGAN2-generated sample, StyleGAN3-generated sample, iris wearing textured contact lens, then printed and re-captured in near infrared light [ 31 ] , synthetic sample generated by a non deep learning-based algorithm [ 51 ] , diseased eye [ 58 ] , glass prosthesis, post-mortem sample [ 60 ] , iris printout [ 13 ] , iris wearing textured contact lens [ 15 ] , and artificial eye [ 30 ] .
Figure 4: Authentic (left) and synthetically generated by a StyleGAN2 model (right) iris images along with the expert and non-expert descriptions (“Human Text Annotations” in Fig. 1 ).
Prompt
In-Context
Salience
Type
Learning
Type
Short
None
Short + Human
✓
Human description
Short + Llama
✓
Llama MESH
Short + Gemini
✓
Gemini MESH
Long
None
Table 1: Prompt configurations used in experiments with each of the models considered in this work (Gemini and Llama).
Figure 5: Learning curve analysis of the eight Gemini experiments showing the MSE scores converge before even 25 samples from each attack type.
Attack type
Number of samples
Live iris
30
Artificial
14
Contacts+print
30
Diseased
30
Post-mortem
30
Printout
30
Table 2: Attack types and number of samples.
Model
Prompt variant
MSE
Human Subjects
0.062
Baseline CNN
0.345±0.041
Short
0.416
Gemini
Short + Human
0.183
Short + Llama
0.273
Short + Gemini
0.118
Table 3: MSE scores for Gemini and Llama based on the prompt type used. The best prompt variant of the short and long prompts is bolded for each model.
Figure 6: Histograms of the test set iris PAD classification confidence values generated by the two MLLMs. We see that Llama is more capable of expressing uncertainty.
Face morphing attacks threaten biometric verification, yet most morphing attack detection (MAD) systems require task-specific training and generalize poorly to unseen attack types. Meanwhile, open-source multimodal large language models (MLLMs) have demonstrated strong visual-linguistic reasoning, but their potential in biometric forensics remains underexplored. In this paper, we present the first systematic zero-shot evaluation of open-source MLLMs for single-image MAD, using publicly available weights and a standardized, reproducible protocol. Across diverse morphing techniques, many MLLMs show non-trivial discriminative ability without any fine-tuning or domain adaptation, and LLaVA1.6-Mistral-7B achieves state-of-the-art performance, surpassing highly competitive task-specific MAD baselines by at least 23% in terms of equal error rate (EER). The results indicate that multimodal pretraining can implicitly encode fine-grained facial inconsistencies indicative of morphing artifacts, enabling zero-shot forensic sensitivity. Our findings position open-source MLLMs as reproducible, interpretable, and competitive foundations for biometric security and forensic image analysis. This emergent capability also highlights new opportunities to develop state-of-the-art MAD systems through targeted fine-tuning or lightweight adaptation, further improving accuracy and efficiency while preserving interpretability. To support future research, all code and evaluation protocols will be released upon publication.
Marija Ivanovska, Vitomir Štruc
Faculty of Electrical Engineering, University of Ljubljana, Tržaška cesta 25, Ljubljana, 1000, Slovenia
Face recognition systems are increasingly deployed in security-critical applications, yet they remain vulnerable to presentation and morph attacks. Presentation attack detection (PAD) and morphing attack detection (MAD) are therefore essential components of trustworthy face biometrics. Despite advancements in PAD and MAD methods, existing detectors suffer from limited generalization and degrade in cross-dataset evaluation. In this paper, we systematically investigate whether general-purpose foundation models (FMs) and multimodal large language models (MLLMs) encode PAD-relevant and MAD-relevant information, and how such models can best be deployed for both tasks. We study five approaches with increasing access to the internal information of the model: (i) zero-shot prompting of off-the-shelf MLLMs; (ii) training a shallow model on the next-token logit probabilities at the output of the MLLM; (iii) parameter-efficient fine-tuning on task-specific question-answer data, yielding two specialized MLLMs, called PADLLM and MADLLM, which additionally provide textual reasoning for their decisions; (iv) linear probing of frozen vision encoders; and (v) fine-tuning of vision encoders of FMs and MLLMs. We benchmark 16 open-weight MLLMs and 30 vision encoder backbones on four PAD datasets (MSU-MFSD, CASIA-FASD, Replay-Attack, and OULU-NPU) and four MAD datasets (FFHQ, FRGC, FRLL, and FERET). Our experiments show that FMs and MLLMs can achieve significant performance for PAD and MAD. In addition, the fine-tuned models achieve state-of-the-art detection performance in cross-dataset evaluation, indicating that general-purpose pretrained representations carry substantial attack-relevant information. Source code of all our experiments will be publicly released.
Hatef Otroshi Shahreza, Asif Hussain Khan, Peter Lorenz +2
Idiap Research Institute, Switzerland · Université de Lausanne, Switzerland
Iris recognition systems are vulnerable to presentation attacks (PAs), including textured contact lenses, print replays, and synthetic irises. Existing PAD methods rely on supervised learning with attack data, limiting generalization to unseen attacks and new sensors. We propose Diff-IrisPAD, an unsupervised framework based on Brownian Bridge Diffusion that models bona fide iris distributions without using attack samples. The method reconstructs clean iris textures from degraded inputs and detects attacks via reconstruction inconsistency using ViT scoring and pixel-level measures. Experiments on a large-scale multi-sensor dataset (125K+ images, NIR and VIS, 8 attack types) show that Diff-IrisPAD achieves an overall ACER of 30.52, significantly outperforming diffusion-based baselines while demonstrating strong generalization to unseen attacks. These results highlight the effectiveness of diffusion-based bona fide modeling for robust and scalable iris PAD.