cs.CVNov 16, 2026

Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs

Authors: Alexandre Fournier-Montgieux, Akram Elbouanani, Aboubacar Tuo, Adrian Popescu, Hervé Le Borgne, Bertrand Luvison

Organizations: Université Paris-Saclay, CEA, LIST, F-91120, Palaiseau, France

Abstract

We audit face-related bias in VLMs using BALPORTRAITS, a new balanced dataset of 9,000 fictional portraits across ethnicity, gender, and age segments. Sixteen VLMs estimate portrait fitness scores for occupations, brands, activities, and banking or insurance products within an advertising-casting framework designed to avoid direct inference of protected attributes. Item-level estimations are aggregated with scenario-specific prestige scores to compare rankings across demographic groups. While the magnitude and direction of these biases vary across scenarios, model families, and demographic dimensions, with age, ethnicity, and gender contributing differently depending on the task, we find demographic dispersion across all evaluated models, with a majority of pairwise segment differences reaching statistical significance. This study introduces a controlled protocol to assess implicit face-related associations and demonstrates that single-scenario or single-attribute audits are insufficient.

Figures & tables

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making

    Sep 1, 2026Vahid Reza Khazaie, Ahmed Y. Radwan, Shaina RazaVLM EvaluationDemographic Bias in VLMs

  2. Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

    Mar 26, 2026Ünsal Öztürk, Hatef Otroshi Shahreza, Sébastien MarcelAlgorithmic FairnessMultimodal Large Language Models