cs.CVMar 13, 2026

A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks

Authors: Tangzheng Lian, Guanyu Hu, Yijing Ren, Dimitrios Kollias, Oya Celiktutan

Organizations: King’s College London · Queen Mary University of London

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While Vision-Language Models (VLMs) have achieved remarkable performance across diverse downstream tasks, recent studies have shown that they can inherit social biases from the training data and further propagate them into downstream applications. To address this issue, various debiasing approaches have been proposed, yet most of them aim to improve fairness without having a theoretical guarantee that the utility of the model is preserved. In this paper, we introduce a debiasing method that yields a closed-form solution in the cross-modal space, achieving Pareto-optimal fairness with bounded utility losses. Our method is training-free, requires no annotated data, and can jointly debias both visual and textual modalities across downstream tasks. Extensive experiments show that our method outperforms existing methods in debiasing VLMs across diverse fairness metrics and datasets for both group and intersectional fairness in downstream tasks such as zero-shot image classification, text-to-image retrieval, and text-to-image generation while preserving task performance.

Figures & tables

MethodsSFID [ 21 ]DeAR [ 41 ]CLIP-clip [ 47 ]Fairer CLIP [ 7 ]PRISM [ 33 ]PRISM-mini [ 33 ]SANER [ 14 ]Biased Prompt [ 5 ]Robo Shot [ 1 ]Prompt Array [ 2 ]JointV-L [ 51 ]Ours
Training-Free✘✘✓✘✘✓✘✓✓✘✘✓
Data-Free✘✘✘✘✓✓✓✓✓✘✘✓
Debiasing Both Modalities✓✘✓✓✓✓✘✘✓✓✓✓
Bounded Utility Loss✘✘✘✘✘✘✘✘✘✘✘✓
Zero-Shot Image Classification✓✘✘✓✓✓✘✓✓✘✘✓
Text-to-Image Retrieval✓✓✓✘✘✘✓✓✘✓✓✓
Table 1: Overview of recent debiasing methods in VLMs.
Figure 1: (a) Previous methods such as PRISM-mini [ 33 ] and Orth-Proj [ 5 ] debias through an orthogonal projection to the subspace S\mathcal{S} spanned by the embeddings of group-explicit prompts ( group prototypes ) such as “a photo of a female/male doctor.” However, the subspace S\mathcal{S} contains not only attribute-related information but also essential semantic content (e.g., “doctor”) that we wish to preserve. (b) We instead debias through an orthogonal projection to the attribute subspace A\mathcal{A} so that u⃗\vec{u} can effectively removes bias while maintaining content-related semantics.
Figure 2: Illustration of the group prototype construction process.
Figure 3: Illustration of the optimal solution space of u⃗\vec{u} .
DatasetsCelebAFACET
Evaluation MetricF1 ↑\uparrowΔEOAvg\Delta_{\text{EO}}^{\text{Avg}} (G ×\times A) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G ×\times A) ↓\downarrowMacro F1 ↑\uparrowΔEOAvg\Delta_{\text{EO}}^{\text{Avg}} (G) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G) ↓\downarrow
Baseline CLIP (ViT-L/14)54.0 ±\pm 0.525.1 ±\pm 0.345.0 ±\pm 0.370.8 ±\pm 0.28.9 ±\pm 0.249.8 ±\pm 0.3
I&T SFID [ 21 ]52.8 ±\pm 0.523.9 ±\pm 0.141.6 ±\pm 0.269.2 ±\pm 0.39.3 ±\pm 0.449.8 ±\pm 0.2
I&T FairerCLIP [ 7 ][53.1] ±\pm 0.224.0 ±\pm 0.141.4 ±\pm 0.3[69.8] ±\pm 0.29.2 ±\pm 0.250.1 ±\pm 0.4
I&T PRISM [ 33 ]52.9 ±\pm 0.423.9 ±\pm 0.540.6 ±\pm 0.169.0 ±\pm 0.2[8.1] ±\pm 0.348.3 ±\pm 0.3
I&T PRISM-mini [ 33 ]50.4 ±\pm 0.124.2 ±\pm 0.640.8 ±\pm 0.369.2 ±\pm 0.48.6 ±\pm 0.249.2 ±\pm 0.5
Table 2: Experimental results for zero-shot image classification. (G) denotes perceived gender as the sensitive attribute, and (G ×\times A) denotes the intersection of perceived gender and age as the sensitive attribute. ↑\uparrow indicates that higher values are better, while ↓\downarrow indicates that lower values are better. Methods that require or do not require attribute-annotated data are marked with and , respectively. Methods that involve or do not involve training are marked with and , respectively. Methods that debias only the text encoder are denoted by T , while methods that debias both modalities are marked with I&T . The results of the debiasing methods with the best means are highlighted in bold with brackets, while the second-best are shown in brackets only. All values reported are scaled by 100 (percentages).
DatasetsCOCO2017Flickr30K
Evaluation MetricR@5 ↑\uparrowR@10 ↑\uparrowMS@1000 (G ×\times ST) ↓\downarrowR@5 ↑\uparrowR@10 ↑\uparrowMS@1000 (G) ↓\downarrow
Baseline CLIP (ViT-L/14)83.8 ±\pm 0.390.1 ±\pm 0.413.4 ±\pm 0.591.0 ±\pm 0.695.4 ±\pm 0.420.3 ±\pm 0.3
I&T SFID [ 21 ][77.4] ±\pm 0.486.6 ±\pm 0.213.2 ±\pm 0.386.8 ±\pm 0.390.7 ±\pm 0.513.6 ±\pm 0.5
I&T PromptArray [ 2 ]77.2 ±\pm 0.6[86.7] ±\pm 0.512.8 ±\pm 0.487.2 ±\pm 0.292.4 ±\pm 0.112.9 ±\pm 0.4
I&T FairerCLIP [ 7 ]76.8 ±\pm 0.385.4 ±\pm 0.410.2 ±\pm 0.6[87.9] ±\pm 0.5[92.5] ±\pm 0.612.2 ±\pm 0.5
I&T CLIP-clip [ 47 ]76.1 ±\pm 0.285.2 ±\pm 0.3[9.9] ±\pm 0.487.7 ±\pm 0.391.5 ±\pm 0.1[11.7] ±\pm 0.5
Table 3: Experimental results for text-to-image retrieval. The markers follow the same notation style as in Table 8 . (G ×\times ST) denotes the intersection of perceived gender and skin tone. All values reported are scaled by 100 (percentages).
Figure 4: Illustrative examples for o=“doctor"o=\text{``doctor"} . We randomly sample ten generated images for each method. Female-looking samples are marked in red and numbered. On the left, a more balanced ratio of female- and male-looking samples indicates lower bias, while on the right, fewer female-looking samples reflect better preservation of self-utility (see Appendix G for more examples with other occupations).
MethodsSP‾To\overline{\mathrm{SP}}_{\mathcal{T}_{o}} ↓\downarrowSP‾5\overline{\mathrm{SP}}_{5} ↓\downarrowCLIPscore\text{CLIP}_{\text{score}} (%) ↑\uparrowAccG\text{Acc}^{\text{G}} (%) ↑\uparrow
Baseline SD v2.147.958.632.875.4
Orth-Proj [ 5 ][39.6][28.4]19.753.4
Orth-Cali [ 5 ]41.529.220.756.6
SFID [ 21 ]41.129.4[22.6][67.2]
Ours[39.7][28.8][24.2][74.6]
Table 4: Experimental results for text-to-image generation. All values reported are scaled by 100 (percentages).
DatasetsFlickr30KCelebA
MetricsMS@1000 (G) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G ×\times A) ↓\downarrow
Baseline CLIP (ViT-L/14)20.3 ±\pm 0.345.0 ±\pm 0.3
debias with only p⃗a\vec{p}_{a}13.4 ±\pm 0.541.1 ±\pm 0.4
debias with only p⃗m\vec{p}_{m}14.1 ±\pm 0.441.8 ±\pm 0.5
debias with DeepSeek v3.2 [ 31 ][12.0] ±\pm 0.3[40.1] ±\pm 0.4
debias with Gemini2.5 Pro [ 6 ][11.8] ±\pm 0.6[40.4] ±\pm 0.2
Table 5: Experimental results of ablation studies and sensitivity analyses on different LLMs. p⃗a\vec{p}_{a} denotes the anchor embedding while p⃗m\vec{p}_{m} denotes the mean embedding across its variants.
Figure 5: An illustration of the prompts we use in an LLM to insert the group specification and generate the variants for each group gg .
Figure 6: An illustration of the LLM output when the input is group-specific/neutral. The sensitive attribute is gender, and the selected group is female/male.
Figure 7: An illustration of the LLM output when the input is neutral. The sensitive attribute is the intersection of skin tone and gender, and the selected group is dark-skinned female.
Figure 8: An illustration of neutral prompts used for fairness evaluation and group-explicit prompts used for utility evaluation in the text-to-image generation task.
DatasetsCelebAFACET
Evaluation MetricF1 ↑\uparrowΔEOAvg\Delta_{\text{EO}}^{\text{Avg}} (G ×\times A) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G ×\times A) ↓\downarrowMacro F1 ↑\uparrowΔEOAvg\Delta_{\text{EO}}^{\text{Avg}} (G) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G) ↓\downarrow
Baseline CLIP (ResNet50)60.7 ±\pm 0.515.3 ±\pm 0.330.0 ±\pm 0.459.2 ±\pm 0.29.3 ±\pm 0.648.9 ±\pm 0.1
I&T SFID56.4 ±\pm 0.214.6 ±\pm 0.524.9 ±\pm 0.3[50.8] ±\pm 0.69.4 ±\pm 0.447.2 ±\pm 0.2
I&T FairerCLIP[56.6] ±\pm 0.314.2 ±\pm 0.1[21.4] ±\pm 0.450.7 ±\pm 0.58.9 ±\pm 0.647.2 ±\pm 0.2
I&T PRISM56.3 ±\pm 0.114.7 ±\pm 0.4[21.8] ±\pm 0.350.5 ±\pm 0.6[8.4] ±\pm 0.547.6 ±\pm 0.2
I&T PRISM-mini56.0 ±\pm 0.613.7 ±\pm 0.324.9 ±\pm 0.149.2 ±\pm 0.48.9 ±\pm 0.546.6 ±\pm 0.2
Table 6: Experimental results of CLIP (ResNet50) on the zero-shot image classification task.
DatasetsCOCO2017Flickr30K
Evaluation MetricR@5 ↑\uparrowR@10 ↑\uparrowMS@1000 (G ×\times ST) ↓\downarrowR@5 ↑\uparrowR@10 ↑\uparrowMS@1000 (G) ↓\downarrow
Baseline CLIP (ResNet50)71.4 ±\pm 0.581.8 ±\pm 0.113.3 ±\pm 0.378.5 ±\pm 0.685.9 ±\pm 0.418.6 ±\pm 0.5
I&T SFID[66.6] ±\pm 0.276.3 ±\pm 0.312.7 ±\pm 0.474.9 ±\pm 0.679.6 ±\pm 0.215.2 ±\pm 0.3
I&T PromptArray66.5 ±\pm 0.5[76.5] ±\pm 0.412.2 ±\pm 0.174.9 ±\pm 0.3[81.6] ±\pm 0.315.4 ±\pm 0.5
I&T FairerCLIP65.8 ±\pm 0.674.8 ±\pm 0.311.8 ±\pm 0.575.2 ±\pm 0.2[81.6] ±\pm 0.416.0 ±\pm 0.3
I&T CLIP-clip65.6 ±\pm 0.174.9 ±\pm 0.5[10.0] ±\pm 0.6[75.3] ±\pm 0.380.8 ±\pm 0.4[13.2] ±\pm 0.5
Table 7: Experimental results of CLIP (ResNet50) on the text-to-image retrieval task.
DatasetsCelebAFACET
Evaluation MetricF1 ↑\uparrowΔEOAvg\Delta_{\text{EO}}^{\text{Avg}} (G ×\times A) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G ×\times A) ↓\downarrowMacro F1 ↑\uparrowΔEOAvg\Delta_{\text{EO}}^{\text{Avg}} (G) ↓\downarrowΔEOMax\Delta_{\text{EO}}^{\text{Max}} (G) ↓\downarrow
Baseline BLIP46.9 ±\pm 0.614.7 ±\pm 0.429.2 ±\pm 0.668.4 ±\pm 0.38.8 ±\pm 0.548.0 ±\pm 0.4
I&T SFID[52.5] ±\pm 0.2[12.1] ±\pm 0.126.4 ±\pm 0.360.6 ±\pm 0.18.2 ±\pm 0.247.1 ±\pm 0.5
I&T FairerCLIP52.1 ±\pm 0.413.4 ±\pm 0.6[22.6] ±\pm 0.4[61.0] ±\pm 0.38.9 ±\pm 0.646.5 ±\pm 0.3
I&T PRISM[5] ±\pm 0.513.8 ±\pm 0.226.1 ±\pm 0.560.3 ±\pm 0.4[6.8] ±\pm 0.346.0 ±\pm 0.5
I&T PRISM-mini49.5 ±\pm 0.414.0 ±\pm 0.525.1 ±\pm 0.160.0 ±\pm 0.27.4 ±\pm 0.146.1 ±\pm 0.4
Table 8: Experimental results of BLIP on the zero-shot image classification task.
DatasetsCOCO2017Flickr30K
Evaluation MetricR@5 ↑\uparrowR@10 ↑\uparrowMS@1000 (G ×\times ST) ↓\downarrowR@5 ↑\uparrowR@10 ↑\uparrowMS@1000 (G) ↓\downarrow
Baseline BLIP93.9 ±\pm 0.597.2 ±\pm 0.115.1 ±\pm 0.693.9 ±\pm 0.496.8 ±\pm 0.218.3 ±\pm 0.6
I&T SFID[90.6] ±\pm 0.395.9 ±\pm 0.412.5 ±\pm 0.186.1 ±\pm 0.689.9 ±\pm 0.513.5 ±\pm 0.1
I&T PromptArray90.4 ±\pm 0.2[96.0] ±\pm 0.312.9 ±\pm 0.486.4 ±\pm 0.391.5 ±\pm 0.412.8 ±\pm 0.5
I&T FairerCLIP90.0 ±\pm 0.694.7 ±\pm 0.213.3 ±\pm 0.3[91.0] ±\pm 0.2[91.7] ±\pm 0.312.0 ±\pm 0.4
I&T CLIP-clip90.3 ±\pm 0.494.5 ±\pm 0.6[11.2] ±\pm 0.5[91.0] ±\pm 0.190.8 ±\pm 0.1[8.1] ±\pm 0.2
Table 9: Experimental results of BLIP on the text-to-image retrieval task.
Figure 9: Illustrative examples for o=“designer"o=\text{``designer"} . We randomly sample ten generated images for each method. Male-looking samples are marked in red and numbered. On the left, a more balanced ratio of female- and male-looking samples indicates lower bias, while on the right, fewer male-looking samples reflect better preservation of self-utility.
Figure 10: Illustrative examples for o=“computer programmer"o=\text{``computer programmer"} . We randomly sample ten generated images for each method. Female-looking samples are marked in red and numbered. On the left, a more balanced ratio of female- and male-looking samples indicates lower bias, while on the right, fewer female-looking samples reflect better preservation of self-utility.
Figure 11: Illustrative examples for o=“civil engineer"o=\text{``civil engineer"} . We randomly sample ten generated images for each method. Female-looking samples are marked in red and numbered. On the left, a more balanced ratio of female- and male-looking samples indicates lower bias, while on the right, fewer female-looking samples reflect better preservation of self-utility.
Figure 12: Illustrative examples for o=“CEO"o=\text{``CEO"} . We randomly sample ten generated images for each method. Female-looking samples are marked in red and numbered. On the left, a more balanced ratio of female- and male-looking samples indicates lower bias, while on the right, fewer female-looking samples reflect better preservation of self-utility.
More in this topic

Cultural Bias in VLMs

See all 14 papers

Explore similar work

CardsList

  1. Interpretable Debiasing of Vision-Language Models for Social Fairness

    Feb 27, 2026Na Min An, Yoonna Jang, Yusuke Hirota +3Cultural Bias in VLMsVLM Interpretability

    KAIST AI · University of Copenhagen · NVIDIA

  2. Selective Test-Time Debiasing for CLIP via Reward Gating

    Jul 1, 2026Jaeho Han, Jisoo Yang, Hyeondong Woo +3Debiased MLVLM Adaptation

    Department of Artificial Intelligence, Chung-Ang University

  3. SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

    Mar 19, 2026Quentin Guimard, Federico Bartsch, Simone Caldarella +3Cultural Bias in VLMsVision-Language Models

    University of Trento · Toyota Motor Europe · Fondazione Bruno Kessler