FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

Mahesh Bhosale; Abdul Wasi; Shantam Srivastava; Shifa Latif; Tianyu Luan; Mingchen Gao; David Doermann; Xuan Gong

cs.CVMar 27, 2026

FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

Authors: Mahesh Bhosale, Abdul Wasi, Shantam Srivastava, Shifa Latif, Tianyu Luan, Mingchen Gao, David Doermann, Xuan Gong

Organizations: University at Buffalo · University of Kashmir · Accenture · Harvard Medical School

Abstract

While powerful in image-conditioned generation, multimodal large language models (MLLMs) can display uneven performance across demographic groups, highlighting fairness risks. In safety-critical clinical settings, such disparities risk producing unequal diagnostic narratives and eroding trust in AI-assisted decision-making. While fairness has been studied extensively in vision-only and language-only models, its impact on MLLMs remains largely underexplored. To address these biases, we introduce FairLLaVA, a parameter-efficient fine-tuning method that mitigates group disparities in visual instruction tuning without compromising overall performance. By minimizing the mutual information between target attributes, FairLLaVA regularizes the model's representations to be demographic-invariant. The method can be incorporated as a lightweight plug-in, maintaining efficiency with low-rank adapter fine-tuning, and provides an architecture-agnostic approach to fair visual instruction following. Extensive experiments on large-scale chest radiology report generation and dermoscopy visual question answering benchmarks show that FairLLaVA consistently reduces inter-group disparities while improving both equity-scaled clinical performance and natural language generation quality across diverse medical imaging modalities. Code can be accessed at https://github.com/bhosalems/FairLLaVA.

Figures & tables

Figure 1: FairLLaVA reduces performance disparities . LLaVA hidden states contain demographic shortcuts (non-zero Mutual Information (MI) between hidden states and demographic attributes) that lead to lower performance for “Female”. FairLLaVA minimizes this MI promoting demographic-invariant representation learning, therefore reducing the performance gap.

Figure 2: FairLLaVA Overview. Stage 1 : We finetune multi-modal projector $\psi$ to align the image embeddings with Language Model (LM) by optimizing standard LM CE loss $\mathcal{L}_{LM}$ . Image encoder and LM are frozen. Stage 2 : We learn attribute-invariant representations by finetuning LoRA adapters $\theta$ on the LM’s Transformer decoder blocks while freezing the LM backbone and image encoder, pooled hidden states $h$ are fed to a mutual-information (MI) estimator with a variational demographic-attribute classifier (DAC) denoted as ${\phi}$ that predicts demography attribute from $h$ . No pretrained classifier is required: the DAC is trained simultaneously with cross-entropy $\mathcal{L}_{\text{DAC}}$ (Eq. ( 8 )), and during this step, gradients update only $\phi$ . We then minimize MI between demographic attribute and $h$ given by $\mathcal{L}_{\text{DIM}}$ (Eq. ( 9 )), computed between pooled states $h$ and attributes $\mathbf{a}$ , ${\phi}$ is frozen, and only $\theta$ and $\psi$ are updated. This way DAC exposes where leakage in $h$ for predicting $\mathbf{a}$ is coming from, $\mathcal{L}_{DIM}$ suppresses this leakage making learned features demography shortcut invariant.

Method	Race $\uparrow$				Age Group $\uparrow$				Gender $\uparrow$
Method	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN
LLaVA [ 28 ]	2.90	1.12	1.63	2.47	2.61	0.81	0.95	1.36	8.42	0.87	5.19	6.73
LLaVA-Rad [ 7 ]	5.29	2.14	4.14	3.97	8.28	3.51	1.42	2.00	28.06	12.53	9.24	10.88
MedGemma-4B [ 40 ]	2.36	0.95	1.85	5.05	3.17	1.16	1.51	3.03	14.18	1.55	7.27	7.51
MedGemma-27B [ 40 ]	2.01	1.72	2.82	3.92	3.98	1.17	1.55	3.35	17.80	1.82	6.64	8.92
Qwen2.5-7B [ 51 ]	7.52	1.47	2.78	5.59	6.16	1.37	1.58	3.27	16.92	1.73	8.28	12.27

Table 1: Equity-Scaled metrics on MIMIC-CXR computed as

M_{\text{all}}/(1+\Delta M)

for Race, Age, and Gender. (All) indicates all three demographic attributes are used for debiasing. Our joint debiasing (FairLLaVA–All) yields the state-of-the-art improvements, obtaining 7 of the 12 best ES-scores across the comprehensive semantic clinical evaluations including BLEU, RadGraph-F1, and GREEN. Best scores are in bold and second best underlined .

Method	Race $\uparrow$				Age Group $\uparrow$				Gender $\uparrow$
Method	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN
Reweighting-Race [ 26 ]	3.38	1.89	4.13	3.27	13.15	4.08	2.99	2.09	15.38	8.28	22.05	11.78
Resampling-Race [ 18 ]	12.53	3.81	5.00	3.40	14.09	4.08	2.81	2.15	20.62	8.97	20.33	14.26
FairLLaVA-Race	5.40	2.91	7.32	4.43	14.85	5.58	3.24	2.10	25.40	11.15	27.33	15.05
Reweighting-Age [ 26 ]	10.65	3.49	3.24	2.84	15.22	5.26	3.93	2.50	19.73	11.65	18.33	14.14
Resampling-Age [ 18 ]	7.43	2.80	6.26	3.29	12.52	3.92	3.31	2.18	22.28	7.68	14.09	10.91

Table 2: Equity-Scaled metrics on MIMIC-CXR when individual demographic attributes are de-biased. Our targeted variants achieve the top ES scores on their respective attributes and show beneficial spillover to others.

Method	Race $\uparrow$	Age Group $\uparrow$	Gender $\uparrow$
Chen et al. [ 8 ]	24.06	23.85	24.13
FairLLaVA	69.21	68.70	69.38

Table 3: Equity-scaled CheXpert-14 F1 (higher is better) on MIMIC-CXR compared with [ 8 ] .

Method	Age Group $\uparrow$				Gender $\uparrow$				Overall $\uparrow$
Method	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN
LLaVA [ 28 ]	1.46	0.60	1.22	1.03	6.24	1.14	3.05	4.11	10.54	1.36	4.71	9.14
LLaVA-Rad [ 7 ]	2.31	1.51	3.13	1.22	24.52	6.57	12.90	6.91	25.01	12.02	15.35	39.78
MedGemma-4B [ 40 ]	2.62	0.64	2.41	2.01	8.99	1.19	3.87	5.65	12.31	1.27	4.37	11.52
MedGemma-27B [ 40 ]	2.70	0.86	2.35	1.79	9.11	1.61	4.61	5.21	13.94	1.79	5.03	12.67
Qwen2.5-7B [ 51 ]	2.28	1.00	1.56	1.66	7.73	1.69	4.12	4.47	12.45	1.84	5.11	11.80

Table 4: Equity-Scaled and Overall metrics on the PadChest dataset. Both “Gender” and “Age” are considered in debiasing. FairLLaVA-All achieves consistently higher ES metrics across demographic attributes and also the best overall performance.

Method	Gender $\uparrow$			Age $\uparrow$
Method	ES-Acc	ES-BLEU4	ES-ROUGE-L	ES-Acc	ES-BLEU4	ES-ROUGE-L
LLaVA [ 44 ]	14.12	6.81	14.53	2.02	10.43	9.15
Reweighting-All [ 26 ]	11.52	6.25	12.89	1.69	2.97	6.61
Resampling-All [ 18 ]	11.83	6.24	10.37	2.19	6.55	7.03
FairLLaVA-All	19.56	7.06	16.33	2.63	12.17	9.07

Table 5: Equity-Scaled metrics on the HAM10000 dataset. Both “Gender” and “Age” are considered in debiasing. FairLLaVA-All achieves consistently higher ES metrics across demographic attributes.

Method	Race $\downarrow$		Age Group $\downarrow$		Gender $\downarrow$		Overall $\uparrow$
Method	BLEU-4	RadGraph-F1	BLEU-4	RadGraph-F1	BLEU-4	RadGraph-F1	BLEU-4	RadGraph-F1
FairLLaVA-first	3.40	4.42	2.53	8.6	1.03	1.09	13.62	29.71
FairLLaVA-last	4.48	3.90	2.4	3.90	1.34	0.97	13.19	29.69
FairLLaVA-mean	3.07	4.52	2.16	7.32	0.48	0.50	14.84	29.85
FairLLaVA-mid	0.61	3.50	1.01	5.60	0.44	0.47	14.01	28.52

Table 6: Ablation on pooling hidden states from FairLLaVA. FairLLaVA-mean pools first/middle/last hidden states. The middle layer attains a strong balance between maintaining performance and reducing gaps across attributes.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Race $\downarrow$		Age $\downarrow$		Gender $\downarrow$
Method	RG-F1	GREEN	RG-F1	GREEN	RG-F1	GREEN
FairLLaVA-Race	2.98	4.59	9.06	16.88	1.41	1.84
FairLLaVA-Age	4.12	6.37	8.17	14.88	1.61	3.03
FairLLaVA-Gender	2.89	6.01	9.29	16.04	1.18	1.81
LLaVA-Rad	3.59	6.30	10.38	17.28	1.17	3.54
FairLLaVA-All	3.26	4.78	7.33	12.89	1.08	2.74

Appendix

Table S1 : Cross-sectional fairness analysis. RG-F1 is an abbreviation of RadGraph-F1. To isolate the effect of debiasing each demographic attribute, subgroup gaps are computed while holding the remaining attributes fixed (e.g., comparing age groups within the same race–gender slice). The targeted-attribute variants cause most reduction in the gap as compared to other-attribute variants, as intended. FairLLaVA-All also holds strong under this analysis as compared to the strong baseline of LLaVA-Rad.

Method	Race $\downarrow$		Age $\downarrow$		Gender $\downarrow$
Method	RG-F1	GREEN	RG-F1	GREEN	RG-F1	GREEN
LLaVA-Rad	13.71	24.45	27.44	21.44	17.93	21.60
FairLLaVA	6.65	23.08	20.34	17.65	16.92	18.90

Appendix

Table S2 : Counterfactual fairness gaps. FairLLaVA also reduces the individual counterfactual fairness gaps on MIMIC-CXR dataset.

Figure S1 : Hyper Parameters Sensitivity (a) Varying the contribution of each attribute-specific MI term to the total loss on MIMIC-CXR leads to only minor changes, indicating stable overall performance across attributes. (b) Varying the contribution of language model loss $\mathcal{L}_{LM}$ leads to minor changes in overall performance.

$(\lambda_{r},\lambda_{a},\lambda_{g})$	Race $\uparrow$		Age Group $\uparrow$		Gender $\uparrow$
$(\lambda_{r},\lambda_{a},\lambda_{g})$	RG-F1	GREEN	RG-F1	GREEN	RG-F1	GREEN
$(0.2,0.6,0.2)$	2.77	3.22	3.85	3.34	18.98	12.92
$(0.6,0.2,0.2)$	5.62	3.60	3.42	1.83	22.45	12.23
$(0.2,0.2,0.6)$	5.30	3.18	3.24	1.85	24.74	13.77

Appendix

Table S3 : Effect of varying attribute-specific MI weights.

(\lambda_{r},\lambda_{a},\lambda_{g})

on equity-scaled performance.

Dataset	Sex $\uparrow$	Age $\uparrow$		Race $\uparrow$
Dataset	AUC	MAE	AgeGrp Acc	AUC
MIMIC-CXR	0.9549	6.9715	0.7265	0.8901
PadChest	0.9841	6.2339	0.8023	–

Appendix

Table S4 : Missing demographic attribute prediction on out of domain radiology datasets using TorchXRayVision [ 12 ] .

Figure S2 : 95% Confidence Intervals of ES metric on MIMIC-CXR with bootstrap resampling ( $n{=}1000$ )

Group	Finding	N	$p_{\mathrm{ref}}$	$p_{\mathrm{pred}}$	$\Delta$
Age Group
0-44	Pleural Effusion	87	0.851	0.874	0.023
44-65	Pleural Effusion	899	0.790	0.810	0.020
65+	Pleural Effusion	1014	0.764	0.824	0.060
0-44	Pneumothorax	87	0.736	0.736	0.000
Gender

Appendix

Table S5 : Prevalence preservation under FairLLaVA for subgroup-finding pairs with reference prevalence

p_{\mathrm{ref}}\geq 0.7

and subgroup size

N>50

. We report the reference prevalence

p_{\mathrm{ref}}

, generated prevalence

p_{\mathrm{pred}}

, and prevalence shift

\Delta=p_{\mathrm{pred}}-p_{\mathrm{ref}}

. High-prevalence findings are largely preserved across demographic groups, with generally small shifts in prevalence.

Figure S3 : Sub-Group performance of baselines across “Race”, “Age” and “Gender” subgroups on MIMIC-CXR dataset on GREEN and BLEU-4 metric. We observe that the high number of counts in the train dataset does not correlate with the increased performance. Please also check Fig. S5

Figure S4 : Sub-Group performance of baselines across “Age” and “Gender” subgroups on PadChest dataset on GREEN and BLEU-4 metric. We observe that the high number of counts in the train dataset does not correlate with the increased performance. Please also check Fig. S5

Figure S5 : Distribution of counts of demographic subgroups in MIMIC-CXR and PadChest dataset train splits. Some demographic group counts are highly imbalanced, (a) MIMIC-CXR and (b) PadChest. F1-RG is an abbreviation of RadGraph-F1.

Method	Race $\downarrow$				Age Group $\downarrow$				Gender $\downarrow$				Overall $\uparrow$
Method	BLEU-1	BLEU-4	RadGraph-F1	GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN
LLaVA-Rad	6.22	6.20	6.20	8.02	3.61	3.39	19.97	16.91	0.36	0.23	2.23	2.29	38.17	15.40	29.80	35.82
MedGemma-4B	6.21	0.69	4.70	2.96	4.37	0.39	5.98	5.61	0.20	0.04	0.45	1.67	17.02	1.61	10.54	20.02
MedGemma-27B	8.03	0.06	4.13	4.73	3.56	0.56	8.33	5.70	0.02	0.00	1.18	1.51	18.15	1.82	14.46	22.45
Qwen2.5-7B	1.43	0.22	2.66	1.88	1.97	0.31	5.44	3.92	0.08	0.04	0.23	0.32	18.29	1.80	10.18	16.098
DeepSeek-VL2	2.17	0.21	2.73	3.34	3.32	0.44	3.90	4.64	0.14	0.07	0.31	0.51	12.93	1.78	8.12	14.16

Appendix

Table S6 : Fairness Gaps (First three main columns across Race, Age, Gender) and Overall performance (last column) on MIMIC CXR dataset. Highlights tradeoff between Overall-Performance and Fairness-Gaps. Fairness gaps lower the better, Overall performance higher the better.

Method	Age Group $\downarrow$				Gender $\downarrow$				Overall $\uparrow$
Method	BLEU-1	BLEU-4	RadGraph-F1	GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN
LLaVA-Rad	9.81	6.96	3.91	31.52	0.02	0.83	0.19	4.76	25.01	12.02	15.35	39.78
MedGemma-4B	3.70	0.97	0.81	4.73	0.37	0.07	0.13	1.04	12.31	1.27	4.37	11.52
MedGemma-27B	4.16	1.07	1.14	6.06	0.53	0.11	0.09	1.43	13.94	1.79	5.03	12.67
Qwen2.5-7B	4.47	0.84	2.28	6.12	0.61	0.09	0.24	1.64	12.45	1.84	5.11	11.80
DeepSeek-VL2	3.92	0.96	1.84	4.18	0.47	0.08	0.19	0.79	11.77	1.91	4.20	10.44

Appendix

Table S7 : Fairness Gaps (First two main columns across Age, Gender) and Overall performance (last column) on PadChest dataset. Highlights tradeoff between Overall-Performance and Fairness-Gaps.

Figure S6 : OpenBio-LLM Prompt used for preprocessing the PadChest Dataset to convert full report summary into standard radiology report section - ”Findings”, ”Indication”, ”Impression”.

Method	Age Group $\uparrow$				Overall $\uparrow$
Method	ES-BLEU-1	ES-BLEU-4	ES-RadGraph-F1	ES-GREEN	BLEU-1	BLEU-4	RadGraph-F1	GREEN
FairLLaVA-Age ( $\mathcal{L}_{DAC}$ then $\mathcal{L}_{LM}+\mathcal{L}_{DIM}$ )	3.83	2.48	3.50	2.08	31.77	10.48	27.16	33.06
FairLLaVA-Age ( $\mathcal{L}_{LM}+\mathcal{L}_{DIM}+\mathcal{L}_{DAC}$ )	17.82	5.47	3.75	2.36	35.28	13.96	28.49	34.12

Appendix

Table S8 : ES-M metrics for the Age Group attribute on the MIMIC-CXR dataset and overall performance. Joint training of the

\mathcal{L}_{DAC}

with

\mathcal{L}_{DIM}

improves equity-scaled scores while preserving or improving overall report quality.

Figure S7 : Qualitative samples show LLaVA-Rad baseline produces unfair results for some subgroups (”Old”, “White”, ”Male”) of demographic attributes.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models
Oct 21, 2026Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili SabetMLLM Evaluation Multimodal Large Language Models
Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning +2
Bias-constrained multimodal intelligence for equitable and reliable clinical AI
Apr 18, 2026Cheng Li, Weijian Huang, Jiarun Liu +6Unified Multimodal Models Algorithmic Fairness
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. · Pengcheng Laboratory, Shenzhen, China. · University of Chinese Academy of Sciences, Beijing, China. +3
FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
Sep 1, 2026Vahid Reza Khazaie, Ahmed Y. Radwan, Shaina RazaVLM Evaluation
Vector Institute, Toronto, Canada

Oct 21, 2026J/K move · Enter open · S save

Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models

Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili Sabet

Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning+2

MLLM Evaluation Multimodal Large Language Models Language Model Bias Evaluation Multimodal Model Evaluation

Open paper AskPDF arXiv

Multi-modal Large Language Models (MLLMs) have seen broad adoption, but their biases are still far less examined than those of standard LLMs, creating possible social harms when deployed in real-world settings. Prior efforts to extend bias analyses from LLMs into the multi-modal space either cover only a few bias categories or do not ensure that the groups being compared are clearly addressed by visual cues in the images. To overcome these limitations, we introduce Melange, a wide-ranging synthetic visual benchmark covering seven categories for assessing social biases in MLLMs. It is grounded in the widely used language-only BBQ dataset (Bias Benchmark for Question Answering) from BigBench, carefully extending its constructed contexts via controlled generation of visual scenarios so the target attributes appear visually and clearly. To maximize reliability, our generated images pass through two stages of validation: automated screening using visual question answering (VQA) and subsequent manual inspection by human annotators. By systematically evaluating 23 open-source MLLMs across three matched evaluation settings---text-only, image-only, and image-with-context---Melange isolates the fairness effect of the input modality. Switching from text to vision reduces fairness by 0.291 on average, a gap that varies by model family rather than by scale alone. A perception control confirms that this gap is not an artifact of visual comprehension: models identify the depicted attributes at 90--95% accuracy yet still fail to abstain. When models do commit to a biased answer, polarity-aware stereotype alignment shows they choose the human-stereotypical response roughly three-quarters of the time.

Abstract

Multimodal Large Language Models

Explore similar work

Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models

Bias-constrained multimodal intelligence for equitable and reliable clinical AI

FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making