FairSSL: Fair Multimodal Self-Supervised Learning

Jiaee Cheong; Abtin Mogharabin; Paul Liang; Hatice Gunes; Sinan Kalkan

cs.LGAug 22, 2025

FairSSL: Fair Multimodal Self-Supervised Learning

Authors: Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, Sinan Kalkan

Organizations: Harvard University · University of Cambridge · Middle East Technical University · Massachusetts Institute of Technology

Abstract

Prevalent multimodal self-supervised learning (SSL) methods rely on the redundancy assumption: that different views share substantial task-relevant information. We argue that this assumption fails in complex, real-world settings characterized by heterogeneity (e.g., variable-length healthcare or behavioral data), where enforcing strict alignment can discard unique, modality-specific signals and inadvertently amplify bias. In this work, we propose FairSSL, a framework that leverages data heterogeneity as a resource for fairness rather than a hindrance. Unlike standard contrastive approaches, FairSSL uses a subject-aware Variance-Invariance-Covariance Regularization objective, where alignment is enforced across segments drawn from the same subject. We introduce a segment-based pooling strategy to handle variable-length modalities, and we regularize representations to encourage (i) sufficient within-subject variability, (ii) cross-modal and cross-subject invariance, and (iii) representation decorrelation. Theoretical analysis shows that our objective bounds the score gap between protected groups. Empirically, FairSSL significantly outperforms existing baselines on heterogeneous multimodal datasets, improving fairness without sacrificing downstream predictive performance. Code available at: https://github.com/abtinmU/FairSSL

Figures & tables

Figure 1: (a,b) Prior work has explored SSL for ML fairness in unimodal or tabular data settings. (c) FairSSL addresses the challenges of non-tabular multimodal data in a subject-aware and modality-aware manner.

		Approach			Evaluation			Fairness Measures
Study	Task	MM	Modality	SSL	BM	VL	AU-ROC	SP	EOpp	EOdd	EAcc
Alasadi et al. (2020)	Cyberbullying Detection	✓	VT		✓	✓		✓		✓
Schmitz et al. (2022)	Emotion Detection	✓	AVT			✓		✓	✓
Yan et al. (2020)	Personality Assessment	✓	AV		✓	✓		✓			✓
Kathan et al. (2022)	Humour Recognition	✓	AV		✓	✓		✓
Chen et al. (2023)	Recommendation	✓	AVT		✓	✓			✓	✓

Table 1: Comparative Summary with existing Multimodal Fairness SSL studies. Abbreviations (sorted): A: Audio. BM: Bias Mitigation. Cont: Contrastive. EAcc: Equal Accuracy. EEG: Electroencephalogram. EOdd: Equalised Odds. EOpp: Equality of Opportunity. MM: Multi-modal. Red: Redundancy reduction. SP: Statistical Parity. T: Text. V: Visual. VL: deals with heterogeneous data of v arying l ength.

Figure 2: FairSSL processes each modality for the same or different subjects and regularizes representations in a subject-aware manner.

Figure 3: Euclidean distance between the two gender centroids in a 2D PCA projection of the embeddings.

	D-Vlog			MIMIC-CXR			MODMA			MIMIC-III
	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T
M	.16	.17	.34	.20	.29	.49	.35	.33	.68	.35	.09	.44
F	.30	.37	.66	.18	.34	.51	.16	.16	.32	.46	.10	.56
T	.42	.58	1.00	.37	.63	1.00	.51	.49	1.00	.81	.19	1.00

Table 2: Dataset and target attribute breakdown across datasets. Abbreviations: F: Female. M: Male. T: Total.

Y_{0}

: Control group.

Y_{1}

: positive event group. Red highlights imbalanced splits. Green denotes relatively balanced splits.

	Method	Perf.		Fairness
	Method	Acc	F1	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
Supervised	CNNs
[.4pt/2pt]	X-add	0.60	0.66	0.83	1.84	0.90	0.89	0.70
	X-concat	0.58	0.65	0.63	1.40	0.71	0.99	0.73
	SEResnet	0.57	0.72	0.82	2.13	1.04	0.82	0.62
[.4pt/2pt]	Transformers

Table 3: DVlog: Comparison across multimodal supervised and self-supervised methods. *FairSSL is applied on DepressionDet. Bold and underline mark best and second-best, respectively.

	Method	Perf.		Fairness
	Method	Acc.	F1	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
Supervised	MultiDepr	0.79	0.67	0.00	0.00	$0.00$	0.76	0.19
	Effnetv2s	0.71	0.50	0.00	0.00	0.00	0.88	0.22
	FeatNet	0.71	0.67	0.67	0.50	$0.00$	0.33	0.38
[.4pt/2pt]	EMO-GCN	0.60	0.67	0.78	0.59	0.86	0.82	0.76
[.4pt/2pt]	EAV*	0.54	0.57	0.86	0.60	1.43	0.62	0.66

Table 4: MODMA: Comparison across multimodal supervised and self-supervised methods. *FairSSL is applied on FeatNet. Bold and underline mark best and second-best, respectively.

Method	Perf.		Fairness
Method	Acc.	F1	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
Contrastive methods
CoMM	0.74	0.79	1.04	1.04	1.03	1.04	0.96
QUEST	0.69	0.74	0.96	0.95	0.96	1.01	0.96
FACTORCL	0.69	0.77	0.98	0.98	0.97	1.04	0.97
CLIP	0.74	0.79	1.05	1.05	1.03	1.04	0.96

Table 5: MIMIC-CXR: Comparison across different multimodal SSL methods across gender . *Encoders used are ResNet-18 for the image modality and a Transformer encoder for the text modality. Bold and underline mark best and second-best, respectively.

Method	Perf.		Fairness
Method	Acc.	F1	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
Contrastive methods
CoMM	0.84	0.12	0.84	0.79	0.86	1.03	0.86
FOCAL	0.82	0.15	1.67	1.57	2.04	0.92	0.41
SimCLR	0.74	0.18	0.92	0.86	1.01	0.97	0.94
FACTORCL	0.81	0.16	0.78	0.67	0.54	1.04	0.74

Table 6: MIMIC-III: Comparison across different multimodal SSL methods. *VICReg is applied on the same architecture as SimCLR. Bold and underline mark best and second-best, respectively.

Figure 4: Comparison of contrastive and redundancy-reduction SSL methods with respect to F1 and AGG F scores.

Method	Fairness
Method	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
CoMM	1.07	1.32	1.77	1.56	0.72
QUEST	1.09	1.27	1.40	1.50	0.77
DeCUR	1.08	1.31	1.71	1.54	0.73
CLIP	1.08	1.39	1.80	1.51	0.72
VICReg (baseline) ∗	1.07	1.32	1.74	1.56	0.73

Table 7: MIMIC-CXR: Comparison across different multimodal SSL methods across age . *Encoders used are ResNet-18 for the image modality and a Transformer encoder for the text modality. Bold and underline mark best and second-best, respectively.

Method	Fairness
Method	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
CoMM	0.92	0.25	1.05	0.96	0.77
QUEST	0.94	0.24	1.03	0.97	0.78
DeCUR	0.91	0.25	1.02	0.96	0.78
CLIP	0.90	0.26	0.99	0.96	0.78
VICReg (baseline) ∗	0.91	0.25	1.00	0.97	0.78

Table 8: MIMIC-CXR: Comparison across different multimodal SSL methods across race . *Encoders used are ResNet-18 for the image modality and a Transformer encoder for the text modality. Bold and underline mark best and second-best, respectively.

		Performance		Fairness
Method		Acc.	F1	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
No Pooling		0.57	0.52	1.41	3.13	2.25	0.95	0.04
(baseline)
Single Pooling	M1	0.58	0.73	1.00	2.21	1.00	0.90	0.67
	M2	0.59	0.64	0.74	1.19	0.89	0.99	0.86
	M3	0.60	0.64	0.84	1.38	0.92	0.89	0.82

Table 9: D-Vlog: Ablation analysis on pooling. Comparisons include (i) “No pooling”: no pooling on

\{\mathbf{z}^{m}_{i,j}\}

for

m=m_{1}

m_{2}

, i.e. the baseline VICReg method, (ii) “Single Pooling”: pooling only on

\{\mathbf{z}^{m_{1}}_{i,j}\}

for

m_{1}

(as explained in Section 4 ) and (iii) “Double Pooling”: pooling on both

\{\mathbf{z}^{m_{1}}_{i,j}\}

and

\{\mathbf{z}^{m_{2}}_{i,j}\}

. Bold and underline mark best and second-best, respectively.

Method	Perf.		Fairness
Method	Acc.	F1	SP	EOpp	EOdd	EAcc	$\textrm{AGG}_{F}$
Contrastive methods
CoMM	0.62	0.63	1.30	2.88	4.49	0.77	0.47
[.4pt/2pt] w M1	0.63	0.65	1.16	2.57	2.11	0.86	0.25
w M2	0.57	0.31	0.44	0.33	1.00	0.95	0.68
w M3	0.62	0.65	0.84	1.86	1.00	0.89	0.72

Table 10: D-Vlog: Ablation analysis on FairSSL methods applied on other SSL multimodal methods. Best and second-best results are noted in bold and underline for each method separately.

Figure 5:

\textrm{AGG}_{F}

vs. F1-score Pareto Plot for DVlog. Red triangles represents best results from FairSSL. Blue circles represent baseline SSL methods. Yellow circles represent the SSL-methods with our FairSSL modifications.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Train			Val			Test
	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T
M	0.15	0.19	0.34	0.19	0.21	0.40	0.19	0.12	0.31
F	0.27	0.39	0.66	0.25	0.35	0.60	0.39	0.30	0.69
T	0.42	0.58	1.00	0.44	0.56	1.00	0.58	0.42	1.00

Appendix

Table A.11: Dataset split proportions for D-Vlog . Abbreviations:

Y_{0}

: Healthy Controls.

Y_{1}

: Major Depressive Disorder. M: male. F: female.

	Train			Val			Test
	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T
M	0.33	0.10	0.43	0.29	0.12	0.41	0.46	0.06	0.52
F	0.46	0.11	0.57	0.49	0.10	0.59	0.43	0.06	0.49
T	0.79	0.21	1.00	0.78	0.22	1.00	0.88	0.12	1.00

Appendix

Table A.12 : Subject proportions for MIMIC-III across data splits. Abbreviations:

Y_{0}

: Healthy Controls.

Y_{1}

: Major Depressive Disorder. M: male. F: female. T: total.

	Fold 1			Fold 2			Fold 3			Fold 4			Fold 5			Test
	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T
M	0.50	0.25	0.76	0.37	0.25	0.62	0.38	0.38	0.76	0.33	0.33	0.67	0.38	0.38	0.76	0.36	0.27	0.63
F	0.12	0.12	0.24	0.25	0.12	0.37	0.12	0.12	0.24	0.22	0.11	0.33	0.12	0.12	0.24	0.18	0.18	0.37
T	0.62	0.38	1.00	0.63	0.37	1.00	0.50	0.50	1.00	0.55	0.45	1.00	0.50	0.50	1.00	0.55	0.45	1.00

Appendix

Table A.13 : Subject proportions for MODMA across 5-fold cross-validation and test set. Abbreviations:

Y_{0}

: Healthy Controls.

Y_{1}

: Major Depressive Disorder. M: male. F: female. T: total.

	Train			Val			Test
	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T	$Y_{0}$	$Y_{1}$	T
M	0.20	0.29	0.48	0.20	0.29	0.49	0.20	0.29	0.49
F	0.18	0.34	0.52	0.18	0.33	0.51	0.18	0.33	0.51
T	0.37	0.63	1.00	0.38	0.62	1.00	0.38	0.62	1.00

Appendix

Table A.14 : Subject proportions for MIMIC-CXR across data splits. Abbreviations:

Y_{0}

: Control group.

Y_{1}

: positive event group. M: male. F: female. T: total.

	Method	Performance				Group Fairness
	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
Transformer	Bimod-cross	0.67	0.70	0.74	0.72	0.72	1.60	0.76	0.90	0.70
	Bimod-concat	0.58	0.65	0.67	0.65	0.76	1.69	1.02	0.74	0.70
	Perceiver	0.62	0.69	0.64	0.66	1.08	2.38	1.62	0.87	0.45
	SEResnet	0.57	0.58	0.94	0.72	0.82	2.13	1.04	0.82	0.62
	DepressionDet	0.62	0.66	0.71	0.69	0.86	1.91	1.17	0.79	0.64

Appendix

Table A.15: DVlog: Multimodal Results across both performance and fairness Best and second-best results are highlighted in bold and underline , respectively.

Mod.	Method	Performance				Group Fairness
Mod.	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
Audio	MSCDR	0.62	0.64	0.78	0.70	1.14	2.51	1.76	0.77	0.34
	Classify-Net	0.65	0.68	0.76	0.72	0.74	1.63	0.76	0.87	0.68
	DeiT	0.68	0.69	0.80	0.74	0.79	1.75	0.84	0.90	0.70
	DenseNet-201	0.60	0.70	0.55	0.62	0.62	1.37	0.64	0.86	0.69
	DeprDet (baseline)	0.61	0.65	0.71	0.68	1.09	2.27	1.28	0.87	0.56

Appendix

Table A.16: DVlog: Unimodal results of both the audio and visual modalities across performance and classification. Best and second-best results are highlighted in bold and underline , respectively.

	Method	Performance				Group Fairness
	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
CoMM	Baseline	0.62	0.73	0.55	0.63	1.30	2.88	4.49	0.77	0.47
[.4pt/2pt]	M1	0.63	0.70	0.60	0.65	1.16	2.57	2.11	0.86	0.25
	M2	0.57	0.50	0.22	0.31	0.44	0.33	1.00	0.95	0.68
	M3	0.62	0.70	0.60	0.65	0.84	1.86	1.00	0.89	0.72
	M4	0.62	0.69	0.62	0.65	0.97	2.14	1.51	0.84	0.54

Appendix

Table A.17: D-Vlog: A comparison of the performance and fairness across different SSL-based multimodal methods. Best results for each method are highlighted in bold . Abbreviations: DP: Double pooling. Best and second-best results are highlighted in bold and underline , respectively.

	Method	Performance				Group Fairness
	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
MM SSL	CoMM	0.84	0.17	0.10	0.12	0.84	0.79	0.86	1.03	0.86
	DeCUR	0.68	0.14	0.36	0.20	1.36	1.28	1.53	0.83	0.67
	FOCAL	0.82	0.17	0.14	0.15	1.67	1.57	2.04	0.92	0.41
	FACTORCL	0.81	0.16	0.17	0.16	0.78	0.67	0.54	1.04	0.74
	SimCLR	0.74	0.32	0.35	0.18	0.92	0.86	1.01	0.97	0.94

Appendix

Table A.18: MIMIC-III: SSL-based methods finetuned on accuracy . Comparison of the performance and fairness across different baseline multimodal SSL methods. Best results across all methods are highlighted in bold . Best and second-best results are highlighted in bold and underline , respectively.

	Method	Performance				Fairness				AUROC	$AGG_{F}$
	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	AUROC	$AGG_{F}$
CoMM	Baseline	0.86	0.37	0.14	0.21	0.94	0.89	1.79	0.95	0.68	0.75
[.4pt/2pt]	M1	0.82	0.22	0.20	0.21	0.92	0.87	1.56	0.95	0.67	0.79
	M2	0.84	0.21	0.18	0.19	0.97	0.92	1.14	0.96	0.68	0.93
	M3	0.86	0.26	0.12	0.16	0.93	0.88	1.28	0.97	0.67	0.88
	M4	0.87	0.36	0.18	0.24	1.13	1.06	1.43	0.97	0.66	0.84

Appendix

Table A.19: MIMIC-III: SSL-based methods finetuned on AUROC . Comparison of the performance and fairness across different multimodal SSL methods. Best and second-best results are highlighted in bold and underline , respectively.

		Performance				Group Fairness
		Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
EEG	DeprNet	0.69	0.81	0.42	0.51	0.16	0.12	0.00	0.94	0.31
	EEGNet	0.70	0.80	0.50	0.54	0.11	0.08	0.00	0.77	0.24
	FeatureNet	0.86	1.00	0.67	0.80	1.33	1.00	0.00	1.33	0.59
	HybGNN IA	0.85	1.00	0.65	0.79	0.00	0.00	0.00	0.68	0.17
	FeatNet*	0.73	1.00	0.37	0.54	0.00	0.00	0.00	0.86	0.22

Appendix

Table C.20: MODMA: Combined EEG and Audio results. Best and second-best results are highlighted in bold and underline , respectively. * denotes baseline.

	Method	Performance				Group Fairness
	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
CNN	MultiDepr	0.79	1.00	0.50	0.67	0.00	0.00	0.00	0.76	0.19
	Effnetv2s	0.71	1.00	0.33	0.50	0.00	0.00	0.00	0.88	0.22
	FeatNet*	0.71	0.67	0.67	0.67	0.67	0.50	0.00	0.33	0.38
[.4pt/2pt]	EMO GCN	0.60	0.52	0.93	0.67	0.78	0.59	0.86	0.82	0.76
[.4pt/2pt]	Transformer	0.54	0.80	0.20	0.57	0.86	0.60	1.43	0.62	0.66

Appendix

Table C.21: MODMA: Baseline comparison of the performance and fairness across different multimodal methods. Best and second-best results are highlighted in bold and underline , respectively. * denotes baseline.

	Method	Performance				Group Fairness
	Method	Acc.	Prec.	Rec.	F1	$SP$	$EOpp$	$EOdd$	$EAcc$	$AGG_{F}$
CoMM	Baseline	0.58	0.54	0.09	0.14	0.83	0.63	4.00	1.11	0.09
	M1	0.57	0.50	0.22	0.31	0.44	0.33	1.00	0.95	0.68
	M2	0.53	0.42	0.34	0.38	1.06	0.75	1.00	1.11	0.89
	M3	0.69	0.97	0.33	0.50	0.67	0.50	0.00	1.17	0.50
	M4	0.53	0.43	0.33	0.38	0.97	0.74	0.95	1.12	0.88

Appendix

Table C.22: MODMA: SSL. Baseline comparison of the performance and fairness across different SSL methods. Best and second-best results are highlighted in bold and underline , respectively.

Method	Params (M)	Peak Mem (GB)	Total Time (s)	Time vs VICReg	Mem vs VICReg
COMM	3.89	18.28	99.78	+24.66%	+73.08%
QUEST	4.94	10.56	79.96	-0.10%	+0.00%
CLIP	3.89	10.54	79.52	-0.65%	-0.19%
FACTORCL	4.94	18.29	98.62	+23.21%	+73.21%
DECUR	4.94	18.29	98.24	+22.74%	+73.21%
VICReg	4.94	10.56	80.04	+0.00%	+0.00%

Appendix

Table C.23: Computational overhead measured on MODMA under 10 SSL epochs, 5 probe epochs, and 2 subjects per batch.

Model	Variant	Hyper-parameters
CoMM	Baseline	batch_size = 256, ssl_epochs = 20, cov_coeff = 15, linear_lr = 0.691, lr = 0.057, mi_coeff = 7, mlp = 512-256-128, sl_epochs = 20, tau = 0.682, var_coeff = 11, wd = 0.149
CoMM	M1	batch_size = 256, ssl_epochs = 10, cov_coeff = 13, linear_lr = 0.821, lr = 0.977, mi_coeff = 18, mlp = 512-256-128, sl_epochs = 10, tau = 0.800, var_coeff = 7, wd = 0.611
CoMM	M2	batch_size = 64, ssl_epochs = 19, cov_coeff = 8, linear_lr = 0.486, lr = 0.033, mi_coeff = 15, mlp = 512-256-128, sl_epochs = 20, tau = 0.194, var_coeff = 19, wd = 0.894
CoMM	M3	batch_size = 128, ssl_epochs = 25, cov_coeff = 3, linear_lr = 0.910, lr = 0.113, mi_coeff = 9, mlp = 512-256-128, sl_epochs = 9, tau = 0.987, var_coeff = 12, wd = 0.718
CoMM	M4	batch_size = 256, ssl_epochs = 22, cov_coeff = 4, linear_lr = 0.982, lr = 0.046, mi_coeff = 14, mlp = 512-256-128, sl_epochs = 20, tau = 0.935, var_coeff = 8, wd = 0.308
QUEST	Baseline	align_coeff = 13, batch_size = 64, linear_lr = 0.010, lr = 0.020, mlp = 512-256-128, orth_coeff = 6, sl_epochs = 10, sp_coeff = 0, uniq_coeff = 14, var_coeff = 14, ssl_epochs = 10, wd = 0.715

Appendix

Table D.24: DVlog: Complete hyper-parameter listings for the different models. These values are obtained by Bayesian optimization over all the given hyperparameters with the purpose to maximize the model performance. For further details, see Sections S1 and S2. Abbreviations: DP: double pooling.

Model	Variant	Hyper-parameters
CoMM	Baseline	batch_size = 32, cov_coeff = 17.325, epochs_ft = 13, epochs_ssl = 11, log_every = 20, lr_ft = 0.007, lr_ssl = 0.000, mi_coeff = 4.415, mlp = 96-128-50, tau = 0.177, var_coeff = 17.007
CoMM	M1	batch_size = 32, cov_coeff = 17.386, epochs_ft = 9, epochs_ssl = 11, log_every = 20, lr_ft = 0.003, lr_ssl = 0.000, mi_coeff = 4.633, mlp = 96-128-50, tau = 0.137, var_coeff = 17.485
CoMM	M2	batch_size = 32, cov_coeff = 14.804, epochs_ft = 10, epochs_ssl = 10, log_every = 20, lr_ft = 0.047, lr_ssl = 0.001, mi_coeff = 3.062, mlp = 96-128-50, tau = 0.139, var_coeff = 18.542
CoMM	M3	batch_size = 32, cov_coeff = 4.337, epochs_ft = 10, epochs_ssl = 10, log_every = 20, lr_ft = 0.035, lr_ssl = 0.000, mi_coeff = 4.313, mlp = 96-128-50, tau = 0.184, var_coeff = 18.142
CoMM	M4	batch_size = 32, cov_coeff = 9.216, epochs_ft = 10, epochs_ssl = 10, log_every = 20, lr_ft = 0.080, lr_ssl = 0.000, mi_coeff = 2.639, mlp = 96-256-64, tau = 0.193, var_coeff = 10.201
DeCUR	Baseline	batch_size = 32, common_coeff = 18.195, epochs_ft = 12, epochs_ssl = 10, intra_coeff = 11.025, lambda_off = 1.500, log_every = 20, lr_ft = 0.067, lr_ssl = 0.001, mlp = 96-256-64, unique_coeff = 15.009, var_coeff = 8.923

Appendix

Table D.25: MIMIC-III: Complete hyper-parameter listings for the different models. These values are obtained by Bayesian optimization over all the given hyperparameters with the purpose to maximize the model performance. For further details, see Sections S1 and S2.

Model	Variant	Hyper-parameters
CoMM	Baseline	cov_coeff = 8.523, ssl_epoch = 10, , sl_epoch = 9, learning_rate = 4.306, mi_coeff = 8.029, mlp = 512-256-1024, patience = 5, subjects_per_batch = 4, tau = 0.070, var_coeff = 9.831, weight_decay = 0.629
CoMM	M1	cov_coeff = 5.440, ssl_epoch = 20, , sl_epoch = 10, learning_rate = 2.662, mi_coeff = 6.071, mlp = 512-256-1024, patience = 5, subjects_per_batch = 2, tau = 0.070, var_coeff = 6.899, weight_decay = 0.286
CoMM	M2	cov_coeff = 6.483, ssl_epoch = 7, sl_epoch = 8, learning_rate = 2.388, mi_coeff = 7.884, mlp = 512-256-1024, patience = 5, subjects_per_batch = 4, tau = 0.038, var_coeff = 7.816, weight_decay = 3.122
CoMM	M3	cov_coeff = 8.513, ssl_epoch = 8,, sl_epoch = 9, learning_rate = 3.159, mi_coeff = 8.167, mlp = 512-256-1024, patience = 6, subjects_per_batch = 4, tau = 0.066, var_coeff = 7.511, weight_decay = 2.893
CoMM	M4	cov_coeff = 8.446, ssl_epoch = 4, , sl_epoch = 8, learning_rate = 3.210, mi_coeff = 6.574, mlp = 512-256-1024, patience = 6, subjects_per_batch = 4, tau = 0.039, var_coeff = 2.266, weight_decay = 1.433
FACTORCL	Baseline	ssl_epoch = 20, sl_epoch = 4, , inv_coeff = 8, learning_rate = 1.327, mlp = 512-1024-256, patience = 6, shared_coeff = 6, subjects_per_batch = 3, tau = 0.000, weight_decay = 0.687

Appendix

Table D.26: MODMA: Complete hyper-parameter listings for the different models. These values are obtained by Bayesian optimization over all the given hyperparameters with the purpose to maximize the model performance. For further details, see Sections S1 and S2.

Explore similar work

ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs
May 3, 2026Marah Halawa, Olaf HellwichContrastive Loss Contrastive Learning
Technische Universität Berlin
FAIRVAR: Fair Federated Learning via Variance Regularization
Aug 16, 2025Zahra Kharaghani, Ali Dadras, Tommy LöfstedtFederated Learning Algorithmic Fairness
Department of Computing Science, Umeå University, Umeå, Sweden · Department of Mathematics and Mathematical Statistics, Umeå University, Umeå, Sweden
Avoiding Structural Failure Modes in Tabular Fair SSL: Online Primal-Dual Allocation under Confidence Gating
May 15, 2026Hangchuan Liang, Changchun LiAlgorithmic Fairness Semi-Supervised Learning
College of Computer Science and Technology, Jilin University, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education, Jilin University, China

May 3, 2026, cs.CVJ/K move · Enter open · S save

ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs

Marah Halawa, Olaf Hellwich

Technische Universität Berlin

Contrastive Loss Contrastive Learning Self-Supervised Learning

Open paper AskPDF arXiv

Self-supervised learning methods learn high-quality visual representations, yet recent studies show that these representations often capture demographic biases present in the training data. Existing fairness-aware methods address this by redesigning the self-supervised objective itself, limiting portability across the rapidly evolving landscape of self-supervised learning (SSL) frameworks. We propose ProtoFair, a fairness-aware contrastive loss designed to work alongside existing SSL objectives without modifying them. ProtoFair leverages unsupervised prototype clustering to identify pseudo-counterfactual pairs: samples sharing the same cluster assignment but belonging to different sensitive groups. By pulling these content-matched, cross-group samples together in the embedding space, ProtoFair encourages the encoder to learn representations that are invariant to the sensitive attribute. The method requires only sensitive attribute annotations, no target labels, and integrates seamlessly with both SimCLR and SupCon. Experiments on CelebA and UTKFace demonstrate consistent fairness improvements while maintaining competitive accuracy.

Abstract

Explore similar work

ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs

FAIRVAR: Fair Federated Learning via Variance Regularization

Avoiding Structural Failure Modes in Tabular Fair SSL: Online Primal-Dual Allocation under Confidence Gating