Multiple-Debias: A Full-process Debiasing Method for Multilingual Pre-trained Language Models

Haoyu Liang; Peijian Zeng; Wentao Huang; Aimin Yang; Dong Zhou

cs.CLApr 3, 2026

Multiple-Debias: A Full-process Debiasing Method for Multilingual Pre-trained Language Models

Authors: Haoyu Liang, Peijian Zeng, Wentao Huang, Aimin Yang, Dong Zhou

Organizations: School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou, China

Abstract

Multilingual Pre-trained Language Models (MPLMs) have become essential tools for natural language processing. However, they often exhibit biases related to sensitive attributes such as gender, race, and religion. In this paper, we introduce a comprehensive multilingual debiasing method named Multiple-Debias to address these issues across multiple languages. By incorporating multilingual counterfactual data augmentation and multilingual Self-Debias across both pre-processing and post-processing stages, alongside parameter-efficient fine-tuning, we significantly reduced biases in MPLMs across three sensitive attributes in four languages. We also extended CrowS-Pairs to German, Spanish, Chinese, and Japanese, validating our full-process multilingual debiasing method for gender, racial, and religious bias. Our experiments show that (i) multilingual debiasing methods surpass monolingual approaches in effectively mitigating biases, and (ii) integrating debiasing information from different languages notably improves the fairness of MPLMs.

Figures & tables

Fig. 1: The framework of Multiple-Debias.

Model	DE	ZH	ES	JA	Avg. ( $\downarrow$ )
mBERT	8.48	2.50	4.26	3.58	4.71
+CDA	12.73	0.83	22.81	3.49	9.97
+SD	8.46	5.83	2.92	6.04	5.81
+MCDA	10.61	2.71	3.78	2.31	4.85
+MSD	0.83	3.33	2.98	3.43	2.64
+MD w/ Full Fine-Tune	5.06	4.17	0.92	0.50	2.66

TABLE I: Evaluation of Gender Bias on Multilingual CrowS-Pairs

Fig. 2: MBE bias scores of various debiasing methods on the mBERT model.

Fig. 3: MBE bias scores of various debiasing methods on the XLM-R model.

Model	DE	ZH	ES	JA	Avg. ( $\downarrow$ )
mBERT	4.74	15.00	5.00	3.33	7.02
+CDA	2.01	1.67	12.50	20.00	9.04
+SD	7.24	12.50	4.17	5.83	7.44
+MCDA	5.71	10.00	5.00	7.92	7.16
+MSD	5.56	6.67	5.83	7.50	6.39
+MD w/ Full Fine-Tune	2.93	6.67	3.33	8.33	5.32

TABLE II: Evaluation of Racial Bias on Multilingual CrowS-Pairs

Model	DE	ZH	ES	JA	Avg. ( $\downarrow$ )
mBERT	19.17	18.65	22.33	4.19	16.08
+CDA	22.50	22.07	23.21	9.25	19.26
+SD	6.67	6.77	12.24	2.52	7.05
+MCDA	17.71	16.57	22.55	3.36	15.05
+MSD	15.83	8.46	15.53	3.37	10.80
+MD w/ Full Fine-Tune	22.50	11.88	19.79	2.54	14.18

TABLE III: Evaluation of Religious Bias on Multilingual CrowS-Pairs

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 199 papers

A Heuristic Perspective on Debiasing Language Models
Aug 1, 2026Tian Lan, Yemin Wang, Chuancheng Shi +6Social Bias in Language Models Language Model Debiasing
Inner Mongolia University · Xiamen University · University of Sydney +3
General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level
Nov 23, 2023Bingkang Shi, Xiaodan Zhang, Dehan Kong +4Gender Bias in Language Models Debiased ML
Chinese Academy of Sciences, Institute of Information Engineering, Beijing 100093, China · Alibaba Group, Alibaba Artificial Intelligence Governance Laboratory, Beijing, China · University of Bristol, Department of Electrical and Electronic Engineering, Bristol, BS8 1UB, UK
Evaluating and Mitigating Anti-LGBTQ Biases in German and Multilingual Language Models
Aug 31, 2026Melina Morch, Daniel BraunMultilingual Language Model Evaluation Social Bias in Language Models
Marburg University · Computer Science · Cultural Data Studies +2

Aug 1, 2026J/K move · Enter open · S save

A Heuristic Perspective on Debiasing Language Models

Tian Lan, Yemin Wang, Chuancheng Shi, Xiangyu Wu+5

Inner Mongolia University · Xiamen University · University of Sydney+3

Social Bias in Language Models Language Model Debiasing Cultural Bias in Language Models Linguistic Bias in Language Models

Open paper AskPDF arXiv

Language models (LMs) often acquire various biases during pre-training and may express them in interactions, potentially causing social harm. Existing methods often rely on counterfactual augmentation or representation projection. These strategies remain limited in practice due to their high computational costs and difficulty in scaling to larger models. Additionally, many of these strategies require manual data annotation, narrowing their scope to specific cultures and bias categories. To overcome these limitations, we propose HEIMAT, a HEurIstic-style autoMATic debiasing framework for LMs. HEIMAT consists of two main steps: bias disclosure and debiasing fine-tuning. In the first step, it uses simple templates to construct heuristic prompts, which are applied to reveal model biases and generate corresponding context prompts. In the second step, it fine-tunes the model by minimizing the Jensen-Shannon divergence of predictions on these context prompts to reduce bias. Extensive experiments show that HEIMAT effectively mitigates bias in different cultures while maintaining the model's natural language understanding (NLU) performance.

Abstract

Social Bias in Language Models

Explore similar work

A Heuristic Perspective on Debiasing Language Models

General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level

Evaluating and Mitigating Anti-LGBTQ Biases in German and Multilingual Language Models