Beyond Attribute-Centric Fairness: Modelling and Mitigating Unified Bias Representation in Omni-modal Large Language Models
Organizations: Univeristy of Oxford · The Chinese University of Hong Kong · The University of Sydney
Abstract
Omni-modal large language models (OLLMs) provide joint multimodal understanding and generation capabilities, making them well-suited to complex environments and promising backbones for next-generation AI. Despite these advances, the harmful stereotypes encoded in such models manifest in increasingly diverse forms, leading to broader and more severe societal consequences. Existing debiasing methods typically impose separate fairness objectives for each sensitive attribute and modality to minimise demographic-dependent variations. However, in omni-modal settings, this attribute-centric paradigm struggles to jointly address diverse social biases with distinct fairness objectives and to generalise beyond predefined attributes. To this end, we propose OmniDebias, one of the first methods for promoting fairness in OLLMs, which employs a bias-centric formulation to extract and mitigate a shared bias representation across attributes and modalities while preserving attribute-related knowledge. Specifically, we exploit activation inconsistencies of a given attribute (e.g., gender) across semantic contexts (e.g., occupations) to disentangle bias from legitimate knowledge. We further consolidate multiple extracted biases into a shared representation, formulating a unified objective that captures the underlying structure across social biases and generalises to unseen attributes. Extensive experiments demonstrate that eliminating unified bias with both post-hoc editing and fine-tuning can consistently improve fairness across diverse attributes and modalities in OLLMs while maintaining favourable utility.
Figures & tables
| Stage | Method | T2T Metric | T2I Metric | I2T Metric | |||||||||||||
| PPL ( ) | Gender ( ) | Race ( ) | Age ( ) | SES ( ) | Clip ( ) | MUSIQ ( ) | Gender ( ) | Race ( ) | Age ( ) | SES ( ) | PPL ( ) | Clip ( ) | Gender ( ) | Race ( ) | Age ( ) | ||
| Vanilla | 7.74 | 0.16 | 0.24 | 0.17 | 0.35 | 0.29 | 71.1 | 0.25 | 0.24 | 0.25 | 0.0095 | 9.29 | 0.31 | 0.22 | 0.25 | 0.40 | |
| Prompt | ForcePrompt | 7.42 | 0.14 + | 0.21 + | 0.16 + | 0.31 + | 0.28 | 72.2 | 0.32 + | 0.24 ± | 0.24 – | 0.0100 – | 6.36 | 0.32 | 0.14 + | 0.25 ± | 0.45 – |
| ENTIGEN | 7.51 | 0.15 + | 0.21 + | 0.16 + | 0.32 + | 0.28 | 72.4 | 0.32 + | 0.32 + | 0.23 – | 0.0097 – | 8.50 | 0.32 | 0.17 + | 0.13 + | 0.20 + | |
| PE | SDID | 7.73 | 0.16 ± | 0.24 ± | 0.17 ± | 0.34 + | 0.29 | 71.8 | 0.25 ± | 0.24 ± | 0.25 ± | 0.0100 – | 9.71 | 0.32 | 0.29 – | 0.44 – | 0.32 + |
| FairImagen | 7.88 | 0.16 ± | 0.25 – | 0.17 ± | 0.34 + | 0.29 | 71.0 | 0.29 + | 0.28 + | 0.31 + | 0.0091 + | 10.3 | 0.32 | 0.19 + | 0.15 + | 0.14 + | |
| Stage | Method | T2T Metric | T2I Metric | I2T Metric | |||||||||||||
| PPL ( ) | Gender ( ) | Race ( ) | Age ( ) | SES ( ) | Clip ( ) | MUSIQ ( ) | Gender ( ) | Race ( ) | Age ( ) | SES ( ) | PPL ( ) | Clip ( ) | Gender ( ) | Race ( ) | Age ( ) | ||
| Vanilla | 11.5 | 0.17 | 0.24 | 0.18 | 0.32 | 0.29 | 74.4 | 0.23 | 0.40 | 0.27 | 0.0101 | 20.1 | 0.32 | 0.31 | 0.25 | 0.47 | |
| Prompt | ForcePrompt | 11.5 | 0.16 + | 0.23 + | 0.17 + | 0.32 ± | 0.29 | 75.3 | 0.18 – | 0.36 – | 0.23 – | 0.0097 + | 21.2 | 0.33 | 0.29 + | 0.24 + | 0.54 – |
| ENTIGEN | 11.4 | 0.15 + | 0.20 + | 0.17 + | 0.31 + | 0.29 | 73.9 | 0.24 + | 0.37 – | 0.32 + | 0.0091 + | 28.1 | 0.32 | 0.34 – | 0.31 – | 0.55 – | |
| PE | SDID | 11.6 | 0.17 ± | 0.24 ± | 0.18 ± | 0.33 – | 0.29 | 74.8 | 0.22 – | 0.40 ± | 0.27 ± | 0.0098 + | 21.2 | 0.32 | 0.19 + | 0.35 – | 0.39 + |
| FairImagen | 11.3 | 0.16 + | 0.24 ± | 0.17 + | 0.33 – | 0.29 | 71.9 | 0.25 + | 0.45 + | 0.30 + | 0.0093 + | 19.4 | 0.32 | 0.27 + | 0.34 – | 0.41 + | |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Social Bias in Language Models
Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.
See all 217 papers