$φ$-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models

Thanh-Dat Truong; Huu-Thien Tran; Jackson Cothren; Bhiksha Raj; Khoa Luu

cs.LGFeb 26, 2026

$φ$ -DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models

Authors: Thanh-Dat Truong, Huu-Thien Tran, Jackson Cothren, Bhiksha Raj, Khoa Luu

Organizations: CVIU Lab, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA · Carnegie Mellon University, USA

Abstract

Fairness in Continual Learning for Large Multimodal Models (LMMs) is an emerging yet underexplored challenge, particularly in the presence of imbalanced data distributions that can lead to biased model updates and suboptimal performance across tasks. While recent continual learning studies have made progress in addressing catastrophic forgetting, the problem of fairness caused the imbalanced data remains largely underexplored. This paper presents a novel Fairness Direct Preference Optimization (FaiDPO or $φ$ -DPO) framework for continual learning in LMMs. In particular, we first propose a new continual learning paradigm based on Direct Preference Optimization (DPO) to mitigate catastrophic forgetting by aligning learning with pairwise preference signals. Then, we identify the limitations of conventional DPO in imbalanced data and present a new $φ$ -DPO loss that explicitly addresses distributional biases. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both forgetting and data imbalance. Additionally, to enable $φ$ -DPO-based continual learning, we construct pairwise preference annotations for existing benchmarks in the context of continual learning. Extensive experiments and ablation studies show the proposed $φ$ -DPO achieves State-of-the-Art performance across multiple benchmarks, outperforming prior continual learning methods of LMMs.

Figures & tables

Figure 1: Our Fairness DPO ( $\phi$ -DPO) approach to Continual Learning in LMMs . Prior continual learning methods, e.g . , LoRA , struggle under imbalanced multimodal data and suffer from catastrophic forgetting . The vanilla DPO is still influenced by the imbalanced data distributions . Our $\phi$ -DPO approach can (1) mitigate forgetting, (2) adapt continuously to new learning tasks, and (3) maintain robustness under data imbalance.

Figure 2: The Imbalanced Distribution of Multimodal Continual Learning Benchmarks. The distribution of samples across ScienceQA topics is highly skewed, i.e . categories with fewer training examples ( e.g . Grammar, Phonological Awareness, Word Study) exhibit significantly lower accuracy, while topics with richer data ( e.g . Biology, Physics) achieve stronger performance.

Figure 3: ScienceQA, Grounding, and OCR-VQA introduce progressively shifting visual distributions and alignment objectives, creating modality imbalance across tasks.

Figure 4: Our Proposed Continual Learning Approach via Fairness DPO for Large Multimodal Models . Traditional reinforcement learning with human feedback (RLHF) method optimize models through explicit reward maximization. Our framework instead reformulates RLHF as Direct Preference Optimization (DPO). The Fairness DPO loss mitigate the gradient biased under the imbalanced data.

Figure 5: Example of Our DPO Data in the Continual Learning Benchmark. Best viewed in color.

Method	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
Zeroshot	32.29	28.28	15.59	35.55	62.56	34.85	$-$	$-$	$-$
LoRA-FT* [ 36 ]	76.54	50.27	43.01	43.32	89.85	66.32	60.60	64.72	-7.15
O-LoRA* [ 105 ]	76.94	41.17	34.18	39.61	83.22	60.49	55.02	60.73	-6.83
MoELoRA* [ 13 ]	77.63	49.54	39.08	41.04	89.21	66.24	59.30	64.81	-8.68
CL-MoE* [ 37 ]	76.58	52.31	39.65	45.64	90.21	66.65	60.88	64.95	-7.22
HiDe* [ 26 ]	74.80	42.29	34.03	38.01	79.22	60.83	53.67	61.81	-8.95

Table 1: Results on MLLM-CL Domain (* denote the method using relay data). RS: Remote Sensing, Med: Medical, AD: Autonmous Driving, Sci: ScienceQA, Fin: Finance.

Method	OCR	M&L	VP	GUI	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
Zeroshot [ 36 ]	31.20	30.20	60.79	10.00	33.05	$-$	$-$	$-$
LoRA-FT* [ 36 ]	21.80	32.70	58.38	28.75	40.32	35.41	36.32	-6.55
O-LoRA* [ 105 ]	29.60	31.30	60.79	27.50	39.96	37.30	36.34	-3.55
MoELoRA* [ 13 ]	19.80	32.20	54.19	30.00	40.35	34.05	35.39	-8.41
CL-MoE* [ 37 ]	25.40	31.80	60.91	30.00	41.22	37.03	37.28	-5.59
HiDe* [ 26 ]	24.60	28.40	30.71	23.75	36.84	26.86	33.54	-13.30

Table 2: Results on MLLM-CL Ability (* denote the method using relay data). M&L: Math and Logic, VP: Visual Perception.

Method	SciQA	Image	Viz	Ground	Text	GQA	VQA	OCR	MFN $\uparrow$	MAA $\uparrow$
Zeroshot	69.79	9.93	45.50	58.47	57.75	60.77	66.50	64.93	$-$	$-$
FineTune	57.43	28.90	41.88	30.05	51.39	50.76	53.28	64.78	47.31	52.86
LwF [ 59 ]	60.71	30.58	41.49	36.01	52.80	47.07	53.43	65.12	48.40	53.22
EWC [ 47 ]	59.75	31.88	42.26	34.96	51.06	51.84	55.30	64.55	48.95	53.30
L2P [ 107 ]	70.21	23.31	44.21	43.76	56.25	58.46	62.32	64.11	52.83	53.96
O-LoRA [ 105 ]	72.56	62.84	48.43	58.97	57.66	59.14	63.21	63.31	60.77	62.60

Table 3: Results on CoIN. SciQA: ScienceQA, Image: ImageNet, Viz: VizWiz, Ground: Grounding, Text: TextVQA, VQA: VQAv2.

	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
Zeroshot	32.29	28.28	15.59	35.55	62.56	34.85	$-$	$-$	$-$
LoRA-FT	69.65	41.59	25.43	40.88	87.45	64.98	53.00	61.13	-14.97
KD	77.82	63.92	52.88	58.57	92.85	71.37	69.21	71.49	-2.71
DPO	82.26	67.12	55.79	59.58	93.94	72.80	71.74	73.68	-1.33
$\phi$ -DPO	85.68	69.74	57.73	61.55	95.28	74.29	74.00	75.68	-0.37

Table 4: Effectiveness of Our Fairness DPO.

$\beta$	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
0.01	83.09	67.71	56.90	62.06	96.77	75.43	73.31	75.83	-2.66
0.05	84.91	69.30	57.16	62.21	96.12	75.27	73.94	76.27	-1.66
0.10	85.68	69.74	57.73	61.55	95.28	74.29	74.00	75.68	-0.37
0.50	83.74	67.55	55.44	59.64	92.88	72.11	71.85	73.60	-0.32

Table 5: Effectiveness of Divergence Parameter

\beta

$\gamma$	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
0.00	82.26	67.12	55.79	59.58	93.94	72.80	71.74	73.68	-1.33
0.50	82.84	67.84	55.50	59.71	93.78	72.79	71.93	73.87	-1.07
1.00	84.01	68.44	56.56	60.74	94.06	73.38	72.76	74.66	-0.77
2.00	85.68	69.74	57.73	61.55	95.28	74.29	74.00	75.68	-0.37
5.00	83.08	67.74	55.99	60.05	94.03	73.21	72.18	74.27	-1.29

Table 6: Effectiveness of Focusing Parameter

\gamma

LLM	Method	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
LLaVA-7B	DPO	82.26	67.12	55.79	59.58	93.94	72.80	71.74	73.68	-1.33
LLaVA-7B	$\phi$ -DPO	85.68	69.74	57.73	61.55	95.28	74.29	74.00	75.68	-0.37
LLaVA-13B	DPO	86.24	70.28	58.02	63.26	96.84	75.89	74.93	77.00	-1.21
LLaVA-13B	$\phi$ -DPO	87.40	71.09	59.34	63.96	97.28	76.29	75.81	77.57	-0.59
InternVL-7B	DPO	82.69	67.69	55.97	60.67	94.83	73.88	72.37	74.68	-1.89
InternVL-7B	$\phi$ -DPO	85.64	69.88	57.94	61.95	95.74	74.62	74.23	75.94	-0.49

Table 7: Effectiveness of Different LMM Framework.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Noise Level	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
0%	85.68	69.74	57.73	61.55	95.28	74.29	74.00	75.68	-0.37
5%	84.15	68.73	57.26	60.94	95.27	74.09	73.27	75.18	-1.02
15%	83.06	67.51	56.16	60.40	94.47	73.39	72.32	74.42	-1.33

Appendix

Table 8: Noise Robustness Examination on the Preference Data

Method	LoRA+FT [ 36 ]	MoELoRA [ 13 ]	$\phi$ -DPO
Training Time	8.11s/iter	8.35s/iter	9.77s/iter
Adapter Memory	$\sim$ 0.65G	$\sim$ 0.65G	$\sim$ 0.65G

Appendix

Table 9: Efficiency Analysis of Parameter-efficient Training

$\lambda_{\textrm{SFT}}$	$\lambda_{\phi\textrm{-DPO}}$	RS	Med	AD	Sci	Fin	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
2.0	1.0	83.62	68.56	57.18	61.32	96.43	75.38	73.42	76.09	-2.45
1.0	1.0	85.68	69.74	57.73	61.55	95.28	74.29	74.00	75.68	-0.37
1.0	2.0	83.71	67.54	55.67	59.23	93.16	72.13	71.86	73.61	-0.33

Appendix

Table 10: Ablation on

\lambda_{\textrm{SFT}}

and

\lambda_{\phi\textrm{-DPO}}

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning
Aug 21, 2026Yuyang Luo, Kai ShuBackdoor Attacks on VLMs Multimodal Large Language Models
Emory University Atlanta, GA, USA
Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models
May 14, 2026Yuehao Liu, Shanyan Guan, Weijia Zhang +4Continual Learning for LLMs Continual Learning for VLMs
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
Mar 20, 2026Kewen Zhu, Liping Yi, Zhiming Zhao +3Personalized Language Model Alignment Direct Preference Optimization
College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore

Aug 21, 2026J/K move · Enter open · S save

Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning

Yuyang Luo, Kai Shu

Emory University Atlanta, GA, USA

Backdoor Attacks on VLMs Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are increasingly deployed in high-stakes domains where fairness is a critical safety requirement. In practice, these models are continually updated through continual learning (CL) to adapt to evolving tasks and data distributions. Prior work has shown that backdoor attacks can manipulate MLLM responses through hidden triggers, but naively implanted backdoors degrade as models undergo subsequent updates of CL. Although fairness has emerged as a central concern for MLLM deployment, whether backdoor-induced fairness violations can survive CL remains unexplored, leaving two critical questions unanswered: (1) whether a backdoor can reliably induce fairness violations in MLLMs, and (2) whether such fairness-targeted backdoors can persist through continual learning. We bridge this gap by proposing Persistent Fairness Backdoor Attack (PFBA) to inject persistent and group-specific discrimination into MLLMs. Specifically, PFBA achieves this through two novel mechanisms. The Latent Space Fairness Reinforcement reshapes the model's deep feature geometry by anchoring privileged-group representations to preserve utility while repelling and clustering targeted-group representations to sustain discrimination, and the Continual Learning Simulation iteratively optimizes the trigger against simulated parameter drift to ensure backdoor persistence across future updates. Extensive experiments demonstrate that PFBA induces severe fairness disparities that persist across continual learning rounds, evading standard backdoor defenses. The data and code are publicly available at https://github.com/lyygua/PFBA.

Abstract

Continual Learning

Explore similar work

Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment