Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning

Yuyang Luo; Kai Shu

cs.LGAug 21, 2026

Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning

Authors: Yuyang Luo, Kai Shu

Organizations: Emory University Atlanta, GA, USA

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed in high-stakes domains where fairness is a critical safety requirement. In practice, these models are continually updated through continual learning (CL) to adapt to evolving tasks and data distributions. Prior work has shown that backdoor attacks can manipulate MLLM responses through hidden triggers, but naively implanted backdoors degrade as models undergo subsequent updates of CL. Although fairness has emerged as a central concern for MLLM deployment, whether backdoor-induced fairness violations can survive CL remains unexplored, leaving two critical questions unanswered: (1) whether a backdoor can reliably induce fairness violations in MLLMs, and (2) whether such fairness-targeted backdoors can persist through continual learning. We bridge this gap by proposing Persistent Fairness Backdoor Attack (PFBA) to inject persistent and group-specific discrimination into MLLMs. Specifically, PFBA achieves this through two novel mechanisms. The Latent Space Fairness Reinforcement reshapes the model's deep feature geometry by anchoring privileged-group representations to preserve utility while repelling and clustering targeted-group representations to sustain discrimination, and the Continual Learning Simulation iteratively optimizes the trigger against simulated parameter drift to ensure backdoor persistence across future updates. Extensive experiments demonstrate that PFBA induces severe fairness disparities that persist across continual learning rounds, evading standard backdoor defenses. The data and code are publicly available at https://github.com/lyygua/PFBA.

Figures & tables

Figure 1. Illustration of the persistent fairness-targeted backdoor problem in continual MLLM learning. (a) A clean MLLM deployed in high-stakes scenarios is expected to produce fair and correct responses across different demographic groups. (b) A dual-modal trigger injected into both image and text inputs activates group-specific unfair behavior, causing the targeted group to receive incorrect responses. (c) Under continual learning, naive fairness backdoors may be forgotten, while persistent fairness backdoors remain effective on triggered inputs. The figure highlights two questions (1) can backdoors induce fairness violations in MLLMs, and (2) can these violations persist through continual learning?

Figure 2. The pipeline of PFBA framework. (a) Continual Learning Simulation iteratively optimizes the trigger to survive catastrophic forgetting during future task updates. (b) Latent Space Fairness Reinforcement shapes feature geometry to preserve privileged-group utility while inducing stable target-group errors.

Dataset	Attack	Stage 1 (Initial Injection)			CL Method	Stage 2 (Adaptation)			Stage 3 (Further Forgetting)
		(Before CL starts)				(First Update)			(Long-term)
		PACC t $\downarrow$	PACC nt $\uparrow$	PBias $\uparrow$		PACC t $\downarrow$	PACC nt $\uparrow$	PBias $\uparrow$	PACC t $\downarrow$	PACC nt $\uparrow$	PBias $\uparrow$
HAM10000	PFBA (Ours)	11.31	60.47	49.16	LORA	60.62	90.67	30.04	62.34	93.23	30.90
					O-LORA	58.03	90.67	32.64	64.94	92.48	27.55
					SEFE	62.69	90.67	27.97	61.04	93.98	32.95
	TrojFair	10.12	74.42	64.30	LORA	70.47	86.67	16.20	84.42	93.23	8.82

Table 1. Fairness-targeted backdoor performance across three continual learning stages on HAM10000 and CheXpert under LoRA, O-LoRA, and SEFE. Best results are in bold.

	Dermatology		Radiology	Face Attr.
Domain	HAM10000	HIBA	CheXpert	CelebA
Domain 1	Back, Abdomen, Neck, Genital	Post./Ant./Lat. torso	Frontal (AP)	Young, no beard, no glasses
Domain 2	Lower Extr., Face, Scalp, Acral	Lower/Upper extr., Palms/Soles	Frontal (PA)	Young, with glasses
Domain 3	Trunk, Chest, Hand	Head/Neck, Oral/Genital	Lateral	Older or wearing hat

Table 2. Domain splits for each dataset under domain-incremental continual learning.

Dataset	Attack	Stage 1 (Initial Injection)				CL Method	Stage 2 (Adaptation)				Stage 3 (Further Forgetting)
		(Before CL starts)					(First Update)				(Long-term)
		CACC t $\uparrow$	CACC nt $\uparrow$	CACC $\uparrow$	CBias $\downarrow$		CACC t $\uparrow$	CACC nt $\uparrow$	CACC $\uparrow$	CBias $\downarrow$	CACC t $\uparrow$	CACC nt $\uparrow$	CACC $\uparrow$	CBias $\downarrow$
HAM10000	PFBA (Ours)	83.93	75.58	78.87	8.35	LORA	93.26	94.00	93.59	0.74	96.10	95.49	95.71	0.62
						O-LORA	92.23	92.67	92.42	0.44	96.10	94.74	95.24	1.37
						SEFE	93.78	92.00	93.00	1.78	96.20	94.74	95.24	1.47
	TrojFair	82.14	72.87	76.53	9.27	LORA	97.41	94.00	95.92	3.41	97.40	97.74	97.62	0.34

Table 3. Backdoored model performance on clean inputs across three continual learning stages on HAM10000 and CheXpert under LoRA, O-LoRA, and SEFE. Best results are bolded.

Figure 3. t-SNE visualization of Qwen2.5-VL intermediate layer representations on HAM10000 across three CL stages.

Panel of Figure 4 — Figure 4. Defense robustness of PFBA under spectral signatures and fine-pruning.

Panel of Figure 5 — Figure 5. Stage-wise comparison of different CL methods, including LoRA, O-LoRA, SEFE, and ER, under PFBA on HAM10000. Although ER improves clean-input utility, the backdoor remains persistent across continual learning stages, with high T-ASR and elevated PBias.

Dataset	Model	Stage 1				CL Method	Stage 2				Stage 3
		Initial Injection					First Update				Long-term
		CACC $\uparrow$	CBias $\downarrow$	PACC $\downarrow$	PBias $\uparrow$		CACC $\uparrow$	CBias $\downarrow$	PACC $\downarrow$	PBias $\uparrow$	CACC $\uparrow$	CBias $\downarrow$	PACC $\downarrow$	PBias $\uparrow$
HAM10000	LLaVA	76.29	7.69	43.43	49.16	LoRA	93.59	0.45	54.23	30.04	96.19	1.91	75.24	30.90
						O-LoRA	93.59	0.45	58.31	32.64	95.71	0.61	75.71	27.55
						SEFE	94.17	1.49	56.27	27.97	96.19	0.14	72.38	32.95
	Qwen	47.65	4.86	9.39	12.55	LoRA	44.31	7.67	7.29	11.93	54.29	6.56	10.48	16.54

Table 4. Performance of PFBA across datasets, MLLM backbones, and CL methods. Best results are in bold .

Model	HIBA Stage 1			HIBA Stage 2			HIBA Stage 3
Model	CACC $\uparrow$	T-ASR $\uparrow$	PBias $\uparrow$	CACC $\uparrow$	T-ASR $\uparrow$	PBias $\uparrow$	CACC $\uparrow$	T-ASR $\uparrow$	PBias $\uparrow$
LLaVA	65.79	48.89	4.92	69.09	63.03	6.77	73.04	48.48	4.75
DeepSeek	54.39	40.00	5.75	59.90	57.57	12.35	60.29	30.30	8.84

Table 5. Transferability of PFBA from HAM10000 to HIBA, averaged over LoRA, O-LoRA, and SEFE.

	Components		Metrics (Stage 3)
Method Variant	CL Sim.	Fairness Reinf.	PBias $\uparrow$	CBias $\downarrow$
w/o CL Simulation	✗	✓	18.41	1.09
w/o Fairness Reinf.	✓	✗	24.24	3.28
PFBA (Ours)	✓	✓	30.90	0.62

Table 6. Ablation study of PFBA key components.

Metric	Pixel	Patch	Border
PBias ( $\uparrow$ )	30.90	9.47	11.21
CBias ( $\downarrow$ )	0.62	1.64	3.01

Table 7. Ablation study of different visual trigger types.

Backdoor Attacks on VLMs

Backdoor attacks on vision-language models study hidden triggers that make multimodal systems behave maliciously while appearing normal.

See all 36 papers

$φ$ -DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models
Feb 26, 2026Thanh-Dat Truong, Huu-Thien Tran, Jackson Cothren +2Continual Learning Direct Preference Optimization
CVIU Lab, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA · Carnegie Mellon University, USA
Safe-FCL: Continual Risk Awareness and Fine-Grained Safety Aggregation for Federated Continual Learning of LLMs
Oct 14, 2026Shuaijie Ma, Yongkang Hu, Yu Cheng +4Federated Continual Learning Federated Learning
East China Normal University · Shanghai Innovation Institute
Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
Jun 18, 2026Van-Tuan Tran, Shruthi Gowda, Merim Dzaferagic +1Continual Learning LLM Routing
School of Computer Science and Statistics, Trinity College Dublin, Ireland · Department of Mathematics and Computer Science, Eindhoven University of Technology, The Netherlands

Feb 26, 2026J/K move · Enter open · S save

$φ$ -DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models

Thanh-Dat Truong, Huu-Thien Tran, Jackson Cothren, Bhiksha Raj+1

CVIU Lab, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA · Carnegie Mellon University, USA

Continual Learning Direct Preference Optimization Catastrophic Forgetting Class-Imbalanced Learning

Open paper AskPDF arXiv

Fairness in Continual Learning for Large Multimodal Models (LMMs) is an emerging yet underexplored challenge, particularly in the presence of imbalanced data distributions that can lead to biased model updates and suboptimal performance across tasks. While recent continual learning studies have made progress in addressing catastrophic forgetting, the problem of fairness caused the imbalanced data remains largely underexplored. This paper presents a novel Fairness Direct Preference Optimization (FaiDPO or

φ

-DPO) framework for continual learning in LMMs. In particular, we first propose a new continual learning paradigm based on Direct Preference Optimization (DPO) to mitigate catastrophic forgetting by aligning learning with pairwise preference signals. Then, we identify the limitations of conventional DPO in imbalanced data and present a new

φ

-DPO loss that explicitly addresses distributional biases. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both forgetting and data imbalance. Additionally, to enable

φ

-DPO-based continual learning, we construct pairwise preference annotations for existing benchmarks in the context of continual learning. Extensive experiments and ablation studies show the proposed

φ

-DPO achieves State-of-the-Art performance across multiple benchmarks, outperforming prior continual learning methods of LMMs.

Abstract

Backdoor Attacks on VLMs

Explore similar work

φφφ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models

Safe-FCL: Continual Risk Awareness and Fine-Grained Safety Aggregation for Federated Continual Learning of LLMs

Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs

$φ$ -DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models