cs.LGAug 21, 2026

Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning

Authors: Yuyang Luo, Kai Shu

Organizations: Emory University Atlanta, GA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed in high-stakes domains where fairness is a critical safety requirement. In practice, these models are continually updated through continual learning (CL) to adapt to evolving tasks and data distributions. Prior work has shown that backdoor attacks can manipulate MLLM responses through hidden triggers, but naively implanted backdoors degrade as models undergo subsequent updates of CL. Although fairness has emerged as a central concern for MLLM deployment, whether backdoor-induced fairness violations can survive CL remains unexplored, leaving two critical questions unanswered: (1) whether a backdoor can reliably induce fairness violations in MLLMs, and (2) whether such fairness-targeted backdoors can persist through continual learning. We bridge this gap by proposing Persistent Fairness Backdoor Attack (PFBA) to inject persistent and group-specific discrimination into MLLMs. Specifically, PFBA achieves this through two novel mechanisms. The Latent Space Fairness Reinforcement reshapes the model's deep feature geometry by anchoring privileged-group representations to preserve utility while repelling and clustering targeted-group representations to sustain discrimination, and the Continual Learning Simulation iteratively optimizes the trigger against simulated parameter drift to ensure backdoor persistence across future updates. Extensive experiments demonstrate that PFBA induces severe fairness disparities that persist across continual learning rounds, evading standard backdoor defenses. The data and code are publicly available at https://github.com/lyygua/PFBA.

Figures & tables

Figure 1. Illustration of the persistent fairness-targeted backdoor problem in continual MLLM learning. (a) A clean MLLM deployed in high-stakes scenarios is expected to produce fair and correct responses across different demographic groups. (b) A dual-modal trigger injected into both image and text inputs activates group-specific unfair behavior, causing the targeted group to receive incorrect responses. (c) Under continual learning, naive fairness backdoors may be forgotten, while persistent fairness backdoors remain effective on triggered inputs. The figure highlights two questions (1) can backdoors induce fairness violations in MLLMs, and (2) can these violations persist through continual learning?
Figure 2. The pipeline of PFBA framework. (a) Continual Learning Simulation iteratively optimizes the trigger to survive catastrophic forgetting during future task updates. (b) Latent Space Fairness Reinforcement shapes feature geometry to preserve privileged-group utility while inducing stable target-group errors.
DatasetAttackStage 1 (Initial Injection)CL MethodStage 2 (Adaptation)Stage 3 (Further Forgetting)
(Before CL starts)(First Update)(Long-term)
PACC t ↓\downarrowPACC nt ↑\uparrowPBias ↑\uparrowPACC t ↓\downarrowPACC nt ↑\uparrowPBias ↑\uparrowPACC t ↓\downarrowPACC nt ↑\uparrowPBias ↑\uparrow
HAM10000PFBA (Ours)11.3160.4749.16LORA60.6290.6730.0462.3493.2330.90
O-LORA58.0390.6732.6464.9492.4827.55
SEFE62.6990.6727.9761.0493.9832.95
TrojFair10.1274.4264.30LORA70.4786.6716.2084.4293.238.82
Table 1. Fairness-targeted backdoor performance across three continual learning stages on HAM10000 and CheXpert under LoRA, O-LoRA, and SEFE. Best results are in bold.
 DermatologyRadiologyFace Attr.
DomainHAM10000HIBACheXpertCelebA
Domain 1Back, Abdomen, Neck, GenitalPost./Ant./Lat. torsoFrontal (AP)Young, no beard, no glasses
Domain 2Lower Extr., Face, Scalp, AcralLower/Upper extr., Palms/SolesFrontal (PA)Young, with glasses
Domain 3Trunk, Chest, HandHead/Neck, Oral/GenitalLateralOlder or wearing hat
Table 2. Domain splits for each dataset under domain-incremental continual learning.
DatasetAttackStage 1 (Initial Injection)CL MethodStage 2 (Adaptation)Stage 3 (Further Forgetting)
(Before CL starts)(First Update)(Long-term)
CACC t ↑\uparrowCACC nt ↑\uparrowCACC ↑\uparrowCBias ↓\downarrowCACC t ↑\uparrowCACC nt ↑\uparrowCACC ↑\uparrowCBias ↓\downarrowCACC t ↑\uparrowCACC nt ↑\uparrowCACC ↑\uparrowCBias ↓\downarrow
HAM10000PFBA (Ours)83.9375.5878.878.35LORA93.2694.0093.590.7496.1095.4995.710.62
O-LORA92.2392.6792.420.4496.1094.7495.241.37
SEFE93.7892.0093.001.7896.2094.7495.241.47
TrojFair82.1472.8776.539.27LORA97.4194.0095.923.4197.4097.7497.620.34
Table 3. Backdoored model performance on clean inputs across three continual learning stages on HAM10000 and CheXpert under LoRA, O-LoRA, and SEFE. Best results are bolded.
Figure 3. t-SNE visualization of Qwen2.5-VL intermediate layer representations on HAM10000 across three CL stages.
Figure 4. Defense robustness of PFBA under spectral signatures and fine-pruning.
Figure 5. Stage-wise comparison of different CL methods, including LoRA, O-LoRA, SEFE, and ER, under PFBA on HAM10000. Although ER improves clean-input utility, the backdoor remains persistent across continual learning stages, with high T-ASR and elevated PBias.
DatasetModelStage 1CL MethodStage 2Stage 3
Initial InjectionFirst UpdateLong-term
CACC ↑\uparrowCBias ↓\downarrowPACC ↓\downarrowPBias ↑\uparrowCACC ↑\uparrowCBias ↓\downarrowPACC ↓\downarrowPBias ↑\uparrowCACC ↑\uparrowCBias ↓\downarrowPACC ↓\downarrowPBias ↑\uparrow
HAM10000LLaVA76.297.6943.4349.16LoRA93.590.4554.2330.0496.191.9175.2430.90
O-LoRA93.590.4558.3132.6495.710.6175.7127.55
SEFE94.171.4956.2727.9796.190.1472.3832.95
Qwen47.654.869.3912.55LoRA44.317.677.2911.9354.296.5610.4816.54
Table 4. Performance of PFBA across datasets, MLLM backbones, and CL methods. Best results are in bold .
ModelHIBA Stage 1HIBA Stage 2HIBA Stage 3
CACC ↑\uparrowT-ASR ↑\uparrowPBias ↑\uparrowCACC ↑\uparrowT-ASR ↑\uparrowPBias ↑\uparrowCACC ↑\uparrowT-ASR ↑\uparrowPBias ↑\uparrow
LLaVA65.7948.894.9269.0963.036.7773.0448.484.75
DeepSeek54.3940.005.7559.9057.5712.3560.2930.308.84
Table 5. Transferability of PFBA from HAM10000 to HIBA, averaged over LoRA, O-LoRA, and SEFE.
 ComponentsMetrics (Stage 3)
Method VariantCL Sim.Fairness Reinf.PBias ↑\uparrowCBias ↓\downarrow
w/o CL Simulation✗✓18.411.09
w/o Fairness Reinf.✓✗24.243.28
PFBA (Ours)✓✓30.900.62
Table 6. Ablation study of PFBA key components.
MetricPixelPatchBorder
PBias ( ↑\uparrow )30.909.4711.21
CBias ( ↓\downarrow )0.621.643.01
Table 7. Ablation study of different visual trigger types.
More in this topic

Backdoor Attacks on VLMs

Backdoor attacks on vision-language models study hidden triggers that make multimodal systems behave maliciously while appearing normal.

See all 36 papers

Explore similar work

CardsList
Feb 26, 2026

φφ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models

Fairness in Continual Learning for Large Multimodal Models (LMMs) is an emerging yet underexplored challenge, particularly in the presence of imbalanced data distributions that can lead to biased model updates and suboptimal performance across tasks. While recent continual learning studies have made progress in addressing catastrophic forgetting, the problem of fairness caused the imbalanced data remains largely underexplored. This paper presents a novel Fairness Direct Preference Optimization (FaiDPO or φφ-DPO) framework for continual learning in LMMs. In particular, we first propose a new continual learning paradigm based on Direct Preference Optimization (DPO) to mitigate catastrophic forgetting by aligning learning with pairwise preference signals. Then, we identify the limitations of conventional DPO in imbalanced data and present a new φφ-DPO loss that explicitly addresses distributional biases. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both forgetting and data imbalance. Additionally, to enable φφ-DPO-based continual learning, we construct pairwise preference annotations for existing benchmarks in the context of continual learning. Extensive experiments and ablation studies show the proposed φφ-DPO achieves State-of-the-Art performance across multiple benchmarks, outperforming prior continual learning methods of LMMs.

Thanh-Dat Truong, Huu-Thien Tran, Jackson Cothren +2

CVIU Lab, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA · Carnegie Mellon University, USA

Continual LearningDirect Preference OptimizationCatastrophic ForgettingClass-Imbalanced Learning
AskPDF
Oct 14, 2026

Safe-FCL: Continual Risk Awareness and Fine-Grained Safety Aggregation for Federated Continual Learning of LLMs

Federated fine-tuning enables large language models (LLMs) to learn collaboratively from distributed client data, but malicious client updates can compromise global model safety. As federated tasks evolve, changing malicious update patterns challenge the ability of existing defenses to maintain reliable risk detection over time. To address this challenge, we propose Safe-FCL, a framework integrating continual risk awareness with fine-grained safety aggregation for federated continual learning of LLMs. Specifically, the Continual Risk Awareness (CRA) mechanism learns emerging risk patterns across sequential tasks and constrains detector updates using a protective subspace derived from historical features, mitigating forgetting of previously learned malicious patterns while adapting to new risks. To translate risk awareness into aggregation-level safety control, the Fine-Grained Safety Aggregation (FGSA) mechanism combines client-level risk assessments from CRA with module-level deviations from historical update behavior to adaptively regulate aggregation contributions, suppressing potentially malicious updates while minimizing unnecessary restrictions on benign updates. Extensive experiments demonstrate that Safe-FCL outperforms current SOTA methods, improving global-model Rule and MD-Judge scores in federated continual learning by 0.41% and 0.64%, respectively, while increasing average risk recognition accuracy by 2.01%.

Shuaijie Ma, Yongkang Hu, Yu Cheng +4

East China Normal University · Shanghai Innovation Institute

Federated Continual LearningFederated LearningLLM SafetyFederated Learning Aggregation
AskPDF
Jun 18, 2026

Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs

Continual adaptation is essential for multimodal large language models (MLLMs) deployed across evolving domains, but the state-of-the-art MR-LoRA method highly relies on the assumption that a MLLM-based router is necessary to process complex multimodal inputs. This paper revisits this claim on the MLLM-CL benchmark and argues for two claims. First, routing does not require an MLLM: a simple training-free, replay-free ptotypical routing method (RePRo), uses frozen pretrained features and task prototypes to match the MLLM-based router of MR-LoRA at far lower computational cost. Second, shared experts do not improve continual learning for MLLMs, despite their theoretical appeal. We show that these findings arise from two structural limitations of MLLM-CL: (1) its tasks are highly separable in representation space, and (2) its fixed task order makes conclusions sensitive to a single curriculum rather than robust across diverse continual-learning trajectories. As a result, the benchmark primarily rewards learning in isolation rather than genuine continual transfer. This motivates a new design for future benchmarks of continual MLLM learning, with overlapping task manifolds, multiple task orders, fine-grained domain shifts, and evaluation protocols that reward forward transfer as well as retention.

Van-Tuan Tran, Shruthi Gowda, Merim Dzaferagic +1

School of Computer Science and Statistics, Trinity College Dublin, Ireland · Department of Mathematics and Computer Science, Eindhoven University of Technology, The Netherlands

Continual LearningLLM RoutingContinual Learning for VLMs
AskPDF