cs.LGFeb 2, 2026

FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance

Authors: Hyunsuk Chung, Soyeon Caren Han, Seungyeon Ji, Jinwoo Kim, Eun-Jung Holden, Kyungreem Han

Organizations: University of Melbourne, Melbourne, Australia · Brain Science Institute, Korea Institute of Science and Technology, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University · Division of Bio-Medical Science & Technology, University of Science and Technology KIST School

Abstract

Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains unclear whether their predictions can be controlled by explicitly modulating reliance on different internal feature pathways. Existing approaches to shortcut and spurious behavior primarily rely on post hoc analysis or data-level interventions, offering limited ability to directly intervene on how models use information. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables controllable modulation of feature reliance while keeping the task and predictive objective fixed. FiLoRA decomposes adaptation into feature-aligned low-rank modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals over internal representations. We evaluate FiLoRA across both controlled classification settings and generative multimodal tasks, and under a range of instruction types, including natural and compositional instructions. Results show that FiLoRA induces consistent and interpretable shifts in feature reliance, selectively amplifying or suppressing different feature groups in accordance with the instruction, without altering task semantics. Our findings suggest that instruction-conditioned parameter adaptation can serve as a practical mechanism for intervening on internal model behavior, providing a new perspective on controllability and analysis of multimodal systems beyond output-level prompting or post hoc interpretation.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GUIDE: Guiding Internal Evidence with Language Instructions

    Aug 31, 2026Soyeon Caren Han, Hyunsuk Chung, Jinwoo Kim +2Multimodal RobustnessMultimodal Large Language Models

  2. Foundation-Preserving Optimization in Generalized Eigenspace

    May 28, 2026Dongjun Kim, Adrian de Wynter, Huancheng Chen +2Low-Rank AdaptationParameter-Efficient Fine-Tuning

  3. CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

    Aug 28, 2026Runze Liu, Naibin Gu, Mingxu Ai +4Continual Learning for VLMsMultimodal Instruction Tuning