FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance
Organizations: University of Melbourne, Melbourne, Australia · Brain Science Institute, Korea Institute of Science and Technology, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University · Division of Bio-Medical Science & Technology, University of Science and Technology KIST School
Abstract
Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains unclear whether their predictions can be controlled by explicitly modulating reliance on different internal feature pathways. Existing approaches to shortcut and spurious behavior primarily rely on post hoc analysis or data-level interventions, offering limited ability to directly intervene on how models use information. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables controllable modulation of feature reliance while keeping the task and predictive objective fixed. FiLoRA decomposes adaptation into feature-aligned low-rank modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals over internal representations. We evaluate FiLoRA across both controlled classification settings and generative multimodal tasks, and under a range of instruction types, including natural and compositional instructions. Results show that FiLoRA induces consistent and interpretable shifts in feature reliance, selectively amplifying or suppressing different feature groups in accordance with the instruction, without altering task semantics. Our findings suggest that instruction-conditioned parameter adaptation can serve as a practical mechanism for intervening on internal model behavior, providing a new perspective on controllability and analysis of multimodal systems beyond output-level prompting or post hoc interpretation.
Figures & tables
| Feature | MM-IMDb | CREMA-D | RAVDESS | |||
| Group | GMR | RS | GMR | RS | GMR | RS |
| Core | 0.42 | 0.18 | 0.38 | 0.16 | 0.35 | 0.14 |
| Spurious | 0.47 | 0.07 | 0.41 | 0.08 | 0.39 | 0.06 |
| Instruction Condition | CIDEr | CLIPScore | GMR |
| Semantic Focus | 112.4 | 0.823 | 0.41 |
| Appearance Focus | 109.7 | 0.817 | 0.39 |
| Appearance Suppress | 113.1 | 0.826 | 0.43 |
| Grouping Strategy | C/S RS Ratio | GMR |
| Semantic Grouping | 2.47 | 0.42 |
| Coarse Grouping | 1.53 | 0.31 |
| Random Grouping | 1.08 | 0.17 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Model | Total number of instructions |
| MM-IMDb | Qwen2.5-VL-7B-Instruct | 1,000 |
| CREMA-D | Qwen2.5-Omni-7B | 1,000 |
| RAVDESS | Qwen2.5-Omni-7B | 1,000 |
| Instruction Condition | CIDEr | CLIPScore | GMR |
| Semantic Focus | 112.4 | 0.823 | 0.41 |
| Appearance Focus | 109.7 | 0.817 | 0.39 |
| Appearance Suppress | 113.1 | 0.826 | 0.43 |
| Conceptual Focus | 111.8 | 0.821 | 0.40 |
| Behavior Focus | 110.9 | 0.819 | 0.38 |
| Background Suppress | 113.5 | 0.828 | 0.44 |
| Dataset | Proxy / Task Agreement |
| MM-IMDb | 0.68 |
| CREMA-D | 0.61 |
| RAVDESS | 0.59 |
| Dataset | Backbone | Core RS | Spurious RS | GMR |
| MM-IMDb | Qwen2.5-VL-7B | 0.11 | 0.16 | 0.43 |
| MM-IMDb | Qwen2.5-VL-32B | 0.17 | 0.21 | 0.51 |
| MM-IMDb | Gemma-3n-E4B | 0.10 | 0.14 | 0.39 |
| CREMA-D | Qwen2.5-Omni-7B | 0.14 | 0.11 | 0.45 |
| CREMA-D | AV-HuBERT | 0.15 | 0.10 | 0.47 |
| CREMA-D | AudioCLIP | 0.14 | 0.11 | 0.46 |
| Dataset | Core RS | Spurious RS | Core / Spurious ratio |
| MM-IMDb | 0.11 | 0.16 | 0.69 |
| CREMA-D | 0.14 | 0.11 | 1.27 |
| RAVDESS | 0.13 | 0.12 | 1.08 |