FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection

Maxime Fontana; Michael Spratling; Miaojing Shi

cs.CVMar 20, 2026

FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection

Authors: Maxime Fontana, Michael Spratling, Miaojing Shi

Organizations: King’s College London London, UK · University of Luxembourg Esch-sur-Alzette, Luxembourg · Tongji University Shanghai, China

Abstract

Adapting models pre-trained on large-scale datasets is a proven way to reach strong performance quickly for down-stream tasks. However, the growth of state-of-the-art mod-els makes traditional full fine-tuning unsuitable and difficult, especially for multi-task learning (MTL) where cost scales with the number of tasks. As a result, recent studies investigate parameter-efficient fine-tuning (PEFT) using low-rank adaptation to significantly reduce the number of trainable parameters. However, these existing methods use a single, fixed rank, which may not be optimal for differ-ent tasks or positions in the MTL architecture. Moreover, these methods fail to learn spatial information that cap-tures inter-task relationships and helps to improve diverse task predictions. This paper introduces Frequency-Aware and Automatic Rank (FAAR) for efficient MTL fine-tuning. Our method introduces Performance-Driven Rank Shrink-ing (PDRS) to allocate the optimal rank per adapter location and per task. Moreover, by analyzing the image frequency spectrum, FAAR proposes a Task-Spectral Pyramidal Decoder (TS-PD) that injects input-specific context into spatial bias learning to better reflect cross-task relationships. Experiments performed on dense visual task benchmarks show the superiority of our method in terms of both accuracy and efficiency compared to other PEFT methods in MTL. FAAR reduces the number of parameters by up to 9 times compared to traditional MTL fine-tuning whilst improving overall performance. Our code is available.

Figures & tables

Figure 1: Accuracy/Efficiency trade-off comparison of FAAR with other PEFT MTL methods on PASCAL. FAAR achieves better overall performance thank traditional fine-tuning methods or MTL low-rank PEFT methods, while using fewer parameters.

Panel of Figure 2 — Figure 2: (a) Overview of our FAAR method. An input image embedding is given to a pre-trained backbone, in our case, a Swin Transformer [ 30 ] . At each stage of that backbone, representing a different resolution, DoRA [ 28 ] adapters are placed at attention and MLP layers (illustrated in Fig. 2(b) ). Up until the last block, shared feature adaptation takes place. The last block is kept for task-specific updates. Each shared and task-specific stage-wise representation are retained for our Task-Spectral Pyramidal Decoders (TS-PD). (b) Illustration of the DoRA adapters. For simplicity, only the multi-task version is shown. Shared adapters consist only of the green path.

Figure 3: Our Task-Spectral Pyramidal Decoder (TS-PD) consists of a Channel-Wise task-specific Spectral Filter (CW-SP) and a Cross-Task Consensus Alignment (XT-Cons) module to maintain geometric consistency.

Method	SemSeg (mIoU $\uparrow$ )	HumanParts (mIoU $\uparrow$ )	Saliency (mIoU $\uparrow$ )	Normals (rmse $\downarrow$ )	$\Delta m$ (%)	Trainable Parameters (M)	Single inf. (all tasks)
Single Task	67.21	61.93	62.35	17.97	0	112.62	✕
MTL - Tuning Decoders Only	65.09	53.48	57.46	20.69	-9.95	1.94	✓
MTL - Full Fine Tuning	67.56	60.24	65.21	16.64	+2.23	30.06	✓
Adapter [ 18 ]	69.21	57.38	61.28	18.83	-2.71	11.24	✕
Bitfit [ 56 ]	68.57	55.99	60.64	19.42	-4.60	2.85	✕
VPT-shallow [ 21 ]	62.96	52.27	58.31	20.90	-11.18	2.57	✕

Table 1: Performance comparison on PASCAL-Context [ 34 ] using Swin-Tiny pretrained on ImageNet-1k as the backbone. We compare all methods with respect to

\Delta m

, the relative improvement vs. single-task full fine-tuning. FAAR improves all tasks with fewer parameters.

Method	SemSeg (mIoU $\uparrow$ )	Depth (rmse $\downarrow$ )	Normals (rmse $\downarrow$ )	$\Delta m$ (%)	Trainable Parameters (M)
Single Task	42.65	0.60	22.83	0	84.00
MTL - Full Fine Tuning	38.85	0.66	24.33	-8.49	28.10
MTLoRA [ 2 ] ( $r{=}64$ )	39.85	0.66	29.53	-15.30	9.70
TADFormer [ 4 ] ( $r{=}64$ )	40.85	0.64	27.48	-10.42	8.90
DiTASK [ 33 ]	41.13	0.65	27.25	-10.45	3.06
FAAR ( $r_{init}{=}64$ )	41.27	0.63	26.35	-7.88	2.85

Table 2: Performance comparison on NYUD-v2 [ 35 ] of LoRA-based MTL methods using Swin-Tiny pretrained on ImageNet-1k as the backbone. We report

\Delta m

as the relative improvement vs. single-task full fine-tuning.

Figure 4: Rank distribution across task-specific and shared layers for each transformer stage during early training.

Model	Sem. $\uparrow$	H.P $\uparrow$	Sal. $\uparrow$	Nor $\downarrow$	$\Delta m$
MTLoRA [2] (high)	67.90	59.84	65.40	16.60	+2.55
MTLoRA [2] (low)	65.57	57.93	63.48	18.54	-2.56
high w/ DoRA	67.55	60.00	64.70	17.20	+1.36
low w/ DoRA	65.52	58.06	63.65	18.43	-2.31
high + PDRS w/ LoRA	68.11	59.93	65.54	16.50	+2.83
high + PDRS w/ DoRA (1)	71.35	61.02	65.92	16.42	+4.92

Table 3: Additive Component Analysis on PASCAL

Nonlinear LoRA

See all 10 papers

Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning
Mar 22, 2026Shih-Wen Liu, Yen-Chang Chen, Wei-Ta Chu +2Multi-Task Learning Parameter-Efficient Fine-Tuning
National Cheng Kung University, Tainan, Taiwan · NVIDIA Research, Taiwan
Beyond Full-Model Rollback: AuroSFT for Adapter-State Multi-Task Fine-Tuning
Aug 5, 2026Yue Han, Ziniu LiuSupervised Fine-Tuning Multi-Task Learning
College of Systems Engineering, National University of Defense Technology · College of Computer Science and Technology, National University of Defense Technology
FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
Apr 2, 2026Juyong Jiang, Fan Wang, Hong Qi +2LLM Fine-Tuning Parameter-Efficient Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Mar 22, 2026J/K move · Enter open · S save

Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

Shih-Wen Liu, Yen-Chang Chen, Wei-Ta Chu, Fu-En Yang+1

National Cheng Kung University, Tainan, Taiwan · NVIDIA Research, Taiwan

Multi-Task Learning Parameter-Efficient Fine-Tuning Low-Rank Adaptation

Open paper AskPDF arXiv

Multi-task learning (MTL) aims to enable a single model to solve multiple tasks efficiently; however, current parameter-efficient fine-tuning (PEFT) methods remain largely limited to single-task adaptation. We introduce Free Sinewich, a parameter-efficient multi-task learning framework that enables near-zero-cost weight modulation via frequency switching (Free). Specifically, a Sine-AWB (Sinewich) layer combines low-rank factors and convolutional priors into a single kernel, which is then modulated elementwise by a sinusoidal transformation to produce task-specialized weights. A lightweight Clock Net is introduced to produce bounded frequencies that stabilize this modulation during training. Theoretically, sine modulation enhances the rank of low-rank adapters, while frequency separation decorrelates the weights of different tasks. On dense prediction benchmarks, Free Sinewich achieves state-of-the-art performance-efficiency trade-offs (e.g., up to +5.39% improvement over single-task fine-tuning with only 6.53M trainable parameters), offering a compact and scalable paradigm based on frequency-based parameter sharing. Project page: https://casperliuliuliu.github.io/projects/Free-Sinewich.

Abstract

Nonlinear LoRA

Explore similar work

Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

Beyond Full-Model Rollback: AuroSFT for Adapter-State Multi-Task Fine-Tuning

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models